Migrating Data from v3

Terminology Note

In v3 we had “resource graphs”, while in v4 and later we call these “Resource Models”. Conceptually they are the same. We’ll be referring to them here as “v3 graphs” and “Resource Models”, respectively.

Note

In the following guides, you’ll see mention of “v4”. However, all of these steps work for Arches v5, as well.

Upgrading your Arches installation is a complex process, as a significant backend redesign was implemented in v4. We have developed the following documentation (and the code to support it) to guide you through the process. You will be performing a combination of shell commands and basic file manipulation.

Before migrating data, you’ll need to install core Arches and create a new project. You can name your project whatever you want, but throughout this documentation we’ll refer to it as my_project. You can customize the templates and images in your project any time (before or after migrating the data). We recommend adding a Mapbox key right away so you can use the map for visual checks during the migration.

See also

Please see the main installation guide.

Before moving on, you must be able to run the Django development and view your project in a browser at http://localhost:8000.

Once you are ready, you can begin the migration process. The overall form of the process goes like this:

• Export data from existing v3 Arches installation

• Create a package (Arches-HIP users: this is already done for you)

• Place the v3 data into the Package directories

• Run commands to convert the v3 data to v4 data

• Load the package into any v4 project

Exporting Your Data From v3

You must export all of your data from v3. Before you begin, however, you’ll need to install some enhanced commands into your v3 app. This is a simple process:

1. Download and unzip arches3-export-utils-master.zip (source)

2. Copy the “management” directory into your v3 app alongside the settings.py file.

3. In your v3 environment, run python manage.py v3v4 --help to make sure the new commands have been been installed.

Warning

Be sure to backup your v3 database before beginning the export process.

Now you are ready to begin exporting your data from v3. Follow these steps:

• Export v3 Business Data
  • Export v3 Resource Relations
  • Export v3 Reference Data
  • Transfer all v3 uploaded media files

After you have all the v3 data exported, you are ready to follow the appropriate workflow for your deployment.

Migrating Your Data

The workflow you must use for the migration depends on the nature of your v3 deployment.

Arches-HIP App

If your v3 deployment of Arches was based on Arches-HIP, and you did not modify any of the graphs (beyond perhaps changing node names) you can use the Arches-HIP Workflow. If you have changed the RDM content that’s fine, it will be preserved through the migration.

• Arches-HIP Workflow
  • 1. Download the prepared v4 HIP package.
  • 2. Move your v3 data into the package.
  • 3. Convert the v3 reference data.
  • 4. Convert the v3 JSON/JSONL business data.
  • 5. Convert the v3 resource relations.
  • 6. Load the entire package.

App With Custom Graphs

If you have a v3 deployment with custom resource graphs, you’ll need to use the following workflow. Be aware, you’ll need to remake your custom resource graphs in v4 (as “Resource Models”). This is listed as Step 6 below.

• Custom App Workflow
  • 1. Create a new package
  • 2. Prepare your package.
  • 3. Move your exported v3 data into the package.
  • 4. Move the v3 resource graph _nodes.csv files from v3 into your package.
  • 5. Convert the v3 reference data.
  • 6. Build the v4 Resource Models.
  • 7. Generate and populate the node lookup files.
  • 8. Convert the v3 JSON/JSONL business data
  • 9. Write the v4 resource relations file.
  • 10. Load the entire package (optional)