Custom App Workflow#

Experienced developers should be able to use some of these steps individually to accomplish discrete tasks, but we generally recommend following this workflow as a whole.

Note

All of the commands below must be run from within your v4 project.

1. Create a new package#

python manage.py packages -o create_package -d pkg

The result should be a new package within your project named pkg:

my_project/
└─ manage.py
└─ my_project/
└─ pkg/

Now go into your project’s my_project/my_project/settings.py file and add this new line somewhere after the APP_ROOT line:

PACKAGE_DIR = os.path.join(os.path.dirname(APP_ROOT),'pkg')

Note

You can actually name your new package whatever you want, and place it wherever you want, as long as PACKAGE_DIR holds the path to it. You can even omit PACKAGE_DIR entirely if you pass --target path/to/package to all of the v3 commands below.

Finally, load this package into your project:

python manage.py packages -o load_package -s pkg -db true

Important

We recommend using the -db true flag here, which will completely erase your v4 project database and create a fresh installation. If you have already added a lot of new user logins to your v4 project, these will be lost. If you have already added settings to your project like a MapBox API key, for example, follow these steps to retain them before running the command with -db true:

In your v4 project, run python manage.py packages -o save_system_settings
Find the newly created file my_project/my_project/system_settings/System_Settings.json and move it into my_project/pkg/system_settings.
When you do run the load package command, say “y” to the prompt about overwriting project settings (they will be imported from this new settings file).

Before moving on you should be able to view your project in a browser and login with the default admin/admin credentials.

2. Prepare your package.#

python manage.py v3 start-migration

This will create some new directories and content in your package:

pkg/
  └─ reference_data/
      └─ v3topconcept_lookup.json
  └─ v3data/
      └─ business_data/
      └─ graph_data/
      └─ reference_data/

3. Move your exported v3 data into the package.#

Move v3resources-all-<date>.json from Export v3 Business Data into v3data/business_data. This file name could be slightly different for you (or you may have multiple files) based on how you ran the v3 export.

Move v3relations-all-<date>.csv from Export v3 Resource Relations into v3data/business_data.

Move v3scheme-arches-<date>.xml from Export v3 Reference Data into v3data/reference_data.

4. Move the v3 resource graph _nodes.csv files from v3 into your package.#

In your Arches v3 deployment, you should be able to find these files in your original source_data/resource_graphs directory, whose contents should be a _edges.csv and _nodes.csv for every resource graph in your database. We only want the _nodes.csv files.

Move the _nodes.csv files into v3data/graph_data.

After completing steps 3 and 4, your v4 package should look like this:

pkg/
  └─ v3data/
      └─ business_data/
          └─ v3resources-all-<date>.json
          └─ v3relations-all-<date>.csv
      └─ graph_data/
          └─ RESOURCE_GRAPH_NAME.Exx_nodes.csv
          └─ etc.
      └─ reference_data/
          └─ v3scheme-arches-<date>.xml
      └─ rm_configs.json

5. Convert the v3 reference data.#

See Arches-HIP workflow Step 3, and return to this page when you’ve finished.

6. Build the v4 Resource Models.#

Now that the v3 reference data has been converted and loaded, you are ready to create the v4 Resource Models. This migration process does not attempt to create them based on your old v3 graphs. There are a number of reasons for this, but most simply, v4 graphs have different constraints and support different datatypes and structures than those in v3. In other words, your v4 database will be better off with graphs that have been created natively, not translated from v3.

Generally, we would expect the v4 graphs to look like their v3 analogs, but we have built in quite a bit of wiggle room:

The graph names can differ

The node names can differ

The graph structure can differ (though maintaining the same general branching structure is advisable)

However, there must still be a one-to-one relationship between v3 and v4 graphs and their nodes.

When it comes to node datatypes, the translation from v3 to v4 is pretty straight-forward.

Datatype Translation – v3 to v4#
v3 businesstable	v4 datatype
`strings`	`string`
`dates`	`date` or `edtf`
`geometries`	`geojson-feature-collection`
`domains`	`concept` - if single value per v3 branch
`domains`	`concept-list` - if multiple values per v3 branch were allowed

Important

When you set a v4 node to concept or concept-list, you will need to select which collection to use. This is why it’s best to have migrated and loaded your RDM scheme (step 5 above) before making the Resource Models.

7. Generate and populate the node lookup files.#

Begin by running:

python manage.py v3 generate-rm-configs

which will create v3data/rm_configs.json. This file will be used to link the name of your v4 Resource Models with the names of their corresponding v3 graphs, as well as point to the files that link each node. Initially its content will look like:

{
    "Activity": {
        "v3_entitytypeid": "<fill out manually>",
        "v3_nodes_csv": "run 'python manage.py v3 generate-lookups",
        "v3_v4_node_lookup": "run 'python manage.py v3 generate-lookups"
    }
}

where "Activity" is the name of a v4 Resource Model. As the file says, you must now fill out the v3_entitytypeid value for all items. Typically, this will look something like "ACTIVITY.E7"–upper-case with a CRM class appended to it.

Now, also as the file says, run:

python manage.py v3 generate-lookups

and you’ll see the rest of the values get filled out.

There will now be more CSV files in the v3data/graph_data directory. There is one per v3 graph, and they are used to match the names of v3 node names (column one), with v4 node names (column two). All of the v3 nodes will be listed for you, but you have to fill out the v4 node names manually, using your new Resource Models for reference. A portion of a filled out file could look like:

ACTIVITY.E7_v4_lookup.csv#
v3_node	v4_node
ACTIVITY_TYPE.E55	Activity Type
ADDRESS_TYPE.E55	Address Type
etc…	etc…

Finally, you can use:

python manage.py v3 test-lookups

to check your work. Once this test passes, you can move on.

8. Convert the v3 JSON/JSONL business data#

See Arches-HIP workflow Step 4. (You can continue using that workflow until you are finished with the migration.)

9. Write the v4 resource relations file.#

See Arches-HIP workflow Step 5. (You can continue using that workflow until you are finished with the migration.)

10. Load the entire package (optional)#

See Arches-HIP workflow Step 6.