Datatypes#
A DataType defines a type of business data. DataTypes are associated with Nodes and Widgets. When you are designing your Cards, the Widgets with the same DataType as the Node you are collecting data for will be available. In your Branches, each Node with a DataType will honor the DataType configuration you specify when you create it.
The simplest (non-configurable, non-searchable) DataTypes consist of a single Python file. If you want to provide Node-specific configuration to your DataType (such as whether to expose a Node with that DataType to Advanced Search or how the data is rendered), you’ll also develop a UI component comprising a Django template and JavaScript file.
In your Project, these files must be placed accordingly:
Optional Configuration Component:
/myproject/myproject/media/js/views/components/datatypes/sample_datatype.js/myproject/myproject/templates/views/components/datatypes/sample_datatype.htmDataType File:
/myproject/myproject/datatypes/sample_datatype.py
To begin, let’s examine the sample-datatype included with Arches:
1from arches.app.datatypes.base import BaseDataType
2from arches.app.models import models
3from arches.app.models.system_settings import settings
4
5sample_widget = models.Widget.objects.get(name="sample-widget")
6
7details = {
8 "datatype": "sample-datatype",
9 "iconclass": "fa fa-file-code-o",
10 "modulename": "datatypes.py",
11 "classname": "SampleDataType",
12 "defaultwidget": sample_widget,
13 "defaultconfig": {"placeholder_text": ""},
14 "configcomponent": "views/components/datatypes/sample-datatype",
15 "configname": "sample-datatype-config",
16 "isgeometric": False,
17 "issearchable": False,
18}
19
20
21class SampleDataType(BaseDataType):
22 def validate(self, value, row_number=None, source=None):
23 errors = []
24 try:
25 value.upper()
26 except:
27 errors.append(
28 {
29 "type": "ERROR",
30 "message": "datatype: {0} value: {1} {2} {3} - {4}. {5}".format(
31 self.datatype_model.datatype, value, row_number, source, "this is not a string", "This data was not imported.",
32 ),
33 }
34 )
35 return errors
36
37 def append_to_document(self, document, nodevalue, nodeid, tile):
38 document["strings"].append({"string": nodevalue, "nodegroup_id": tile.nodegroup_id})
39
40 def transform_export_values(self, value, *args, **kwargs):
41 if value != None:
42 return value.encode("utf8")
43
44 def get_search_terms(self, nodevalue, nodeid=None):
45 terms = []
46 if nodevalue is not None:
47 if settings.WORDS_PER_SEARCH_TERM == None or (len(nodevalue.split(" ")) < settings.WORDS_PER_SEARCH_TERM):
48 terms.append(nodevalue)
49 return terms
50
51 def append_search_filters(self, value, node, query, request):
52 try:
53 if value["val"] != "":
54 match_type = "phrase_prefix" if "~" in value["op"] else "phrase"
55 match_query = Match(field="tiles.data.%s" % (str(node.pk)), query=value["val"], type=match_type,)
56 if "!" in value["op"]:
57 query.must_not(match_query)
58 query.filter(Exists(field="tiles.data.%s" % (str(node.pk))))
59 else:
60 query.must(match_query)
61 except KeyError:
62 pass
Writing Your DataType#
Your DataType needs, at minimum, to implement the validate
method. You’re also likely to implement the
transform_import_values or transform_export_values
methods. Depending on whether your DataType is spatial, you may need
to implement some other methods as well. If you want to expose Nodes
of your DataType to Advanced Search, you’ll also need to implement the
append_search_filters method.
You can get a pretty good idea of what methods you need to implement
by looking at the BaseDataType class in the Arches source code
located at arches/app/datatypes/base.py and below:
1import json
2from django.core.urlresolvers import reverse
3from arches.app.models import models
4
5class BaseDataType(object):
6
7 def __init__(self, model=None):
8 self.datatype_model = model
9
10 def validate(self, value, row_number=None, source=None):
11 return []
12
13 def append_to_document(self, document, nodevalue, nodeid, tile):
14 """
15 Assigns a given node value to the corresponding key in a document in
16 in preparation to index the document
17 """
18 pass
19
20 def after_update_all(self):
21 """
22 Refreshes mv_geojson_geoms materialized view after save.
23 """
24 pass
25
26 def transform_import_values(self, value, nodeid):
27 """
28 Transforms values from probably string/wkt representation to specified
29 datatype in arches
30 """
31 return value
32
33 def transform_export_values(self, value, *args, **kwargs):
34 """
35 Transforms values from probably string/wkt representation to specified
36 datatype in arches
37 """
38 return value
39
40 def get_bounds(self, tile, node):
41 """
42 Gets the bounds of a geometry if the datatype is spatial
43 """
44 return None
45
46 def get_layer_config(self, node=None):
47 """
48 Gets the layer config to generate a map layer (use if spatial)
49 """
50 return None
51
52 def should_cache(self, node=None):
53 """
54 Tells the system if the tileserver should cache for a given node
55 """
56 return False
57
58 def should_manage_cache(self, node=None):
59 """
60 Tells the system if the tileserver should clear cache on edits for a
61 given node
62 """
63 return False
64
65 def get_map_layer(self, node=None):
66 """
67 Gets the array of map layers to add to the map for a given node
68 should be a dictionary including (as in map_layers table):
69 nodeid, name, layerdefinitions, isoverlay, icon
70 """
71 return None
72
73 def clean(self, tile, nodeid):
74 """
75 Converts '' values to null when saving a tile.
76 """
77 if tile.data[nodeid] == '':
78 tile.data[nodeid] = None
79
80 def get_map_source(self, node=None, preview=False):
81 """
82 Gets the map source definition to add to the map for a given node
83 should be a dictionary including (as in map_sources table):
84 name, source (json)
85 """
86 tileserver_url = reverse('tileserver')
87 if node is None:
88 return None
89 source_config = {
90 "type": "vector",
91 "tiles": ["%s/%s/{z}/{x}/{y}.pbf" % (tileserver_url, node.nodeid)]
92 }
93 count = None
94 if preview == True:
95 count = models.TileModel.objects.filter(data__has_key=str(node.nodeid)).count()
96 if count == 0:
97 source_config = {
98 "type": "geojson",
99 "data": {
100 "type": "FeatureCollection",
101 "features": [
102 {
103 "type": "Feature",
104 "properties": {
105 "total": 1
106 },
107 "geometry": {
108 "type": "Point",
109 "coordinates": [
110 -122.4810791015625,
111 37.93553306183642
112 ]
113 }
114 },
115 {
116 "type": "Feature",
117 "properties": {
118 "total": 100
119 },
120 "geometry": {
121 "type": "Point",
122 "coordinates": [
123 -58.30078125,
124 -18.075412438417395
125 ]
126 }
127 },
128 {
129 "type": "Feature",
130 "properties": {
131 "total": 1
132 },
133 "geometry": {
134 "type": "LineString",
135 "coordinates": [
136 [
137 -179.82421875,
138 44.213709909702054
139 ],
140 [
141 -154.16015625,
142 32.69486597787505
143 ],
144 [
145 -171.5625,
146 18.812717856407776
147 ],
148 [
149 -145.72265625,
150 2.986927393334876
151 ],
152 [
153 -158.37890625,
154 -30.145127183376115
155 ]
156 ]
157 }
158 },
159 {
160 "type": "Feature",
161 "properties": {
162 "total": 1
163 },
164 "geometry": {
165 "type": "Polygon",
166 "coordinates": [
167 [
168 [
169 -50.9765625,
170 22.59372606392931
171 ],
172 [
173 -23.37890625,
174 22.59372606392931
175 ],
176 [
177 -23.37890625,
178 42.94033923363181
179 ],
180 [
181 -50.9765625,
182 42.94033923363181
183 ],
184 [
185 -50.9765625,
186 22.59372606392931
187 ]
188 ]
189 ]
190 }
191 },
192 {
193 "type": "Feature",
194 "properties": {
195 "total": 1
196 },
197 "geometry": {
198 "type": "Polygon",
199 "coordinates": [
200 [
201 [
202 -27.59765625,
203 -14.434680215297268
204 ],
205 [
206 -24.43359375,
207 -32.10118973232094
208 ],
209 [
210 0.87890625,
211 -31.653381399663985
212 ],
213 [
214 2.28515625,
215 -12.554563528593656
216 ],
217 [
218 -14.23828125,
219 -0.3515602939922709
220 ],
221 [
222 -27.59765625,
223 -14.434680215297268
224 ]
225 ]
226 ]
227 }
228 }
229 ]
230 }
231 }
232 return {
233 "nodeid": node.nodeid,
234 "name": "resources-%s" % node.nodeid,
235 "source": json.dumps(source_config),
236 "count": count
237 }
238
239 def get_pref_label(self, nodevalue):
240 """
241 Gets the prefLabel of a concept value
242 """
243 return None
244
245 def get_display_value(self, tile, node):
246 """
247 Returns a list of concept values for a given node
248 """
249 return unicode(tile.data[str(node.nodeid)])
250
251 def get_search_terms(self, nodevalue, nodeid=None):
252 """
253 Returns a nodevalue if it qualifies as a search term
254 """
255 return []
256
257 def append_search_filters(self, value, node, query, request):
258 """
259 Allows for modification of an elasticsearch bool query for use in
260 advanced search
261 """
262 pass
263
264 def handle_request(self, current_tile, request, node):
265 """
266 Updates files
267 """
268 pass
the validate method#
Here, you write logic that the Tile model will use to accept or reject a Node’s data before saving. This is the core implementation of what your DataType is and is not.
The validate method returns an array of errors. If the array is
empty, the data is considered valid. You can populate the errors array
with any number of dictionaries with a type key and a message
key. The value for type will generally be ERROR, but you can
provide other kinds of messages.
the append_search_filters method#
In this method, you’ll create an ElasticSearch query Nodes matching this datatype based on input from the user in the Advanced Search screen. (You design this input form in your DataType’s front-end component.)
Arches has its own ElasticSearch query DSL builder class.
You’ll want to review that code for an idea of what to do. The search
view passes your DataType a Bool() query from this class, which you
call directly. You can invoke its must, filter, should, or
must-not methods and pass complex queries you build with the DSL
builder’s Match class or similar. You’ll execute this search
directly in your append_search_filters method.
In-depth documentation of this part is planned, but for now, look at the core datatypes located in Arches’ source code for examples of the approaches you can take here.
Note
If you’re an accomplished Django developer, it should also be
possible to use Elastic’s own Python DSL builder in your Project
to build the complex search logic you’ll pass to Arches’ Bool()
search, but this has not been tested.
Configuring your DataType#
You’ll need to populate the details dictionary to configure your
new DataType.
details = {
"datatype": "sample-datatype",
"iconclass": "fa fa-file-code-o",
"modulename": "datatypes.py",
"classname": "SampleDataType",
"defaultwidget": sample_widget,
"defaultconfig": {"placeholder_text": ""},
"configcomponent": "views/components/datatypes/sample-datatype",
"configname": "sample-datatype-config",
"isgeometric": False,
"issearchable": False,
- datatype:
Required The name of your datatype. The convention in Arches is to use kebab-case here.
- iconclass:
Required The FontAwesome icon class your DataType should use. Browse them here.
- modulename:
Required This should always be set to
datatypes.pyunless you’ve developed your own Python module to hold your many DataTypes, in which case you’ll know what to put here.- classname:
Required The name of the Python class implementing your datatype, located in your DataType’s Python file below these details.
- defaultwidget:
Required The default Widget to be used for this DataType.
- defaultconfig:
Optional You can provide user-defined default configuration here.
- configcomponent:
Optional If you develop a configuration component, put the fully-qualified name of the view here. Example:
views/components/datatypes/sample-datatype- configname:
Optional The name of the Knockout component you have registered in your UI component’s JavaScript file.
- isgeometric:
Required Used by the Arches UI to determine whether to create a Map Layer based on the DataType, and also for caching. If you’re developing such a DataType, set this to True.
- issearchable:
Optional Determines if the datatype participates in advanced search. The default is false.
Important
configcomponent and configname are required together.
Developing the Configuration Component#
Your component JavaScript file should register a Knockout component
with your DataType’s configname. This component should be an
object with two keys: viewModel, and template
The value for viewModel should be a function where you put the
logic for your template. You’ll be setting up Knockout observable and
computed values tied to any form elements you’ve developed to collect
Advanced Search or Node-level configuration information from the user.
The value for template should be another object with the key
require, and the value should be
text!datatype-config-templates/<your-datatype-name>. Arches will
know what to do with this – it comes from the value you supplied in
your Python file’s details dictionary for configcomponent.
Pulling it all together, here’s the JavaScript portion of Arches’
date DataType.
define(['knockout', 'datatype-config-templates/date.htm'], function (ko, dateTemplate) {
var name = 'date-datatype-config';
ko.components.register(name, {
viewModel: function(params) {
var self = this;
this.search = params.search;
if (this.search) {
var filter = params.filterValue();
this.viewMode = 'days';
this.op = ko.observable(filter.op || '');
this.searchValue = ko.observable(filter.val || '');
this.filterValue = ko.computed(function () {
return {
op: self.op(),
val: self.searchValue()
}
}).extend({ throttle: 750 });
params.filterValue(this.filterValue());
this.filterValue.subscribe(function (val) {
params.filterValue(val);
});
}
},
template: dateTemplate,
});
return name;
});
Advanced Search Rendering#
If you’re supporting Advanced Search functionality for Nodes with your
DataType, your Django template will include a search block,
conditionally rendered by Knockout.js if the search view is
active. Here’s the one from the boolean datatype:
<!-- ko if: $data.search -->
{% block search %}
<div class="col-sm-12">
<select class="resources" data-bind="value: searchValue, chosen: {width: '100%', disable_search_threshold: 15}, options: [{id:'t', name:trueLabel}, {id:'f', name:falseLabel}], optionsText: 'name', optionsValue: 'id'">
</select>
</div>
{% endblock search %}
<!-- /ko -->
Note the <!-- ko if: $data.search --> directive opening and
closing the search block. This is not an HTML comment – it’s
Knockout.js-flavored markup for the conditional rendering.
Arches’ built-in date DataType does not use the Django template
block directive, but only implements advanced search, and contains
a more sophisticated example of the component logic needed:
{% load i18n %}
<!-- ko if: $data.search -->
<div class="col-md-4 col-lg-3">
<select class="resources" tabindex="-1" style="display: none;" data-bind="value: op, chosen: {width: '100%', disable_search_threshold: 15}">
<option value="eq"> = </option>
<option value="gt"> > </option>
<option value="lt"> < </option>
<option value="gte"> >= </option>
<option value="lte"> <= </option>
</select>
</div>
<div class="col-md-8 col-lg-9">
<input type="" placeholder="{% trans "Date" %}" class="form-control input-md widget-input" data-bind="value: searchValue, datepicker: {format: 'YYYY-MM-DD', viewMode: viewMode, minDate: false, maxDate: false}">
</div>
<!-- /ko -->
Node-specific Configuration#
This section of your template should be enclosed in Knockout-flavored
markup something like: <!-- ko if: $data.graph -->, and in your
Knockout function you should follow the convention and end up with
something like if (this.graph) {
Here, you put form elements corresponding to any configuration you’ve
implemented in your DataType. These should correspond to keys in your
DataType’s defaultconfig.
Arches’ boolean DataType has the following defaultconfig:
{'falseLabel': 'No', 'trueLabel': 'Yes'}
You can see the corresponding data bindings in the Django template:
<!-- ko if: $data.graph -->
<div class="control-label">
{% trans "Label 'True'" %}
</div>
<div class="col-xs-12 pad-no crud-widget-container">
<input type="" id="" class="form-control input-md widget-input" data-bind="value: trueLabel, valueUpdate: 'keyup'">
</div>
<div class="control-label">
{% trans "Label 'False'" %}
</div>
<div class="col-xs-12 pad-no crud-widget-container">
<input type="" id="" class="form-control input-md widget-input" data-bind="value: falseLabel, valueUpdate: 'keyup'">
</div>
<!-- /ko -->
And finally, here is the boolean DataType’s JavaScript file in its entirety:
define(['knockout', 'datatype-config-templates/boolean.htm'], function (ko, booleanTemplate) {
var name = 'boolean-datatype-config';
ko.components.register(name, {
viewModel: function(params) {
var self = this;
this.search = params.search;
this.graph = params.graph;
this.trueLabel = params.config ? params.config.trueLabel : params.node.config.trueLabel;
this.falseLabel = params.config ? params.config.falseLabel : params.node.config.falseLabel;
if (this.search) {
var filter = params.filterValue();
this.searchValue = ko.observable(filter.val || '');
this.filterValue = ko.computed(function () {
return {
val: self.searchValue()
}
});
params.filterValue(this.filterValue());
this.filterValue.subscribe(function (val) {
params.filterValue(val);
});
}
},
template: booleanTemplate,
});
return name;
});
Registering your DataType#
These commands are identical to working with Widgets, but you use the
word datatype instead. Please refer to Command Line Reference - Widget Commands.