¶ Property Graph Injector tutorial
The document presents a new thingin the future injector allowing to register and update avatars in the platform.
¶ Why another injector?
While there is several others injectors available on the platform, The main purposes of this injector are:
- Ability to register in thingin property graphs. That is, graphs that may contains properties on nodes and/or relations.
- Ability to make optional the use of ontologies or at least reduce their usage to describe graphs of avatars, .
In fact, we noticed that in certain use cases, on one hand properties on relations may be required and on the other hand, finding or even defining the ontology to describe avatars is difficult and cumbersome. The idea behind this new injector is to ease the usage and adoption of the platform.
¶ How does it work?
The injector takes as input:
- One or several data sources: containing the avatars to register in the platform. Currently the injector accept sources in JSON and CSV data format.
- A mapping file defined in JSON format. The mapping describing the structure of the property graph of avatars contained in the data source.
Using the mapping the injector is able to translate the property graph structure into an RDF format that can be processed by Thingin. To achieve it, it relies on :
- SPARQL-Generate, a library that can convert raw data in various format into RDF.
- NGSI-LD metamodel: In fact in order to translate the property graph in to RDF, we need semantic concepts to describe such structure into RDF. The metamodel of NGSI-LD, an ETSI standard, offers some means to describe achieve it. Note that in it current state, due to some constraints on the thingin implementation and requirements, the injector is not entirelly based on NGSI-LD. Some concepts are still defined in-house.
¶ How to write a Mapping
The main structure of the JSON Mapping file is composed of three JSON arrays (sources, prefixes and entities)
{
"mapping": {
"sources": [],
"prefixes": [],
"entities": []
}
}
¶ Sources
A source object describes a data source file using the following elements :
- Path: defines the path file of the source
- Format: defines the format of the data file
- sourceId: An identifier, unique whithin the mapping, used to assign the source to its corresponding entity
- Root: Defines the root element from which starts the parsing of the file (only for JSON Files)
- Separator: Defines the separator character for CSV Files
{
"sourceId": "2",
"root": "$[*]",
"format": "json",
"path": "/path_to_json_file"
}
¶ Prefixes
- Prefixes allow users to define a short version of
URIs
{
"longUri": "http://orange-labs.fr/fog/ont/iot.owl#",
"shortUri": "iot"
}
¶ Entities
An entity object defines the model of the avatars that are going to be injected in Thing'in graph using the following elements :
- SourceId: Refers to the data file to which this entity is assigned
- UriTemplate: Defines a template that generates unique URIs for each entity
- labels: Tags allowing users to describe their data
- Relations: Describes the relations of the entity
- Properties: Describes the properties or attributes of the entity
{
"sourceId" : "1",
"labels" : [
"multiLineString-example"
],
"uriTemplate": {
"uri": "http://example.org/thingin/multiLineString",
"idRoot": "id"
},
"properties" : [],
"relations" : []
}
¶ Relations
A relation is described using a label and the URI of the destination entity
{
"label": "IsInEcoleMappingRelation",
"dest": {
"uri": "http://salma.org/Ecole",
"idRoot": "ecole.id"
},
"properties": []
}
¶ Properties
A property is described using the following elements :
- Label:a label for the property. Labels that are not already registered in Thingin will be registered by the injector
- Root : The path of the property in the data file
- ValueType : The type of the property
The JSON Mapping supports basic data types as well as GeoTypes :
- String
- Integer
- Long
- Float
- Double
- Boolean
- Date
- Time
- DateTime
- Point
- MultiPoint
- Polygon
- MultiPolygon
- LineString
- MultiLineString
Details on GeoTypes may be found at https://geojson.org/.
¶ Basic properties type
Here is an example of a String property
{
"root": "ecole.filiere.name",
"label": "FiliereName",
"valueType": "String"
}
¶ GeoType Property
Here is an example of a Point property
{
"label" : "geo:coordinates",
"valueType": "point",
"x-root": "lineStrings[0].points[0].x",
"y-root": "lineStrings[0].points[0].y"
}
Or in Thing'in geojson format
{
"label" : "http://www.opengis.net/gml/pos",
"valueType": "point",
"x-root": "Longitude",
"y-root": "Latitude"
}
Here is an example of a Polygon property
{
"label" : "polygon",
"valueType": "polygon",
"points": [
{
"x-root": "points[0].x",
"y-root": "points[0].y"
},
{
"x-root": "points[1].x",
"y-root": "points[1].y"
}
]
}
¶ Server mode via the portail
In order to use the PG-injector via the Thingin portal, the user needs to be authenticated and have the roles provider
The injector page can be accessed through the menu provide>generic injector
The generic-injector page display the list of all injectors of the current user.
Click on deploy a new injector to create a new injector. It will display a form to enter information to create the injector. Click on deploy button to create the injector
The injectors manager will instantiate the injector in a PAAS. Once the injector is ready, the user will be able to add a mapping and data source files.
The following buttons can be used to manage the injector:
- set mapping: to upload a mapping file to the injector
- data: to upload data source files
- start now: when the injector is provided with mapping and data files, we can start the injection process by clicking this button
- history: to see the history of states of the injector
- logs: to see log send from the injector
- console: to receive in near-realtime log from injector. It requires to register an endpoint to receive the log. It not implemented yet
- delete: to delete the injector. It will won't affect avatar previously created by the injector. Only the instance of the injector will be deleted from the PAAS
When no activity is observe on an injector. It will be disable to avoir unnecessary resource consumption in the PAAS. The injector will have a status REMOVED. In order to reactivate the injector, one should click on the redeploy button.
¶ CLI-mode execution using source code
Having access to the source code, it is possible to use the injector in command line.
¶ Requirements
- Maven 3.8.2
- Java 15
¶ Compilation
To compile and test ;
mvn test
Depending on the execution environment, the following environment variables have to be defined.
INJECTOR_PROXY_ENABLED=true
INJECTOR_PROXY_HTTP_HOST=<TBD>
INJECTOR_PROXY_HTTPS_HOST=<TBD>
APP_STORAGE_FS_ROOT=data/ngsild-injector/local
APP_THINGIN_TOKEN=<TBD>
Hereafter an example of a compilation and launching of the injector once the environment variables are set:
mvn clean package -DskipTests=true
java -jar target/ngsild-injector-0.0.1-SNAPSHOT.jar cli-mode -m examples/mapping-shops.json -f examples/shops.csv
Another example using several data sources files with relations created between avatars from those files
mvn clean package -DskipTests=true
java -jar target/ngsild-injector-0.0.1-SNAPSHOT.jar cli-mode -m examples/mapping-ecoles-etudiants.json -f examples/ecoles.json,examples/etudiants.json
¶ Complete example of data and mapping
Here a example of a complete mapping.The associated data can be found in the next section
{
"mapping": {
"sources": [
{
"sourceId": "1",
"format" : "csv",
"path" : "shops.csv",
"separator": ","
}
],
"prefixes": [
{
"shortUri": "ont",
"longUri": "http://orange-labs.fr/fog/ont/iot.owl#"
},
{
"shortUri": "geo",
"longUri": "http://orange-labs.fr/thingin/geo"
}
],
"entities" : [
{
"sourceId": "1",
"labels" : [ "example:Shops" ],
"uriTemplate": {
"uri" : "http://pg-injectors-example.org/Shops",
"idRoot": "ID"
},
"properties": [
{
"root": "Name",
"label" : "example:name" ,
"valueType": "String"
},
{
"root": "Description",
"label" : "example:description" ,
"valueType": "String"
},
{
"label" : "geo:coordinates",
"valueType": "Point",
"x-root": "Longitude",
"y-root": "Latitude"
}
]
}
]
}
}
Heare after a data source in CSV format
ID,Name,Longitude,Latitude,Location,Description
1,Décathlon Easy Rennes,-1.69074096595433,48.1365100667282,Rennes,"Magasin d'articles de sports, 6 Rue René Collin, 02 99 33 57 24"
2,Giant Rennes,-1.68425903310585,48.1337239005906,Rennes,"Magasin de vélos, 15 Rue Bahon Rault, 02 99 63 92 68"
3,Cyclable,-1.68475341763705,48.1232744341933,Rennes,"Magasin de vélos, 206 Rue Saint-Malo, 02 99 13 18 02"
4,L'Echoppe du Deux Roues,-1.68140258777992,48.1152800736173,Rennes,"Magasin de vélos, 16 Rue Saint-Louis, 02 23 46 30 01"
5,Bycyclette,-1.68607177673616,48.1097786443101,Rennes,"Magasin de vélos, 3 Quai de la Prévalaye, 02 23 42 10 83"
6,Deux Y,-1.68568725537591,48.1097052880127,Rennes,"Magasin d'articles de sports, 3 Quai de la Prévalaye, 02 23 42 10 83"
7,Culture Vélo Rennes,-1.70864868010651,48.1040198513725,Rennes,"Magasin de vélos, 106 Rue Eugène Pottier, 02 99 01 09 20"
8,Cycles Guédard,-1.67634887707053,48.1037630745792,Rennes,"Magasin de vélos, 13 Boulevard de Beaumont, 02 99 30 43 78"
9,EBIKES City Rennes,-1.67827148437111,48.1068076275357,Rennes,"Magasin de vélos, 9 Rue Émile Souvestre, 02 23 44 96 25"
10,Leconte Cycles & Scooters,-1.6776672363661,48.1072844693562,Rennes,"Concessionnaire de scooters, 2 Rue d'Isly, 02 99 30 89 02"
11,Cyclable,-1.66497802800715,48.1105488799858,Rennes,"Magasin de vélos, 65 Avenue Aristide Briand, 02 99 38 46 08"
12,Bicycle Times L'Atelier Vélo,-1.65135498223676,48.1064775041142,Rennes,"Magasin de réparation de vélos, 48 Boulevard Villebois Mareuil, 06 32 70 39 18"
13,MBK Cotin SARL,-1.65827636776903,48.1003881780504,Rennes,"Concessionnaire de scooters, 29 Rue de Vern, 02 99 50 56 25"
14,Cycl Shop,-1.62822876144007,48.1057438886084,Rennes,"Concessionnaire de véhicules à moteur, 13 Rue de Bray, 02 99 83 21 22"
15,Cycles Béon Sport,-1.61372680820662,48.1076512674233,Rennes,"Magasin de vélos, 44 Rue de Bray, 02 99 22 79 15"
16,CycleBox,-1.62769372782374,48.08911148458,Chantepie,"Magasin de vélos, 5 Rue du Moulin, 02 99 32 45 47"