Navigation

Insertion of new observations

You can store new observations in istSOS using various methods:

  • XML with the OGC SOS request insert oservation
  • istSOS CSV format with the Python script csv2istsos.py provided with the software
  • Proprietary plain text files with the Python script raw2istsos.py provided with the software

Loading data with OGC-SOS InsertObservation request

Even if you can use the csv2istsos.py script to facilitate the data loading, users may also use the SOS insertObservation request directly.

To execute the XML request from the interface:

  1. Open the requests test page: http://localhost/istsos/admin/requests
  2. Select the desired service instance
  3. Choose the “POST” option
  4. Paste into the field the InsertObservation xml
  5. Press “Send”

Note

Pay attention to the AssignedSensorId parameter: this according to the standard is returned by the system only when the sensor is registered. To access it, you can use administration interface, looking at the procedure metadata details.

Example

For example, a valid request for loading observations to a procedure named LOCARNO which is observing rainfall and temperature for the time inteval 2014-06-03T15:08:00Z/2014-06-03T15:48:00Z and specifying the respective qualityIndex for each measure, the request is:

<?xml version="1.0" encoding="UTF-8"?>
<sos:InsertObservation
  xmlns:gml="http://www.opengis.net/gml"
  xmlns:om="http://www.opengis.net/om/1.0"
  xmlns:sos="http://www.opengis.net/sos/1.0"
  xmlns:swe="http://www.opengis.net/swe"
  xmlns:xlink="http://www.w3.org/1999/xlink"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://schemas.opengis.net/sos/1.0.0/sosAll.xsd"
  service="SOS" version="1.0.0">
  <sos:AssignedSensorId>xxxxxxxxxxxxxxxxxxxxxxxxxxx</sos:AssignedSensorId>
  <om:Observation>
    <om:procedure xlink:href="urn:ogc:def:procedure:x-istsos:1.0:LOCARNO"/>
    <om:samplingTime>
        <gml:TimePeriod>
            <gml:beginPosition>2014-06-03T15:08:00Z</gml:beginPosition>
            <gml:endPosition>2014-06-03T15:48:00Z</gml:endPosition>
        </gml:TimePeriod>
    </om:samplingTime>
    <om:observedProperty>
        <swe:CompositePhenomenon dimension="5">
            <swe:component xlink:href="urn:ogc:def:parameter:x-istsos:1.0:time:iso8601"/>
            <swe:component xlink:href="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:rainfall"/>
            <swe:component
              xlink:href="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:rainfall:qualityIndex"/>
            <swe:component
              xlink:href="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature"/>
            <swe:component
              xlink:href="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature:qualityIndex"/>
        </swe:CompositePhenomenon>
    </om:observedProperty>
    <om:featureOfInterest xlink:href="urn:ogc:def:feature:x-istsos:1.0:Point:LOCARNO"/>
    <om:result>
        <swe:DataArray>
            <swe:elementCount>
                <swe:value>5</swe:value>
            </swe:elementCount>
            <swe:elementType name="SimpleDataArray">
                <swe:DataRecord definition="urn:ogc:def:dataType:x-istsos:1.0:timeSeries">
                    <swe:field name="Time">
                        <swe:Time definition="urn:ogc:def:parameter:x-istsos:1.0:time:iso8601"/>
                    </swe:field>
                    <swe:field name="air-rainfall">
                        <swe:Quantity definition="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:rainfall">
                            <swe:uom code="mm"/>
                        </swe:Quantity>
                    </swe:field>
                    <swe:field name="air-rainfall:qualityIndex">
                        <swe:Quantity definition="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:rainfall:qualityIndex">
                            <swe:uom code="-"/>
                        </swe:Quantity>
                    </swe:field>
                    <swe:field name="air-temperature">
                        <swe:Quantity definition="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature">
                            <swe:uom code="\xc2\xb0C"/>
                        </swe:Quantity>
                    </swe:field>
                    <swe:field name="air-temperature:qualityIndex">
                        <swe:Quantity definition="urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature:qualityIndex">
                            <swe:uom code="-"/>
                        </swe:Quantity>
                    </swe:field>
                </swe:DataRecord>
            </swe:elementType>
            <swe:encoding>
                <swe:TextBlock blockSeparator="@" decimalSeparator="." tokenSeparator=","/>
            </swe:encoding>
                <swe:values>
                    2014-06-03T14:10:00+0200,0.000000,200,20.000000,200@
                    2014-06-03T14:20:00+0200,0.000000,200,20.100000,200@
                    2014-06-03T14:30:00+0200,0.000000,200,20.200000,200@
                    2014-06-03T14:40:00+0200,0.000000,200,20.500000,200@
                    2014-06-03T14:50:00+0200,0.000000,200,20.500000,200@
                    2014-06-03T15:00:00+0200,0.000000,200,20.400000,200@
                    2014-06-03T15:10:00+0200,0.000000,200,20.400000,200@
                    2014-06-03T15:20:00+0200,0.100000,200,19.600000,200@
                    2014-06-03T15:30:00+0200,0.100000,200,19.100000,200@
                    2014-06-03T15:40:00+0200,0.000000,200,19.000000,200@
                    2014-06-03T15:50:00+0200,0.000000,200,20.600000,200
                </swe:values>
        </swe:DataArray>
    </om:result>
  </om:Observation>
</sos:InsertObservation>

Loading text/csv;subtype=istSOS data with csv2istsos.py

Using this script you should prepare text files with sensor data formatted according to text/csv;subtype=istSOS. This format is a CSV represented by a header as the first line containing the URI names of the observed properties, the following lines contains the data.

Example of a text/csv;subtype=istSOS

File name: PROCEDURENAME_YYYYMMDDhhmmss.dat

urn:ogc:def:parameter:x-istsos:1.0:time:iso8601,urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature
2013-01-01T00:10:00.000000+0100,0.446000
2013-01-01T00:20:00.000000+0100,0.862000
2013-01-01T00:30:00.000000+0100,0.932000
2013-01-01T00:40:00.000000+0100,0.384000

Note

Pay attention to the file name: there is a timestamp (YYYYMMDDhhmmss GMT+0:00). This parameter is used to force the endPosition in the sampling time of a procedure. This is particularly important when the procedure is an irregular time series.

Think of tipping bucket rain gauge, when there is no rain no data are sent. But updating the endPosition we will be able to know that the sensor is working and that there is no rain, instead of thinking that the sensor is not transmitting or that it is broken.

To load the prepared CSV you should run the csv2istsos.py command which is under the script folder of your installation location (e.g.: /usr/local/istsos)

Note

The “csv2istsos.py“ file, is a python script that makes use of the WA-REST features of istSOS to insert observations.

python scripts/csv2istsos.py --help

usage: csv2istsos.py [-h] [-v] [-t] -p procedures [procedures ...]
                     [-q quality index] [-u url] -s service
                     -w working directory [-e file extension]
                     [-usr user name] [-pwd password]

Import data from a csv file.

optional arguments:
  -h, --help            Show this help message and exit
  -v, --verbose         Activate verbose debug
  -t, --test            Use to test the command, deactivating the insert
                        observation operations.
  -p procedures [procedures ...]
                        List of procedures to be aggregated.
  -q quality index      The quality index to set for all the measures of
                        the CSV file, if not set into the CSV.
                        (default: 100).
  -u url                IstSOS Server address IP (or domain name) used for
                        all request. (default: http://localhost:80/istsos).
  -s service            The name of the service instance.
  -w working directory  Working directory where the csv files are located.
  -e file extension     Extension of the CSV file. (default: .dat)
  -usr user name
  -pwd password

Example

For loading all the CSV files in the folder ~/Desktop/dataset referring to the sensor T_LUGANO of the SOS service named demo at the URL http://localhost/istsos

cd /usr/local/istsos
python scripts/csv2istsos.py -p T_LUGANO \
-u http://localhost/istsos -s demo \
-w ~/Desktop/dataset

Loading proprietary plain text files data in Python

Most of the sensor vendors doesn’t really care about interoperability, so sensor owners have to handle incredibly imaginative file formats. Luckily very often output files are in tabular data format (numbers and text) in plain text (CSV-like) that can be reverse engineered.

istSOS have some utility Python classes that can be used to implement a custom converter. In the next paragraphs there are some implementation that uses the raw2istsos.py base class.

If you want to implement something more specific just can take inspiration of the converters that we have prepared and implement one by yourself, and if you want to share we will be happy to add your converter inside the istSOS package.

Generic CSV converter example

File name: maggia_20150612112000.dat

File content:

2015-06-12 09:40:00,0.237
2015-06-12 09:50:00,0.234
2015-06-12 10:00:00,0.237
2015-06-12 10:10:00,0.236
2015-06-12 10:20:00,0.234
2015-06-12 10:30:00,0.237
2015-06-12 10:40:00,0.236
2015-06-12 10:50:00,0.232
2015-06-12 11:00:00,0.236
2015-06-12 11:10:00,0.232
2015-06-12 11:20:00,0.233

This is a quite an understandable format:

  • The file name contains the time of the last observation
  • There is no headers
  • The content is formatted with an ISO8601-like datetime (column index 0) and a numeric value (column index 1)
  • The separator is a comma
  • The time zone have to be known and should not be affected by DST (daylight saving time)

This characteristics can be configured like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from scripts.converter import csv
# Configuring the Converter
conv = csv.CsvImporter('MAGGIA', {
    "headrows": 0,
    "separator": ",",
    "filenamedate": {
      "format": '%Y%m%d%H%M%S',
      "remove": ['maggia_', '.dat']
    },
    "datetime": {
      "column": 0,
      "format": '%Y-%m-%d %H:%M:%S',
      "tz": '+01:00'
    },
    "observations": [{
      "observedProperty": "urn:ogc:def:parameter:x-istsos:1.0:river:water:height",
      "column": 1
    }]
  },
  'http://localhost/istsos', 'demo',
  '/data/maggia', 'maggia_*.dat',
  debug=True,
  archivefolder='/data/archive/maggia'
)
# Converting raw data to text/csv;subtype=istSOS
if conv.execute():
  # Send observation to istSOS
  conv.csv2istsos()

Another example:

A procedure that measures two different observations.

File name: Calcaccia_A_1503261100.dat

Identificazione: Calcaccia A  SN/TD: 739***/MTO-102   Firmware: FW1.18
Valore minimo 0.123 mWS       Valore massimo 0.142 mWS        Valore medio 0.536 mWS  Pressure_Type="rel"
Ora   Data    Pressione [mWS] Conta-impulsi [mm]
09:50:00      26.03.2015      0.140   4.4
10:00:00      26.03.2015      0.137   4.4
10:10:00      26.03.2015      0.139   4.4
10:20:00      26.03.2015      0.141   4.4
10:30:00      26.03.2015      0.139   4.4
10:40:00      26.03.2015      0.139   4.4
10:50:00      26.03.2015      0.139   4.5
11:00:00      26.03.2015      0.139   4.5

Observing the content we can see:

  • The file name contains the time of the last observation
  • There are 3 lines of head rows
  • The date (column index 1) is separated from time (column index 0)
  • The separator is a tab key (/t)
  • The time zone have to be known and should not be affected by DST (daylight saving time)

This characteristics can be configured like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
from scripts.converter import csv
conv = csv.CsvImporter('CALCACCIA', {
    "headrows": 3,
    "separator": "\t",
    "filenamedate": {
      "format": '%y%m%d%H%M',
      "remove": ['Calcaccia_A_','.dat']
    },
    "datetime": {
      "tz": '+01:00',
      "time": {
        "column": 0,
        "format": '%H:%M:%S'
      },
      "date": {
        "column": 1,
        "format": '%d.%m.%Y'
      }
    },
    "observations": [{
        "observedProperty": "urn:ogc:def:parameter:x-istsos:1.0:meteo:air:pressure",
        "column": 2
      },{
        "observedProperty": "urn:ogc:def:parameter:x-istsos:1.0:pulse",
        "column": 3
    }]
  },
  'http://localhost/istsos', 'demo',
  "/data/calcaccia", 'Calcaccia_A_*.dat',
  debug=True,
  archivefolder='/data/archive/calcaccia'
)
# Converting raw data to text/csv;subtype=istSOS
if conv.execute():
  # Send observation to istSOS
  conv.csv2istsos()

The Campbell Scientific / Loggernet example

File name: 16_2015-07-08_182000.dat

File content:

101,2015,189,1800,13.85,41.5,30.3,53.54,979
101,2015,189,1810,13.85,41.45,29.11,61.27,979
99,2015,189,1811,27
99,2015,189,1812,40
99,2015,189,1813,43
99,2015,189,1814,37
99,2015,189,1815,31
101,2015,189,1820,13.85,40.99,26.2,81.3,980

In this file there are more than one observed property from a meteo station with multiple sensors.

Let’s analyse the output format:

  • the first column (column index 0) is an identifier.
  • The columns 1, 2 and 3 are respectivly year, days (from 1 January) and hours and minutes (format HHMM)
  • Rows starting with 99 are irregular time series observations collecting rainfall events (tipping bucket rain gauge)
  • Rows starting with 101 represent a regular time series (10 minutes interval) collecting temperature (column index 6), humidity (col. idx. 7), air pressure (col. idx. 8).

Supponing that raw files are stored into a local folder (e.g. /data/lugano) and we want to insert some observations for the procedure T_LUGANO (air temperature) we should implement something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from scripts.converter import campbell
# Configuring the Converter
conv = campbell.CampbellImporter('T_LUGANO', {
    'rowid': 101,
    'observedProperty': 'urn:ogc:def:parameter:x-istsos:1.0:meteo:air:temperature',
    'column': 6,
    'date': [1,2,3],
    'tz': '+01:00'
  },
  'http://localhost/istsos', 'demo',
  '/data/lugano', '16_*',
  debug=True,
  archivefolder='/data/archive/lugano'
)
# Converting raw data to text/csv;subtype=istSOS
if conv.execute():
  # Send observation to istSOS
  conv.csv2istsos()

The CampbellImporter class (script/converter/campbell.py) is an extension of the basic class Converter (script/raw2istsos.py).

The STS Sensor example

File name: 100538_WTemp.csv

File content:

100538 Vecchio Vedeggio;WTemp C
data_timemeas;data_valueend
2014-12-09 14:30:00;12.526
2014-12-09 14:40:00;12.608
2014-12-09 14:50:01;12.646
2014-12-09 15:00:00;12.667
100538 Vecchio Vedeggio;WTemp C
data_timemeas;data_valueend
2014-12-09 15:10:00;12.682
2014-12-09 15:20:01;12.696
2014-12-09 15:30:00;12.707
2014-12-09 15:40:01;12.714
2014-12-09 15:50:00;12.722
2014-12-09 16:00:00;12.728

This file contains only one observed property. The StsImporter class (script/sts.py) is specifically designed to import this kind of files. Rows that does not contains data are skipped during the importation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from scripts.converter import sts
cov =  sts.StsImporter('WT_VVED_AGN', {
    "tz": "+01:00"
  },
  'http://localhost/istsos', 'demo',
  '/data/sts', '100538_WTemp.csv',
  debug=True,
  archivefolder='/data/archive/vved_agn'
)
if cov.execute():
  cov.csv2istsos()

The Kern datalogger example

Kern file format uses some control characters (Start of Heading /SOH/, Start of Text /STX/, End Of Text /ETX/, end-of-transmission /EOT/, etc.).

File name: HBTIa-17_15_271730_10.csv

File content:

/SOH/TI       2015    271720  000010  HBTIa   17      CUCCIO PORLEZZA
Q     1       1       0       1       LIVELLO
Q     2       3       0       1       TEMPERATURA
Q     3       23      0       1       BATTERIA
/STX/D        271730  1       10      0.377   2       10      18.08   3       10      13.28
/ETX/EOT/

The KernImporter class (script/kern.py) is specialized to import this kind of files, handling all the control characters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from scripts.converter import kern
cov =  kern.KernImporter('A_CUC_POR', {
    "tz": "+01:00",
    "observations": {
      "observedProperty": "urn:ogc:def:parameter:x-istsos:1.0:river:water:height",
      "column": 4
    }
  },
  'http://localhost/istsos', 'sosraw',
  "/data/kern", 'HBTIa-17_*',
  debug=True,
  archivefolder='/data/archive/cuc_por'
)
if cov.execute():
  cov.csv2istsos()
istSOS
Download istSOS

Table Of Contents

Previous topic

Observation Offerings

Next topic

istSOS scheduler for data acquisition

This Page

Navigation

© Copyright 2010-2015, Massimiliano Cannata, Milan Antonovic This work is licensed under a Creative Commons Attribution 4.0 International License. Created using Sphinx 1.2.2.