From b869c1df3f38b7535768ee4950f069833cf65794 Mon Sep 17 00:00:00 2001 From: Mario Fink Date: Fri, 5 Feb 2021 11:00:29 +0100 Subject: [PATCH] clean up docu --- README.md | 154 +++++++++++----------------------------------- doc/deprecated.md | 114 ++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+), 117 deletions(-) create mode 100644 doc/deprecated.md diff --git a/README.md b/README.md index 3bd77ef..e2a5ff6 100644 --- a/README.md +++ b/README.md @@ -55,8 +55,6 @@ The format supports the storage of _multiple data sets (channels)_ in a single file. The channels may be ordered in _multiplex_ mode (ordering w.r.t. time) or _block_ mode (ordering w.r.t. to channels). -### Markers - The markers (keys) are introduced by `"|" = 0x 7c` followed by two uppercase letters. There are _two types_ of markers distinguished by the first letter: @@ -76,123 +74,45 @@ and line feeds (`LF = 0x 0a`) between keys, i.e. the block terminating semicolon and the vertical bar (pipe) of the next key. The following _critical markers_ are defined + +| marker | description | +|--------|-----------------------------------------------------------------------------------------------------| +| CF | format version and processor | +| CK | start of group of keys, no. parameters = 3, indicates (in)correct closure of the measurement series | +| CB | defines a group of cannels | +| CT | text definition including group association index | +| CG | introduces group of components corresponding to CC keys | +| CD1,2 | old/new version of abscissa description | +| CZ | scaling of z-axis for segments | +| CC | start of a component | +| CP | information about buffer, datatype and samples of component | +| Cb | buffer description | +| CR | permissible range of values in component | +| CN | name and comment of channel | +| CS | raw binary data | +| CI | single numerical value (including unit) | +| Ca | add reference key | + +Among the _noncritical_ markers, there are + +| marker | description | +|--------|--------------------------------------------| +| NO | origin of data | +| NT | timestamp of trigger | +| ND | (color) display properties | +| NU | user defined key | +| Np | property of a channel | +| NE | extraction rule for channels from BUS data | + +The format loosely defines some rules for the ordering of the markers in the +file stream. The rules for critical keys include: _CK_ has to follow up on _CF_, +_CK_ may be followed by any number of _CG_ blocks, each _CG_ has to be followed +by (any number of) component sequences comprised of the series _CC_ , _CP_, +(_CR_), (_ND_) and terminated by either _CS_ or the start of a new group, +component, text field or buffer. + ## Installation ## Usage ## References - -# Deprecated!! - -The following markers are defined: - -1. CF (0x 43 46) -1. CK (0x 43 4b) -1. NO (0x 4e 4f) -1. CG (0x 43 47) -1. CD (0x 43 44) -1. NT (0x 4e 54) -1. CC (0x 43 43) -1. CP (0x 43 50) -1. CR (0x 43 52) -1. CN (0x 43 4e) -1. Cb (0x 43 62) -1. CS (0x 43 53) - -Each of these markers are followed by multiple commata (0x 2c) separated parameters -and are terminated by a semicolon `;` = 0x 3b, except for the sequence following -the data marker CS, that may have any number of 0x3b occurencies, while still -terminated by a semicolon at the very end of the file (since CS is the last marker -section in the file). The markers have the following meaning: - -- *CF* (3 parameters) - `|CF,2,1,1;` - specifies file format, key length and processor -- *CK* (4 parameters) - `|CK,1,3,1,1;` - start of group of keys -- *NO* (6 parameters) - `|NO,1,85,0,77,imc STUDIO 5.0 R3 (10.09.2015)@imc DEVICES 2.8R7 (26.8.2015)@imcDev__15190567,0,;` - origin of the file, provides some info about the software package/device - and its version -- *CB* (6 parameters) - group definition -- *CT* (8 parameters) - text definition -- *CG* (5 parameters) - `|CG,1,5,1,1,1;` - definition of a data field - |CG,1,KeyLang,AnzahlKomponenten,Feldtyp,Dimension; -- *CD* (mostly 11 parameters) - since we're dealing with measured entities from the lab this markers contains - info about the measurement frequency, i.e. sample rate. For instance - `|CD,2, 63, 5.0000000000000001E-03,1,1,s,0,0,0, 0.0000000000000000E+00,1;` - indicates a measured entity every 0.005 seconds, i.e. a sample rate = 200Hz -- *NT* (7 parameters) - `|NT,1,16,1,1,1980,0,0,0.0;` - |NT,1,KeyLang,Tag,Monat,Jahr,Stunden,Minuten,Sekunden; - triggerzeit -- *CC* (mostly 4 parameters) - `|CC,1,3,1,1;` - Start einer Komponente (component) -- *CP* (9 parameters) - `|CP,1,16,1,4,7,32,0,0,1,0;` - Pack-Information zu dieser Komponente - CP,1,KeyLang,BufferReferenz,Bytes,Zahlenformat,SignBits,Maske,Offset,DirekteFolgeAnzahl,AbstandBytes; - Bytes = 1...8 - Zahlenformat : 1 = unsigned byte - 2 = signed byte - 3 = unsigned short - 4 = signed short - 5 = unsigned long - 6 = signed long - 7 = float - 8 = double - 9 = imc Devices - 10 = timestamp ascii - 11 = - 12 = - 13 = - -- *CR* (7 parameters) - Wertebereich der Komponente, nur bei analogen, nicht bei digitalen Daten. - |CR,1,KeyLang,Transformieren,Faktor,Offset,Kalibriert,EinheitLang, Einheit; - provides the _physical unit_ of the measured entity, maybe shows the - minimum and maximum value during the measurment, e.g. - `|CR,1,60,0, 1.0000000000000000E+00, 0.0000000000000000E+00,1,4,mbar;` - Transformieren : 0 = nein - 1 = ja, mit faktor und offset transformieren (für ganzzahlige Rohdaten) - Faktor,Offset: physikalischer Wert = Faktor * Rohdatenwerten + Offset -- *CN* (mostly 9 parameters) - gives the _name_ of the measured entity - |CN,1,KeyLang,IndexGruppe,0,IndexBit,NameLang,Name,KommLang,Kommentar; - `|CN,1,27,0,0,0,15,pressure_Vacuum,0,;` -- *Cb* (mostly 14 paramters) (optional?) - this one probably gives the minimum/maximum measured values!! - `|Cb,1,117,1,0,1,1,0,341288,0,341288,1,0.0000000000000000E+00,1.1781711390000000E+09,;` -- *CS* (mostly 4 parameters) - this markers announces the actual measurement data in binary format, - provide the number of values and the actual data, - e.g. `|CS,1, 341299, 1, ...data... ;` - -### Open Issues and question? - -- which parameter indicate(s) little vs. big endian? - -## .parquet-file writer - -The extracted and converted data originating from the *.raw file format may be efficiently grouped and -written as .parquet files -[parquet file writer example](https://github.com/apache/parquet-cpp/blob/master/examples/low-level-api/reader-writer.cc) - - -## References - -- https://ch.mathworks.com/matlabcentral/fileexchange/30187-sequnce-to-read-famos-data-into-matlab-workspace -- https://community.ptc.com/t5/PTC-Mathcad/FAMOS-IMC-raw-data-in-MathCAD/td-p/130378 -- http://marmatek.com/wp-content/uploads/2014/04/imc_STUDIO_Manual.pdf - -### Parquet - -- https://github.com/apache/parquet-cpp -- https://github.com/apache/parquet-cpp/tree/master/examples diff --git a/doc/deprecated.md b/doc/deprecated.md new file mode 100644 index 0000000..83449cf --- /dev/null +++ b/doc/deprecated.md @@ -0,0 +1,114 @@ + +# Deprecated!! + +The following markers are defined: + +1. CF (0x 43 46) +1. CK (0x 43 4b) +1. NO (0x 4e 4f) +1. CG (0x 43 47) +1. CD (0x 43 44) +1. NT (0x 4e 54) +1. CC (0x 43 43) +1. CP (0x 43 50) +1. CR (0x 43 52) +1. CN (0x 43 4e) +1. Cb (0x 43 62) +1. CS (0x 43 53) + +Each of these markers are followed by multiple commata (0x 2c) separated parameters +and are terminated by a semicolon `;` = 0x 3b, except for the sequence following +the data marker CS, that may have any number of 0x3b occurencies, while still +terminated by a semicolon at the very end of the file (since CS is the last marker +section in the file). The markers have the following meaning: + +- *CF* (3 parameters) + `|CF,2,1,1;` + specifies file format, key length and processor +- *CK* (4 parameters) + `|CK,1,3,1,1;` + start of group of keys +- *NO* (6 parameters) + `|NO,1,85,0,77,imc STUDIO 5.0 R3 (10.09.2015)@imc DEVICES 2.8R7 (26.8.2015)@imcDev__15190567,0,;` + origin of the file, provides some info about the software package/device + and its version +- *CB* (6 parameters) + group definition +- *CT* (8 parameters) + text definition +- *CG* (5 parameters) + `|CG,1,5,1,1,1;` + definition of a data field + |CG,1,KeyLang,AnzahlKomponenten,Feldtyp,Dimension; +- *CD* (mostly 11 parameters) + since we're dealing with measured entities from the lab this markers contains + info about the measurement frequency, i.e. sample rate. For instance + `|CD,2, 63, 5.0000000000000001E-03,1,1,s,0,0,0, 0.0000000000000000E+00,1;` + indicates a measured entity every 0.005 seconds, i.e. a sample rate = 200Hz +- *NT* (7 parameters) + `|NT,1,16,1,1,1980,0,0,0.0;` + |NT,1,KeyLang,Tag,Monat,Jahr,Stunden,Minuten,Sekunden; + triggerzeit +- *CC* (mostly 4 parameters) + `|CC,1,3,1,1;` + Start einer Komponente (component) +- *CP* (9 parameters) + `|CP,1,16,1,4,7,32,0,0,1,0;` + Pack-Information zu dieser Komponente + CP,1,KeyLang,BufferReferenz,Bytes,Zahlenformat,SignBits,Maske,Offset,DirekteFolgeAnzahl,AbstandBytes; + Bytes = 1...8 + Zahlenformat : 1 = unsigned byte + 2 = signed byte + 3 = unsigned short + 4 = signed short + 5 = unsigned long + 6 = signed long + 7 = float + 8 = double + 9 = imc Devices + 10 = timestamp ascii + 11 = + 12 = + 13 = + +- *CR* (7 parameters) + Wertebereich der Komponente, nur bei analogen, nicht bei digitalen Daten. + |CR,1,KeyLang,Transformieren,Faktor,Offset,Kalibriert,EinheitLang, Einheit; + provides the _physical unit_ of the measured entity, maybe shows the + minimum and maximum value during the measurment, e.g. + `|CR,1,60,0, 1.0000000000000000E+00, 0.0000000000000000E+00,1,4,mbar;` + Transformieren : 0 = nein + 1 = ja, mit faktor und offset transformieren (für ganzzahlige Rohdaten) + Faktor,Offset: physikalischer Wert = Faktor * Rohdatenwerten + Offset +- *CN* (mostly 9 parameters) + gives the _name_ of the measured entity + |CN,1,KeyLang,IndexGruppe,0,IndexBit,NameLang,Name,KommLang,Kommentar; + `|CN,1,27,0,0,0,15,pressure_Vacuum,0,;` +- *Cb* (mostly 14 paramters) (optional?) + this one probably gives the minimum/maximum measured values!! + `|Cb,1,117,1,0,1,1,0,341288,0,341288,1,0.0000000000000000E+00,1.1781711390000000E+09,;` +- *CS* (mostly 4 parameters) + this markers announces the actual measurement data in binary format, + provide the number of values and the actual data, + e.g. `|CS,1, 341299, 1, ...data... ;` + +### Open Issues and question? + +- which parameter indicate(s) little vs. big endian? + +## .parquet-file writer + +The extracted and converted data originating from the *.raw file format may be efficiently grouped and +written as .parquet files +[parquet file writer example](https://github.com/apache/parquet-cpp/blob/master/examples/low-level-api/reader-writer.cc) + +## References + +- https://ch.mathworks.com/matlabcentral/fileexchange/30187-sequnce-to-read-famos-data-into-matlab-workspace +- https://community.ptc.com/t5/PTC-Mathcad/FAMOS-IMC-raw-data-in-MathCAD/td-p/130378 +- http://marmatek.com/wp-content/uploads/2014/04/imc_STUDIO_Manual.pdf + +### Parquet + +- https://github.com/apache/parquet-cpp +- https://github.com/apache/parquet-cpp/tree/master/examples