The _tdm_reaper_ is a C++ based library that decodes (encodes) the proprietary file format _TDM/TDX_ for measurement data, which relies upon the _technical data management_ data model. The TDM format was introduced by [National Instruments](https://www.ni.com) and is employed by [LabVIEW](https://www.ni.com/de-de/shop/labview.html), LabWindows™/CVI™, Measurement Studio, SignalExpress, and [DIAdem](https://www.ni.com/de-de/shop/data-acquisition-and-control/application-software-for-data-acquisition-and-control-category/what-is-diadem.html). ## Data Format Datasets encoded in the TDM/TDX format come along in pairs comprised of a .tdm (header) and a .tdx (data) file. While the .tdm file is a human-readable file providing meta information about the data set, the .tdx is a binary containing the actual data. The .tdm based on the _technical data management_ model is an XML file and basically describes what data the .tdx contains and how to read it. The [TDM data model](https://www.ni.com/de-de/support/documentation/supplemental/10/ni-tdm-data-model.html) structures the data hierarchically with respect to _file_, (channel) _groups_ and _channels_. The file level XML may contain any number of (channel) groups each of which is made up of an arbitrary number of channels. Thus, the XML tree in the [TDM header file](https://zone.ni.com/reference/de-XX/help/370858P-0113/tdmdatamodel/tdmdatamodel/tdm_headerfile/) looks basically like this: ```xml National Instruments USI 1.5 ... ... ... ... ``` and is comprised of _four_ main XML elements: `usi:documentation`, `usi:model`, `usi:include` and `usi:data`. The element `usi:include` references the data file `example.tdx` and reveals one of _two_ possible orderings of the mass data (.tdx): 1. either _channel wise_ (``) - all values of a specific channel follow subsequently - 1. or _block wise_ (``) - all values of a specific measurement time follow subsequently - ordering. The supported _numerical data types_ are | datatype | channel datatype | numeric | value sequence | size | description | |-------------|------------------|---------|-----------------|-------|-------------------------| | eInt16Usi | DT_SHORT | 2 | short_sequence | 2byte | signed 16 bit integer | | eInt32Usi | DT_LONG | 6 | long_sequence | 4byte | signed 32 bit integer | | eUInt8Usi | DT_BYTE | 5 | byte_sequence | 1byte | unsigned 8 bit integer | | eUInt16Usi | DT_SHORT | 2 | short_sequence | 2byte | unsigned 16 bit integer | | eUInt32Usi | DT_LONG | 6 | long_sequence | 4byte | unsigned 32 bit integer | | eFloat32Usi | DT_FLOAT | 3 | float_sequence | 4byte | 32 bit float | | eFloat64Usi | DT_DOUBLE | 7 | double_sequence | 8byte | 64 Bit double | | eStringUsi | DT_STRING | 1 | string_sequence | | text | The XML element `` is basically comprised of _five_ different types of elements that are ``, ``, ``, `` and ``. The root element `` describes the general properties of the dataset and lists the _id's_ of all channel groups that belong to the dataset. The element `` divides the _channels_ into groups and has a unique _id_ that is referenced by its root element. The `` element in `` lists the unique ids of all channels that belong to that group. Finally, the element `` describes a single column of actual data including its datatype. The remaining element types are `` ```xml Untitled #xpointer(id("usiAB")) #xpointer(id("usiMN")) 15 0 ... #xpointer(id("usiZ")) ``` with a unique id, the `` refering to one specific channel, the `` and its id respectively, the type of representation in `` - being one of _explicit_, _implicit linear_ or _rawlinear_ - and the `` element, which refers to one _value sequence_, and the element `` ```xml Untitled #xpointer(id("usiUV")) N #xpointer(id("usiMN")) ``` that references the channel group in `` it belongs to and provides the _number of rows_ in the channels listed in ``. ## Installation The library can be used both as a _CLI_ based tool and as a _Python_ module. ### CLI tool To install the CLI tool _tdmreaper_ do ```Shell make install ``` which uses `/usr/local/bin` as installation directory. On _macOSX_ please first build the binary locally with `make` and install it in your preferred location. ### Python ...tbc... ## Usage ### CLI tool The usage of the CLI tool is sufficiently clarified by its help message displayed by `tdmreaper --help`. For instance, to extract the data decoded in the pair of files `samples/SineData.tdm` and `samples/SineData.tdx` into the directory `/home/jack/data/`: ```Shell tdmreaper samples/SineData.tdm samples/SineData.tdx --output /home/jack/data ``` The tool can also be used to list the available objects in TDM dataset, which are i.a. _channels_, _channelgroups_ and TDX _blocks_. For instance, to list all channels and channelgroups (without writing any file output): ```Shell tdmreaper samples/SineData.tdm samples/SineData.tdx --listgroups --listchannels ``` The user may also submit a _filenaming rule_ to control names of the files the channel(-group)s are written to. To this end, the _magic flags_ `%G` `%g`, `%C` and `%c` representing the group id, group name, channel index and channel name are defined. The default filenaming option is ```Shell tdmreaper samples/SineData.tdm samples/SineData.tdx --output /home/jack/data --filenames channelgroup_%G.csv ``` which makes the tool write _all channels_ grouped into files according to their group association, while all channelgroup filenames obey the pattern `channelgroup_%G.csv` with `%G` being replaced by the group id. The filenaming rule enables the user to extract only single channelgroup or channel by providing a particular channel(-group) id in the filenaming flag. For example, ```Shell tdmreaper samples/SineData.tdm samples/SineData.tdx --output /home/jack/data -f channel_usi16_%c.csv --includemeta ``` will write the single channel with id `usi16` (including its meta data as header) to the file `/home/jack/data/channel_usi16_A4.csv`. ### Python ...tbc... ## !!! Deprecated !!! The makefile provides targets for using the library both as native C++ extension and as Python module. The package supports usage on Linux and MacOSX. The tdm_ripper module is built on these platforms by ```Shell # Linux pip install Cython make install ``` and ```Shell # macOS pip install Cython make install_osx ``` ## Usage Although the package is built upon a C++ core, which decodes the data, it may be used as a Python module, as well, by interfacing the C++ library with a Cython wrapper. ### C++ core - In order to parse the XML tree of the .tdm file, the library employs pugixml: https://pugixml.org/ and https://github.com/zeux/pugixml - The package currently supports the following datatypes: - eInt8Usi: 8 byte - eInt16Usi: 16 byte - eInt32Usi: 32 byte - eInt64Usi: 64 byte - eUInt8Usi: 8 byte - eUInt16Usi: 16 byte - eUInt32Usi: 32 byte - eUInt64Usi: 64 byte - eFloat32Usi: 32 byte - eFloat64Usi: 64 byte - The core of the library takes care of the decoding by reinterpretation of the binary in the buffer as the required datatype implemented by ```C++ uint8_t *dfcast = reinterpret_cast(&df); for ( int i = 0; i < (int)sizeof(double); i++ ) { dfcast[i] = (int)bych[i]; } ``` where for instance df is the resulting float and bych contains the binary data as an array of chars. - main.cpp contains an example of how the C++ library might be used to provide the channels and groups of the dataset. It is simply build by ```Shell make ``` - extract_all.cpp takes the .tdm, the .tdx file and some output directory as arguments to provide all given information in .csv format without any logging. To build: ```Shell make extall ``` For instance, the executable accepts the following arguments: ```Shell ./extract_all samples/SineData.tdm samples/SineData.tdx data/ ``` ### Python module - The library may also be used as a Python module and supports the use of group channels in NumPy arrays as shown in example.py . - To extract all available information and data in the TDM files without any further interaction, the use of extract_all.py is recommended. To exhibit the required arguments: ```Shell python extract_all.py --help ``` - The same functionality may be obtained from an existing python script by importing the tdm_ripper module and calling the extract_all function. For instance ```Python import tdm_ripper as td files = td.extract_all(b"samples/SineData.tdm",b"samples/SineData.tdx",b"data/",b"my_tdm") ``` where the arguments "data/" and "my_tdm" are optional. "data/" specifies the directory where all .csv output is dumped while "my_tdm" represents a name prefix for all csv. files. Note, that all string arguments must be converted to bytes before passing to the argument list by prepending "b". ## References ### TDM - https://www.ni.com/de-de/support/documentation/supplemental/10/ni-tdm-data-model.html - https://zone.ni.com/reference/en-XX/help/371361R-01/lvconcepts/fileio_tdms_model/ - https://zone.ni.com/reference/en-XX/help/371361R-01/lvhowto/ni_test_data_exchange/ - https://www.ni.com/de-de/support/documentation/supplemental/06/the-ni-tdms-file-format.html - https://zone.ni.com/reference/de-XX/help/370858P-0113/tdmdatamodel/tdmdatamodel/tdm_headerfile/ ### IEEE Standard and datatypes - https://en.wikipedia.org/wiki/IEEE_754 - https://www.ias.ac.in/public/Volumes/reso/021/01/0011-0030.pdf - https://en.cppreference.com/w/cpp/language/types ### Code example - https://www.ni.com/content/dam/web/product-documentation/c_dll_tdm.zip