add timestamp insertion for build on Darwin

* simplify get_channel_overview(...) by removing full group info
* add build-timestamp to version/help message of CLI binary
* simplify/unify group/channel/submatrix/localcolumn overview of CLI
binary

.gitignore

@@ -11,3 +11,15 @@ tdmripper
 *.log
 tdmreaper
 cython/*.cpp
+tdmtermite
+dist/
+*.egg-info/
+output/
+monitor-process.sh
+tdmtest
+pip/*.hpp
+pip/*.md
+pip/*.cpp
+pip/*.pyx
+pip/*.pxd
+pip/LICENSE

Dockerfile

@@ -1,20 +1,30 @@
 
-FROM debian:bullseye-20210111
+FROM debian:bullseye 
 
-USER root
-
-RUN apt-get update && apt-get install -y \
-    build-essential git \
+# install requirements
+RUN apt-get update && apt-get upgrade -y && apt-get install -y \
+    build-essential \
+    g++ make git \
     python3 python3-pip
 
-RUN g++ -v
+# check compiler and current user
+RUN g++ -v && whoami
 
-COPY ./ /tdm_ripper/
+# use /home as working directory
+WORKDIR /home
+
+# get the public TDMtermite repository
+RUN git clone https://github.com/RecordEvolution/TDMtermite.git
 
 # install CLI tool
-RUN cd /tdm_ripper && ls -lh && make install && ls -lh /usr/local/bin/tdmreaper
+RUN cd ./TDMtermite && ls -lh && make install && ls -lh /usr/local/bin/tdmtermite
 
 # install Python module
-RUN cd /tdm_ripper && ls -lh && make cython-requirements && make cython-install
+RUN cd ./TDMtermite && ls -lh && make cython-requirements && make cython-list && make cython-install
+
+# create directory for data exchange
+#RUN [ "/bin/bash", "-c", "mkdir -pv data/{input,output}" ]
+RUN mkdir -pv data
+
+CMD ["sleep","infinity"]
 
-CMD ["sleep","inifity"]

README.md

@@ -1,35 +1,42 @@
 
-<p align="center">
-  <a href="https://github.com/RecordEvolution/tdm_ripper.git">
-    <img
-      alt="tdmreaper.svg"
-      src="assets/tdmreaper.svg"
-      width="400"
-    />
-  </a>
-</p>
+[![Total alerts](https://img.shields.io/lgtm/alerts/g/RecordEvolution/TDMtermite.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/RecordEvolution/TDMtermite/alerts/)
+[![Language grade: C/C++](https://img.shields.io/lgtm/grade/cpp/g/RecordEvolution/TDMtermite.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/RecordEvolution/TDMtermite/context:cpp)
+[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/RecordEvolution/TDMtermite.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/RecordEvolution/TDMtermite/context:python)
+[![LICENSE](https://img.shields.io/github/license/RecordEvolution/TDMtermite)](https://img.shields.io/github/license/RecordEvolution/TDMtermite)
+[![STARS](https://img.shields.io/github/stars/RecordEvolution/TDMtermite)](https://img.shields.io/github/stars/RecordEvolution/TDMtermite)
 
-The _tdm_reaper_ is a C++ based library that decodes (encodes) the proprietary
-file format _TDM/TDX_ for measurement data, which relies upon the
-_technical data management_ data model. The TDM format was introduced by
-[National Instruments](https://www.ni.com) and is employed by
+# TDMtermite
+
+_TDMtermite_ is a C++ based library that decodes the proprietary
+file format _TDM/TDX_ for measurement data. First introduced by
+[National Instruments](https://www.ni.com), the TDM format relies on the
+_technical data management_ data model and is employed by
 [LabVIEW](https://www.ni.com/de-de/shop/labview.html), LabWindows™/CVI™,
 Measurement Studio, SignalExpress, and [DIAdem](https://www.ni.com/de-de/shop/data-acquisition-and-control/application-software-for-data-acquisition-and-control-category/what-is-diadem.html).
 
-## Data Format
+The [Record Evolution Platform](https://www.record-evolution.de/en/home-en/) uses TDMtermite to integrate measurement data into ETL processes. The TDMtermite library is available both as a command line tool and as a Python module. The Python module of TDMtermite enables data scientists to conveniently include TDM formats in their existing data pipelines by providing access to both raw data and metadata in terms of native Python objects.    
 
-Datasets encoded in the TDM/TDX format come along in pairs comprised of a
-.tdm (header) and a .tdx (data) file. While the .tdm file is a human-readable
-file providing meta information about the data set, the .tdx is a binary
+## Overview
+
+* [TDM file format](#Dataformat)
+* [Build and Installation](#Installation)
+* [Usage and Examples](#Usage)
+* [References](#References)
+
+## Dataformat
+
+Datasets encoded in the TDM/TDX format come in pairs comprised of a
+.tdm (header) file and a .tdx (data) file. While the .tdm file is a human-readable
+file providing meta information about the dataset, the .tdx file is a binary file
 containing the actual data. The .tdm based on the _technical data management_
-model is an XML file and basically describes what data the .tdx contains and how
+model is an XML file. It describes what data the .tdx file contains and how
 to read it. The
 [TDM data model](https://www.ni.com/de-de/support/documentation/supplemental/10/ni-tdm-data-model.html)
-structures the data hierarchically with respect to _file_, (channel) _groups_ and
-_channels_. The file level XML may contain any number of (channel) groups each
+structures the data hierarchically with respect to _file_, _(channel)_ _groups_ and
+_channels_. The file-level XML may contain any number of (channel) groups, each
 of which is made up of an arbitrary number of channels. Thus, the XML tree in
 the [TDM header file](https://zone.ni.com/reference/de-XX/help/370858P-0113/tdmdatamodel/tdmdatamodel/tdm_headerfile/)
-looks basically like this:
+looks like this:
 
 ```xml
 <?xml version="1.0" encoding="UTF-8" standalone="no" ?>
@@ -60,14 +67,14 @@ looks basically like this:
 </usi:tdm>
 ```
 
-and is comprised of _four_ main XML elements: `usi:documentation`, `usi:model`,
+The XML tree is comprised of _four_ main XML elements: `usi:documentation`, `usi:model`,
 `usi:include` and `usi:data`. The element `usi:include` references the data file
 `example.tdx` and reveals one of _two_ possible orderings of the mass data (.tdx):
 
-1. either _channel wise_ (`<block>`) - all values of a specific channel follow subsequently -
-1. or _block wise_ (`<block_bm>`) - all values of a specific measurement time follow subsequently -
+1. either _channel-wise_ (`<block>`) - all values of a specific channel follow subsequently
+1. or _block-wise_ (`<block_bm>`) - all values of a specific measurement time follow subsequently.
 
-ordering. The supported _numerical data types_ are
+The supported _numerical data types_ are:
 
 | datatype    | channel datatype | numeric | value sequence  | size  | description             |
 |-------------|------------------|---------|-----------------|-------|-------------------------|
@@ -80,10 +87,10 @@ ordering. The supported _numerical data types_ are
 | eFloat64Usi | DT_DOUBLE        | 7       | double_sequence | 8byte | 64 Bit double           |
 | eStringUsi  | DT_STRING        | 1       | string_sequence |       | text                    |
 
-The XML element `<usi:data>` is basically comprised of _five_ different types of
+The XML element `<usi:data>` is comprised of _five_ different types of
 elements that are `<tdm_root>`, `<tdm_channelgroup>`, `<tdm_channel>`, `<localcolumn>`
 and `<submatrix>`. The root element `<tdm_root>` describes the general properties
-of the dataset and lists the _id's_ of all channel groups that belong to
+of the dataset and lists the _ids_ of all channel groups that belong to
 the dataset. The element `<tdm_channelgroup>` divides the _channels_ into groups
 and has a unique _id_ that is referenced by its root element. The `<channels>`
 element in `<tdm_channelgroup>` lists the unique ids of all channels that belong
@@ -103,7 +110,7 @@ actual data including its datatype. The remaining element types are
 </localcolumn>
 ```
 
-with a unique id, the `<measurement_quantity>` refering to one specific channel,
+with a unique id, the `<measurement_quantity>` referring to one specific channel,
 the `<submatrix>` and its id respectively, the type of representation in
 `<sequence_representation>` - being one of _explicit_, _implicit linear_ or
 _rawlinear_ - and the `<values>` element, which refers to one _value sequence_,
@@ -118,107 +125,121 @@ and the element `<submatrix>`
 </submatrix>
 ```
 
-that references the channel group in `<measurement>` it belongs to and provides
+that references the channel group in `<measurement>` to which it belongs and provides
 the _number of rows_ in the channels listed in `<local_columns>`.
 
 ## Installation
 
-The library can be used both as a _CLI_ based tool and as a _Python_ module.
+The library can be used both as a _CLI_-based tool and as a _Python_ module.
 
 ### CLI tool
 
-To install the CLI tool _tdmreaper_ do
+To install the CLI tool _TDMtermite_, do
 
 ```Shell
 make install
 ```
 
-which uses `/usr/local/bin` as installation directory. On _macOSX_ please first
+which uses `/usr/local/bin` as an installation directory. On _macOSX_, please first
 build the binary locally with `make` and install it in your preferred location.
 
 ### Python
 
-In order to build a _Python module_ from the _C++_ code base the
+In order to build a _Python module_ from the _C++_ code base, the
 [Cython](https://cython.readthedocs.io/en/latest/index.html) package must be
-available, which may be installed via `python3 -m pip install cython` .
-Furthermore, the [Numpy](https://numpy.org) package is recommended to be able
+available. It may be installed via `python3 -m pip install cython` .
+The [Numpy](https://numpy.org) package is recommended
 to pass arrays of data from the C++ kernel to Python. The _makefile_ provides
 the target `make cython-requirements` to install all required Python modules.
-Finally, to build the Python extension _tdm_reaper_ either locally or install
-it the targets `make cython-build` and `make cython-install` are provided.
-Hence, to install the Python module on the system simply do
+Finally, to build the Python extension _tdm_termite_ locally or install
+it, the targets `make cython-build` and `make cython-install` are provided.
+To install the Python module on the system, simply do
 
 ```Shell
 make cython-requirements
 make cython-install
 ```
 
-that makes the module available to be imported as `import tdm_reaper` .
+which makes the module available for import by `import tdm_termite` .
+
+#### Installation with pip
+
+The package is also available via the [Python Package Index](https://pypi.org) at
+[TDMtermite](https://pypi.org/project/TDMtermite/). To install the latest version simply do
+
+```Shell
+python3 -m pip install TDMtermite
+```
+
+##### Unix
+
+Note, that _python3_setuptools_ and _gcc version >= 10.2.0_ are required to
+successfully install and use it.
 
 ## Usage
 
 ### CLI tool
 
 The usage of the CLI tool is sufficiently clarified by its help message displayed
-by `tdmreaper --help`. For instance, to extract the data decoded in the pair of
+by `tdmtermite --help`. To extract the data decoded in the pair of
 files `samples/SineData.tdm` and `samples/SineData.tdx` into the directory
 `/home/jack/data/`:
 
 ```Shell
-tdmreaper samples/SineData.tdm samples/SineData.tdx --output /home/jack/data
+tdmtermite samples/SineData.tdm samples/SineData.tdx --output /home/jack/data
 ```
 
 The tool can also be used to list the available objects in the TDM dataset, which
-are i.a. _channels_, _channelgroups_ and TDX _blocks_. For instance, to list
+are i.a. _channels_, _channelgroups_ and TDX _blocks_. To list
 all channels and channelgroups (without writing any file output):
 
 ```Shell
-tdmreaper samples/SineData.tdm samples/SineData.tdx --listgroups --listchannels
+tdmtermite samples/SineData.tdm samples/SineData.tdx --listgroups --listchannels
 ```
 
 The user may also submit a _filenaming rule_ to control the names of the files the
-channel(-group)s are written to. To this end, the _magic flags_ `%G` `%g`, `%C`
+channel(group)s are written to. To this end, the _magic flags_ `%G` `%g`, `%C`
 and `%c` representing the group id, group name, channel index and channel name
-are defined. The default filenaming option is
+are defined. The default filenaming option is:
 
 ```Shell
-tdmreaper samples/SineData.tdm samples/SineData.tdx --output /home/jack/data --filenames channelgroup_%G.csv
+tdmtermite samples/SineData.tdm samples/SineData.tdx --output /home/jack/data --filenames channelgroup_%G.csv
 ```
 
-which makes the tool write _all channels_ grouped into files according to their
-group association, while all channelgroup filenames obey the pattern `channelgroup_%G.csv`
+This makes the tool write _all channels_ grouped into files according to their
+group association, while all channelgroup filenames obey the pattern `channelgroup_%G.csv`,
 with `%G` being replaced by the group id. The filenaming rule also enables the user
-to extract only a single channel(group) by providing a particular channel(-group)
+to extract only a single channel(group) by providing a particular channel(group)
 id in the filenaming flag. For example,
 
 ```Shell
-tdmreaper samples/SineData.tdm samples/SineData.tdx --output /home/jack/data -f channel_usi16_%c.csv --includemeta
+tdmtermite samples/SineData.tdm samples/SineData.tdx --output /home/jack/data -f channel_usi16_%c.csv --includemeta
 ```
 
-will write the single channel with id `usi16` to the file
-`/home/jack/data/channel_usi16_A4.csv` including its meta-data as a file header.
+This will write the single channel with the id `usi16` to the file
+`/home/jack/data/channel_usi16_A4.csv`, including its meta-data as a file header.
 
 ### Python
 
-To be able to use the Python module _tdm_reaper_ it first has to be build locally
-or installed on the system. In the Python interpreter simply do:
+To be able to use the Python module _tdm_termite_, it first has to be built locally
+or installed on the system. In the Python interpreter, simply do:
 
 ```Python
-import tdm_reaper
+import tdm_termite
 ```
 
-to import the module. The TDM files are provided by creating an instance of
-the _tdm_reaper_ class:
+This will import the module. The TDM files are provided by creating an instance of
+the _tdm_termite_ class:
 
 ```Python
-# create 'tdm_reaper' instance object
+# create 'tdm_termite' instance object
 try :
-    jack = tdm_reaper.tdmreaper(b'samples/SineData.tdm',b'samples/SineData.tdx')
+    jack = tdm_termite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')
 except RuntimeError as e:
     print("failed to load/decode TDM files: " + str(e))
 ```
 
-After initializing the _tdm_reaper_ object it can be used to extract any of the
+After initializing the _tdm_termite_ object, it can be used to extract any of the
 available data. For instance, to list the included channelgroups and channels:
 
 ```Python
@@ -230,16 +251,16 @@ grpids = jack.get_channelgroup_ids()
 chnids = jack.get_channel_ids()
 ```
 
-As a use case, we have look at listing the ids of all channelgroups and printing
+As a use case, we have a look at listing the ids of all channelgroups and printing
 their data to separate files:
 
 ```Python
-import tdm_reaper
+import tdm_termite
 import re
 
-# create 'tdm_reaper' instance object
+# create 'tdm_termite' instance object
 try :
-    jack = tdm_reaper.tdmreaper(b'samples/SineData.tdm',b'samples/SineData.tdx')
+    jack = tdm_termite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')
 except RuntimeError as e :
     print("failed to load/decode TDM files: " + str(e))
 
@@ -267,17 +288,21 @@ for grp in grpids :
         print("failed to print channelgroup: " + str(grp) + " : " + str(e))
 ```
 
-For a full example including more details see [python/usage.py](python/usage.py)
+For details, see this [extensive example](python/usage.py)
 and the absolute minimal example [minimal usage](python/minimal.py). In order
 to simply extract all data of the TDM datatset and dump it to files in a given
 (existing!) directory, do
 
 ```Python
-import tdm_reaper
-jack = tdm_reaper.tdmreaper(b'samples/SineData.tdm',b'samples/SineData.tdx')
-jack.write_all(b"./my_tdm_data/")
+import tdm_termite
+jack = tdm_termite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')
+jack.write_all(b"./my_tdm_data_directory/")
 ```
 
+The interface allows you to construct customized file/column headers from any
+meta-data and provide these headers for usage in file output (see this
+[example](python/custom.py)).
+
 ## References
 
 ### TDM
@@ -297,6 +322,30 @@ jack.write_all(b"./my_tdm_data/")
 
 ### Implementation
 
+- https://en.cppreference.com/w/
 - https://pugixml.org/
 - https://github.com/zeux/pugixml
 - https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html
+
+### Packaging
+
+#### Documentation
+
+- https://packaging.python.org/tutorials/packaging-projects/
+- https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
+- https://test.pypi.org/account/register/
+- https://github.com/pypa/auditwheel
+- https://github.com/pypa/python-manylinux-demo
+- https://github.com/pypa/manylinux
+
+#### C/C++ Extensions
+
+- https://docs.python.org/3/extending/building.html
+
+#### Articles
+
+- https://martinsosic.com/development/2016/02/08/wrapping-c-library-as-python-module.html
+- https://malramsay.com/post/perils-of-packaging/
+- https://github.com/neuronsimulator/nrn/issues/329
+- https://levelup.gitconnected.com/how-to-deploy-a-cython-package-to-pypi-8217a6581f09
+- https://medium.com/swlh/distributing-python-packages-protected-with-cython-40fc29d84caf

assets/index.html

@@ -0,0 +1,55 @@
+
+<html>
+
+<head>
+
+  <title>TDMtermite</title>
+  <style>
+  </style>
+
+</head>
+
+<body>
+
+  <p align="center">
+    <a href="https://github.com/RecordEvolution/TDMtermite.git">
+      <img alt="tdmtermite.svg"
+           src="assets/tdmtermite.svg"
+           width="400"
+      />
+    </a>
+  </p>
+
+  <div style="width: 100%; display: block; margin-left: 28%; margin-right: 28%;">
+
+  <div style="width: 100%; overflow: hidden;">
+
+    <div style="margin: 5px; float: left;">
+      <a href="https://lgtm.com/projects/g/RecordEvolution/TDMtermite/alerts/">
+        <img alt="Total alerts" 
+             src="https://img.shields.io/lgtm/alerts/g/RecordEvolution/TDMtermite.svg?logo=lgtm&logoWidth=18"/>
+      </a>
+    </div>
+
+    <div style="margin: 5px; float: left;">
+      <a href="https://lgtm.com/projects/g/RecordEvolution/TDMtermite/context:cpp">
+        <img alt="Language grade: C/C++" 
+             src="https://img.shields.io/lgtm/grade/cpp/g/RecordEvolution/TDMtermite.svg?logo=lgtm&logoWidth=18"/>
+      </a>
+    </div>
+
+    <div style="margin: 5px; float: left;">
+      <a href="https://lgtm.com/projects/g/RecordEvolution/TDMtermite/context:python">
+        <img alt="Language grade: Python" 
+             src="https://img.shields.io/lgtm/grade/python/g/RecordEvolution/TDMtermite.svg?logo=lgtm&logoWidth=18"/>
+      </a>
+    </div>
+
+  </div>
+  
+  </div>
+
+</body>
+
+</html>
+

assets/tdmtermite.svg

@@ -0,0 +1,100 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   viewBox="0 0 366.72 77.08"
+   version="1.1"
+   id="svg3945"
+   width="366.72"
+   height="77.080002"
+   sodipodi:docname="tdmtermite.svg"
+   inkscape:version="1.0.1 (3bc2e813f5, 2020-09-07)">
+  <sodipodi:namedview
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1"
+     objecttolerance="10"
+     gridtolerance="10"
+     guidetolerance="10"
+     inkscape:pageopacity="0"
+     inkscape:pageshadow="2"
+     inkscape:window-width="1920"
+     inkscape:window-height="1168"
+     id="namedview19"
+     showgrid="false"
+     inkscape:zoom="3.8766824"
+     inkscape:cx="199.46638"
+     inkscape:cy="38.540001"
+     inkscape:window-x="2048"
+     inkscape:window-y="0"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="svg3945" />
+  <metadata
+     id="metadata3951">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title>flasher</dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <defs
+     id="defs3949" />
+  <title
+     id="title3916">flasher</title>
+  <g
+     id="logog">
+    <path
+       id="path138"
+       d="m 32.86,2 -13,7.5 v 0 h -0.05 v 0 l -0.48,0.28 c -4.27,2.46 -5.68,11.38 -6.06,14.75 L 36.2,11.33 c 0.39,-0.19 7.6,-3.69 13.57,-3.69 h 0.14 L 40.13,2 a 8.15,8.15 0 0 0 -7.27,0"
+       transform="translate(-2.04,-1.15)"
+       style="fill:#364d5c" />
+    <path
+       id="path142"
+       d="M 5.68,17.69 A 8.2,8.2 0 0 0 2,24 v 15.78 c 0,4.9 7,10.48 9.75,12.46 V 25.77 c 0,-0.44 0.6,-8.55 3.65,-13.72 z"
+       transform="translate(-2.04,-1.15)"
+       style="fill:#364d5c" />
+    <path
+       id="path146"
+       d="m 12.1,54.12 v 0 C 11.74,53.88 5,49.41 2,44.24 v 11.14 a 8.2,8.2 0 0 0 3.64,6.3 l 13.5,7.79 c 4.28,2.46 12.7,-0.77 15.81,-2.12 z"
+       transform="translate(-2.04,-1.15)"
+       style="fill:#364d5c" />
+    <path
+       id="path150"
+       d="m 36.79,68 c -0.4,0.19 -7.71,3.75 -13.71,3.69 l 9.78,5.64 a 8.15,8.15 0 0 0 7.27,0 l 13.51,-7.8 c 4.27,-2.46 5.68,-11.39 6.06,-14.75 z"
+       transform="translate(-2.04,-1.15)"
+       style="fill:#364d5c" />
+    <path
+       id="path154"
+       d="M 61.2,27.13 V 53.6 c 0,0.44 -0.6,8.55 -3.65,13.72 l 9.77,-5.64 A 8.2,8.2 0 0 0 71,55.38 V 39.59 c 0,-4.94 -7,-10.5 -9.75,-12.46"
+       transform="translate(-2.04,-1.15)"
+       style="fill:#364d5c" />
+    <path
+       id="path158"
+       d="M 67.31,17.69 53.81,9.9 C 49.53,7.44 41.11,10.67 38,12 l 22.85,13.23 v 0 a 43.43,43.43 0 0 1 5.7,4.51 24,24 0 0 1 4.45,5.35 V 24 a 8.2,8.2 0 0 0 -3.64,-6.3"
+       transform="translate(-2.04,-1.15)"
+       style="fill:#364d5c" />
+  </g>
+  <g
+     id="re" />
+  <text
+     xml:space="preserve"
+     style="font-style:normal;font-weight:normal;font-size:40px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#364d5c;fill-opacity:1;stroke:none"
+     x="74.101189"
+     y="54.47554"
+     id="text3955"><tspan
+       id="tspan3953"
+       x="74.101189"
+       y="54.47554"
+       style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:44px;font-family:sans-serif;-inkscape-font-specification:'sans-serif, Bold';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:start;writing-mode:lr-tb;text-anchor:start;fill:#364d5c;fill-opacity:1"><tspan
+   style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold'"
+   id="tspan86">TDMtermite</tspan> </tspan></text>
+</svg>

cython/py_tdm_termite.pyx

@@ -1,17 +1,17 @@
 # distutils: language = c++
 
-from tdm_reaper cimport tdm_reaper
+from tdm_termite cimport tdm_termite
 import json as jn
 # import numpy as np
 
-cdef class tdmreaper:
+cdef class tdmtermite:
 
   # C++ instance of class => stack allocated (requires nullary constructor!)
-  cdef tdm_reaper cpp_tdm
+  cdef tdm_termite cpp_tdm
 
   # constructor
   def __cinit__(self, string tdmfile, string tdxfile):
-    self.cpp_tdm = tdm_reaper(tdmfile,tdxfile)
+    self.cpp_tdm = tdm_termite(tdmfile,tdxfile)
 
   # provide TDM files
   def submit_files(self,string tdmfile, string tdxfile):
@@ -36,9 +36,9 @@ cdef class tdmreaper:
     return jn.loads(chnstr.decode())
 
   # print a channel(-group)
-  def print_channelgroup(self, string id, const char* filename,
-                               bool include_meta, char delimiter):
-    self.cpp_tdm.print_group(id,filename,include_meta,delimiter)
+  def print_channelgroup(self, string id, const char* filename, bool include_meta,
+                               char delimiter, string column_header):
+    self.cpp_tdm.print_group(id,filename,include_meta,delimiter,column_header)
   def print_channel(self, string id, const char* filename,
                           bool include_meta):
     self.cpp_tdm.print_channel(id,filename,include_meta)
@@ -48,4 +48,4 @@ cdef class tdmreaper:
     grpids = self.cpp_tdm.get_channelgroup_ids()
     for id in grpids :
       grpfile = outputdir.decode() + "/channelgroup_" + id.decode() + ".csv"
-      self.cpp_tdm.print_group(id,grpfile.encode(),True,ord(','))
+      self.cpp_tdm.print_group(id,grpfile.encode(),True,ord(','),"".encode())

cython/setup.py

@@ -1,24 +1,47 @@
 from distutils.core import setup
 from distutils.extension import Extension
 from Cython.Build import cythonize
+import os
+import sys
+
+print("building on platform: "+sys.platform)
+os.system("git tag > gittags.log")
+with open ("gittags.log","r") as gt:
+    taglst = gt.readlines()
+os.remove("gittags.log")
+if len(taglst) > 0 :
+    version = taglst[-1].replace('\n','').replace('v','')
+else:
+    version = 'unkown'
+print("building version: "+version)
+
+if sys.platform == "linux" or sys.platform == "darwin" :
+	cmpargs = ['-std=c++17','-Wno-unused-variable']
+	lnkargs = ['-std=c++17','-Wno-unused-variable']
+elif sys.platform == "win32" :
+	cmpargs = ['/EHsc','/std:c++17']
+	lnkargs = []
+else :
+	raise RuntimeError("unknown platform") 
 
 extensions = Extension(
-    name="tdm_reaper",
-    sources=["cython/py_tdm_reaper.pyx"],
+    name="tdm_termite",
+    sources=["cython/py_tdm_termite.pyx"],
     # libraries=[""],
     # library_dirs=["lib"],
-    include_dirs=["lib","pugixml"],
+    include_dirs=["lib","3rdparty/pugixml"],
     language='c++',
-    extra_compile_args=['-std=c++17','-Wno-unused-variable'],
-    extra_link_args=['-std=c++17'],
+    #extra_compile_args=['-std=c++17','-Wno-unused-variable'],
+    extra_compile_args= cmpargs,
+    extra_link_args= lnkargs,
 )
 
 setup(
-    version='0.1',
-    description='TDMReaper cython extension',
+    version=version,
+    description='TDMtermite cython extension',
     author='Record Evolution GmbH',
     author_email='mario.fink@record-evolution.de',
-    url='https://github.com/RecordEvolution/tdm_ripper.git',
-    name="tdm_reaper",
+    url='https://github.com/RecordEvolution/TDMtermite.git',
+    name="tdm_termite",
     ext_modules=cythonize(extensions)
 )

cython/tdm_termite.pxd

@@ -5,14 +5,14 @@ from libcpp.string cimport string
 from libcpp.vector cimport vector
 from libcpp cimport bool
 
-cdef extern from "tdm_reaper.cpp":
+cdef extern from "tdm_termite.cpp":
   pass
 
-cdef extern from "tdm_reaper.hpp":
-  cdef cppclass tdm_reaper:
+cdef extern from "tdm_termite.hpp":
+  cdef cppclass tdm_termite:
     # constructor(s)
-    tdm_reaper() except +
-    tdm_reaper(string tdmfile, string tdxfile) except +
+    tdm_termite() except +
+    tdm_termite(string tdmfile, string tdxfile) except +
     # provide TDM files
     void submit_files(string tdmfile, string tdxfile) except+
     # get list of channel(-group) ids
@@ -24,5 +24,6 @@ cdef extern from "tdm_reaper.hpp":
     string get_channelgroup_info(string id) except+
     string get_channel_info(string id) except+
     # print a channel(-group)
-    void print_group(string id, const char* filename, bool include_meta, char delimiter) except+
+    void print_group(string id, const char* filename, bool include_meta,
+                     char delimiter, string column_header) except+
     void print_channel(string id, const char* filename, bool include_meta) except+

lib/tdm_datatype.hpp

@@ -17,6 +17,7 @@ typedef double eFloat64Usi;
 
 class tdmdatatype
 {
+
 protected:
   eInt16Usi sint16_;    // 0
   eInt32Usi sint32_;    // 1
@@ -26,6 +27,7 @@ protected:
   eFloat32Usi float32_; // 5
   eFloat64Usi float64_; // 6
   short int dtidx_;     // \in \{0,...,6\}
+
 public:
   tdmdatatype(): sint16_(0), sint32_(0),
                  uint8_(0), uint16_(0), uint32_(0),
@@ -43,6 +45,19 @@ public:
   // identify type
   short int& dtype() { return dtidx_; }
 
+  // copy constructor
+  tdmdatatype(const tdmdatatype &num)
+  {
+    this->sint16_ = num.sint16_;
+    this->sint32_ = num.sint32_;
+    this->uint8_ = num.uint8_;
+    this->uint16_ = num.uint16_;
+    this->uint32_ = num.uint32_;
+    this->float32_ = num.float32_;
+    this->float64_ = num.float64_;
+    this->dtidx_ = num.dtidx_;
+  }
+
   // overall assignment operator
   tdmdatatype& operator=(const tdmdatatype &num)
   {
@@ -55,9 +70,10 @@ public:
       this->uint32_ = num.uint32_;
       this->float32_ = num.float32_;
       this->float64_ = num.float64_;
+      this->dtidx_ = num.dtidx_;
     }
 
-		return *this;
+    return *this;
   }
 
   // implement assignment operator for individual datatypes

lib/tdm_termite.cpp

@@ -1,32 +1,97 @@
 // -------------------------------------------------------------------------- //
 
-#include "tdm_reaper.hpp"
+#include "tdm_termite.hpp"
 
 // -------------------------------------------------------------------------- //
 
-tdm_reaper::tdm_reaper()
+tdm_termite::tdm_termite()
 {
-
+  tdx_ifstream_ = new std::ifstream;
 }
 
-tdm_reaper::tdm_reaper(std::string tdmfile, std::string tdxfile, bool showlog):
+tdm_termite::tdm_termite(std::string tdmfile, std::string tdxfile, bool showlog):
   tdmfile_(tdmfile), tdxfile_(tdxfile)
 {
+  tdx_ifstream_ = new std::ifstream;
+
   // start processing tdm data model
   this->process_tdm(showlog);
 }
 
-void tdm_reaper::submit_files(std::string tdmfile, std::string tdxfile, bool showlog)
+tdm_termite::~tdm_termite()
+{
+  // close tdx-file stream and free memory
+  if ( tdx_ifstream_->is_open() ) tdx_ifstream_->close();
+  delete tdx_ifstream_;
+}
+
+tdm_termite::tdm_termite(const tdm_termite& other):
+  tdmfile_(other.tdmfile_), tdxfile_(other.tdxfile_), csvfile_(other.csvfile_),
+  endianness_(other.endianness_), machine_endianness_(other.machine_endianness_),
+  meta_data_(other.meta_data_), tdmdt_name_(other.tdmdt_name_),
+  tdmdt_chan_(other.tdmdt_chan_),
+  tdx_blocks_(other.tdx_blocks_), tdmroot_(other.tdmroot_),
+  tdmchannelgroups_(other.tdmchannelgroups_), tdmchannels_(other.tdmchannels_),
+  tdmchannels_data_(other.tdmchannels_data_), submatrices_(other.submatrices_),
+  localcolumns_(other.localcolumns_), tdxbuffer_(other.tdxbuffer_)
+{
+  tdx_ifstream_ = new std::ifstream;
+  if ( other.tdx_ifstream_->is_open() )
+  {
+    tdx_ifstream_->open(tdxfile_.c_str(),std::ifstream::binary);
+    tdx_ifstream_->seekg(other.tdx_ifstream_->tellg());
+  }
+}
+
+tdm_termite& tdm_termite::operator=(const tdm_termite& other)
+{
+  if ( this != &other )
+  {
+    tdmfile_ = other.tdmfile_;
+    tdxfile_ = other.tdxfile_;
+    csvfile_ = other.csvfile_;
+    endianness_ = other.endianness_;
+    machine_endianness_ = other.machine_endianness_;
+    meta_data_ = other.meta_data_;
+    tdmdt_name_ = other.tdmdt_name_;
+    tdmdt_chan_= other.tdmdt_chan_;
+    tdx_blocks_ = other.tdx_blocks_;
+    tdmroot_ = other.tdmroot_;
+    tdmchannelgroups_ = other.tdmchannelgroups_;
+    tdmchannels_ = other.tdmchannels_;
+    tdmchannels_data_ = other.tdmchannels_data_;
+    submatrices_ = other.submatrices_;
+    localcolumns_ = other.localcolumns_;
+    tdxbuffer_ = other.tdxbuffer_;
+    if ( other.tdx_ifstream_->is_open() )
+    {
+      tdx_ifstream_->open(tdxfile_.c_str(),std::ifstream::binary);
+      tdx_ifstream_->seekg(other.tdx_ifstream_->tellg());
+    }
+  }
+
+  return *this;
+}
+
+void tdm_termite::submit_files(std::string tdmfile, std::string tdxfile, bool showlog)
 {
   // save files
   tdmfile_ = tdmfile;
   tdxfile_ = tdxfile;
 
+  // reset and clear all members
+  tdx_blocks_.clear();
+  tdmchannelgroups_.clear();
+  tdmchannels_.clear();
+  submatrices_.clear();
+  localcolumns_.clear();
+  tdxbuffer_.clear();
+
   // start processing tdm data model
   this->process_tdm(showlog);
 }
 
-void tdm_reaper::process_tdm(bool showlog)
+void tdm_termite::process_tdm(bool showlog)
 {
   // check both tdm, tdx files
   std::filesystem::path ptdm(tdmfile_), ptdx(tdxfile_);
@@ -98,25 +163,31 @@ void tdm_reaper::process_tdm(bool showlog)
   this->process_localcolumns(showlog,xml_doc);
 
   // open .tdx and stream all binary data into buffer
+  // try {
+  //   std::ifstream fin(tdxfile_.c_str(),std::ifstream::binary);
+  //   // if ( !fin.good() ) std::cerr<<"failed to open .tdx-file\n";
+  //
+  //   std::vector<unsigned char> tdxbuf((std::istreambuf_iterator<char>(fin)),
+  //                                     (std::istreambuf_iterator<char>()));
+  //   tdxbuffer_ = tdxbuf;
+  //
+  //   if ( showlog ) std::cout<<"size of .tdx buffer (bytes): "<<tdxbuffer_.size()<<"\n\n";
+  //
+  //   // close .tdx file
+  //   fin.close();
+  // } catch (const std::exception& e ) {
+  //   throw std::runtime_error( std::string("failed to open .tdx and stream data to buffer: ")
+  //                             + e.what() );
+  // }
   try {
-    std::ifstream fin(tdxfile_.c_str(),std::ifstream::binary);
-    // if ( !fin.good() ) std::cerr<<"failed to open .tdx-file\n";
-
-    std::vector<unsigned char> tdxbuf((std::istreambuf_iterator<char>(fin)),
-                                      (std::istreambuf_iterator<char>()));
-    tdxbuffer_ = tdxbuf;
-
-    if ( showlog ) std::cout<<"size of .tdx buffer (bytes): "<<tdxbuffer_.size()<<"\n\n";
-
-    // close .tdx file
-    fin.close();
-  } catch (const std::exception& e ) {
-    throw std::runtime_error( std::string("failed to open .tdx and stream data to buffer: ")
+    tdx_ifstream_->open(tdxfile_.c_str(),std::ifstream::binary);
+  } catch (const std::exception& e) {
+    throw std::runtime_error( std::string("failed to open .tdx file in ifstream: ")
                               + e.what() );
   }
 }
 
-void tdm_reaper::process_include(bool showlog, pugi::xml_document& xml_doc)
+void tdm_termite::process_include(bool showlog, pugi::xml_document& xml_doc)
 {
   // get XML node
   pugi::xml_node tdmincl = xml_doc.child("usi:tdm").child("usi:include");
@@ -170,7 +241,7 @@ void tdm_reaper::process_include(bool showlog, pugi::xml_document& xml_doc)
   if ( showlog ) std::cout<<"number of blocks: "<<tdx_blocks_.size()<<"\n\n";
 }
 
-void tdm_reaper::process_root(bool showlog, pugi::xml_document& xml_doc)
+void tdm_termite::process_root(bool showlog, pugi::xml_document& xml_doc)
 {
   // get XML node
   pugi::xml_node tdmdataroot = xml_doc.child("usi:tdm").child("usi:data")
@@ -190,7 +261,7 @@ void tdm_reaper::process_root(bool showlog, pugi::xml_document& xml_doc)
   if ( showlog ) std::cout<<tdmroot_.get_info()<<"\n";
 }
 
-void tdm_reaper::process_channelgroups(bool showlog, pugi::xml_document& xml_doc)
+void tdm_termite::process_channelgroups(bool showlog, pugi::xml_document& xml_doc)
 {
   // get XML node <usi:data>
   pugi::xml_node tdmdata = xml_doc.child("usi:tdm").child("usi:data");
@@ -228,7 +299,7 @@ void tdm_reaper::process_channelgroups(bool showlog, pugi::xml_document& xml_doc
   if ( showlog ) std::cout<<"number of channelgroups: "<<tdmchannelgroups_.size()<<"\n\n";
 }
 
-void tdm_reaper::process_channels(bool showlog, pugi::xml_document& xml_doc)
+void tdm_termite::process_channels(bool showlog, pugi::xml_document& xml_doc)
 {
   // get XML node <usi:data>
   pugi::xml_node tdmdata = xml_doc.child("usi:tdm").child("usi:data");
@@ -272,7 +343,7 @@ void tdm_reaper::process_channels(bool showlog, pugi::xml_document& xml_doc)
   if ( showlog ) std::cout<<"number of channels: "<<tdmchannels_.size()<<"\n\n";
 }
 
-void tdm_reaper::process_submatrices(bool showlog, pugi::xml_document& xml_doc)
+void tdm_termite::process_submatrices(bool showlog, pugi::xml_document& xml_doc)
 {
   // get XML node <usi:data>
   pugi::xml_node tdmdata = xml_doc.child("usi:tdm").child("usi:data");
@@ -311,7 +382,7 @@ void tdm_reaper::process_submatrices(bool showlog, pugi::xml_document& xml_doc)
   if ( showlog ) std::cout<<"number of submatrices: "<<submatrices_.size()<<"\n\n";
 }
 
-void tdm_reaper::process_localcolumns(bool showlog, pugi::xml_document& xml_doc)
+void tdm_termite::process_localcolumns(bool showlog, pugi::xml_document& xml_doc)
 {
   // get XML node <usi:data>
   pugi::xml_node tdmdata = xml_doc.child("usi:tdm").child("usi:data");
@@ -374,6 +445,11 @@ void tdm_reaper::process_localcolumns(bool showlog, pugi::xml_document& xml_doc)
     {
       locc.values_ = vl.at(0);
     }
+    else if ( vl.size() == 0 )
+    {
+      //std::cerr<<"localcolumn ("<<locc.id_<<","<<locc.name_<<") misses any value-ids"<<"\n";
+      locc.values_ = "none";
+    }
     else
     {
       throw std::logic_error("localcolumn with out/multiple values id(s)");
@@ -389,7 +465,7 @@ void tdm_reaper::process_localcolumns(bool showlog, pugi::xml_document& xml_doc)
       {
         throw std::runtime_error(std::string("measurement_quantity: ")
                                  + locc.measurement_quantity_
-                                 + std::string(" is ambiguous") );
+                                 + std::string(" is missing/ambiguous") );
       }
       std::string dt = tdmchannels_.at(locc.measurement_quantity_).datatype_;
       std::string sequence_type;
@@ -411,8 +487,10 @@ void tdm_reaper::process_localcolumns(bool showlog, pugi::xml_document& xml_doc)
 
       if ( locc.external_id_.empty() )
       {
-        throw std::logic_error( std::string("no external id found for ")
-                              + sequence_type + std::string(" with ") + locc.values_ );
+        //throw std::logic_error( std::string("no external id found for ")
+        //                      + sequence_type + std::string(" with ") + locc.values_ );
+        //std::cerr<<"no external id found for "<<sequence_type<<" with "<<locc.values_<<"\n";
+        locc.external_id_ = "none";
       }
     }
 
@@ -427,7 +505,7 @@ void tdm_reaper::process_localcolumns(bool showlog, pugi::xml_document& xml_doc)
 
 // -------------------------------------------------------------------------- //
 
-std::string tdm_reaper::get_channel_overview(format chformatter)
+std::string tdm_termite::get_channel_overview(format chformatter)
 {
   // summarize all output in single string
   std::string channels_summary;
@@ -437,8 +515,8 @@ std::string tdm_reaper::get_channel_overview(format chformatter)
 
   // compose header
   chformatter.set_header(true);
-  tdm_channelgroup grp;
-  channels_summary += grp.get_info(chformatter);
+  //tdm_channelgroup grp;
+  //channels_summary += grp.get_info(chformatter);
   tdm_channel chn;
   channels_summary += chn.get_info(chformatter);
   std::string rule; // = std::string("#");
@@ -455,8 +533,9 @@ std::string tdm_reaper::get_channel_overview(format chformatter)
                                                    it!=tdmchannels_.end(); ++it)
   {
     // get corresponding group
-    tdm_channelgroup grp = tdmchannelgroups_.at(it->second.group_);
-    channels_summary += grp.get_info(chformatter);
+    // tdm_channelgroup grp = tdmchannelgroups_.at(it->second.group_);
+    // channels_summary += grp.get_info(chformatter);
+
     // ...and actual channel
     channels_summary += it->second.get_info(chformatter);
     channels_summary += std::string("\n");
@@ -466,7 +545,7 @@ std::string tdm_reaper::get_channel_overview(format chformatter)
 }
 
 template<typename tdmelement>
-std::string tdm_reaper::get_overview(format formatter)
+std::string tdm_termite::get_overview(format formatter)
 {
   // summarize all output in single string
   std::string summary;
@@ -492,12 +571,12 @@ std::string tdm_reaper::get_overview(format formatter)
   return summary;
 }
 
-template std::string tdm_reaper::get_overview<tdm_channelgroup>(format formatter);
-template std::string tdm_reaper::get_overview<submatrix>(format formatter);
-template std::string tdm_reaper::get_overview<localcolumn>(format formatter);
-template std::string tdm_reaper::get_overview<block>(format formatter);
+template std::string tdm_termite::get_overview<tdm_channelgroup>(format formatter);
+template std::string tdm_termite::get_overview<submatrix>(format formatter);
+template std::string tdm_termite::get_overview<localcolumn>(format formatter);
+template std::string tdm_termite::get_overview<block>(format formatter);
 
-void tdm_reaper::summarize_member(tdm_channelgroup chp, std::string& summary, format& formatter)
+void tdm_termite::summarize_member(tdm_channelgroup &chp, std::string& summary, format& formatter)
 {
   for ( std::map<std::string,tdm_channelgroup>::iterator it=this->tdmchannelgroups_.begin();
                                                          it!=this->tdmchannelgroups_.end(); ++it)
@@ -507,7 +586,7 @@ void tdm_reaper::summarize_member(tdm_channelgroup chp, std::string& summary, fo
   }
 }
 
-void tdm_reaper::summarize_member(submatrix sbm, std::string& summary, format& formatter)
+void tdm_termite::summarize_member(submatrix &sbm, std::string& summary, format& formatter)
 {
   for ( std::map<std::string,submatrix>::iterator it=this->submatrices_.begin();
                                                   it!=this->submatrices_.end(); ++it)
@@ -517,7 +596,7 @@ void tdm_reaper::summarize_member(submatrix sbm, std::string& summary, format& f
   }
 }
 
-void tdm_reaper::summarize_member(localcolumn lcc, std::string& summary, format& formatter)
+void tdm_termite::summarize_member(localcolumn &lcc, std::string& summary, format& formatter)
 {
   for ( std::map<std::string,localcolumn>::iterator it=this->localcolumns_.begin();
                                                     it!=this->localcolumns_.end(); ++it)
@@ -527,7 +606,7 @@ void tdm_reaper::summarize_member(localcolumn lcc, std::string& summary, format&
   }
 }
 
-void tdm_reaper::summarize_member(block blk, std::string& summary, format& formatter)
+void tdm_termite::summarize_member(block &blk, std::string& summary, format& formatter)
 {
   for ( std::map<std::string,block>::iterator it=this->tdx_blocks_.begin();
                                               it!=this->tdx_blocks_.end(); ++it)
@@ -540,7 +619,7 @@ void tdm_reaper::summarize_member(block blk, std::string& summary, format& forma
 // -------------------------------------------------------------------------- //
 
 // extract channel by id
-std::vector<tdmdatatype> tdm_reaper::get_channel(std::string& id)
+std::vector<tdmdatatype> tdm_termite::get_channel(std::string& id)
 {
   // check for existence of required channel id (=key)
   if ( tdmchannels_.count(id) == 1 )
@@ -548,7 +627,7 @@ std::vector<tdmdatatype> tdm_reaper::get_channel(std::string& id)
     // retrieve full channel info
     tdm_channel chn = tdmchannels_.at(id);
 
-    // extract (first) "localcolumn" for channel
+    // extract (first) "localcolumn" for channel TODO there should only be a single!! local_column!!
     if ( chn.local_columns_.size() != 1 )
     {
       throw std::runtime_error(std::string("invalid local_columns_ of channel: ") + id);
@@ -565,6 +644,12 @@ std::vector<tdmdatatype> tdm_reaper::get_channel(std::string& id)
     }
 
     // use "values" id to map to external block
+    if ( loccol.external_id_ == "none" )
+    {
+      //throw std::runtime_error(std::string("missing external_id in local_column ")+loccol.id_);
+      //std::cerr<<"missing external_id in local_column "<<loccol.id_<<"\n";
+      return std::vector<tdmdatatype>(0);
+    }
     block blk = tdx_blocks_.at(loccol.external_id_);
 
     // declare vector of appropriate length
@@ -576,10 +661,20 @@ std::vector<tdmdatatype> tdm_reaper::get_channel(std::string& id)
     // declare buffer covering the required range of "tdxbuffer_"
     // (consider both channel-wise and block-wise ordering)
     unsigned long int strtidx = blk.block_offset_*blk.block_size_
-                              + blk.byte_offset_,
-                      fnshidx = strtidx + blk.length_*dtyp.size_;
-    std::vector<unsigned char> tdxblk( tdxbuffer_.begin()+strtidx,
-                                       tdxbuffer_.begin()+fnshidx );
+                              + blk.byte_offset_;
+                      // fnshidx = strtidx + blk.length_*dtyp.size_;
+    // std::vector<unsigned char> tdxblk( tdxbuffer_.begin()+strtidx,
+    //                                    tdxbuffer_.begin()+fnshidx );
+    char* blkbuf = new char[blk.length_*dtyp.size_];
+    try {
+      tdx_ifstream_->seekg(strtidx);
+      tdx_ifstream_->read(blkbuf,blk.length_*dtyp.size_);
+    } catch (const std::exception& e) {
+      throw std::runtime_error( std::string("failed to read block from tdx_ifstream_: ")
+                                + e.what() );
+    }
+    std::vector<unsigned char> tdxblk(blkbuf,blkbuf+blk.length_*dtyp.size_);
+    delete []blkbuf;
 
     // distinguish numeric datatypes included in "tdmdatatype"
     if ( blk.value_type_ == std::string("eInt16Usi") )
@@ -645,7 +740,7 @@ std::vector<tdmdatatype> tdm_reaper::get_channel(std::string& id)
 
 // -------------------------------------------------------------------------- //
 
-void tdm_reaper::print_channel(std::string &id, const char* filename, bool include_meta)
+void tdm_termite::print_channel(std::string &id, const char* filename, bool include_meta)
 {
   // check required path
   this->check_filename_path(filename);
@@ -690,7 +785,8 @@ void tdm_reaper::print_channel(std::string &id, const char* filename, bool inclu
   }
 }
 
-void tdm_reaper::print_group(std::string &id, const char* filename, bool include_meta, char sep)
+void tdm_termite::print_group(std::string &id, const char* filename, bool include_meta,
+                             char sep, std::string column_header)
 {
   // check required path
   this->check_filename_path(filename);
@@ -716,6 +812,7 @@ void tdm_reaper::print_group(std::string &id, const char* filename, bool include
 
     int width = 25;
 
+    // file header
     if ( include_meta )
     {
       // group meta data
@@ -769,10 +866,34 @@ void tdm_reaper::print_group(std::string &id, const char* filename, bool include
     for ( std::string chn: chngrp.channels_ )
     {
       std::vector<tdmdatatype> chndat = this->get_channel(chn);
-      if ( chndat.size() > maxrows ) maxrows = chndat.size();
+      if ( chndat.size() > maxrows ) maxrows = (unsigned int)chndat.size();
       allchns.push_back(chndat);
     }
 
+    // provide column header (with channel ids)
+    if ( column_header.empty() )
+    {
+      for ( std::string chn: chngrp.channels_ )
+      {
+        // use given csv separator token
+        if ( sep == ' ' )
+        {
+          fou<<std::setw(width)<<std::left<<chn;
+        }
+        else
+        {
+          fou<<chn;
+        }
+        if ( chn != chngrp.channels_.back() && sep != ' ' ) fou<<sep;
+      }
+    }
+    else
+    {
+      fou<<column_header;
+    }
+    fou<<"\n";
+
+    // write data
     for ( unsigned int row = 0; row < maxrows; row++ )
     {
       for ( unsigned int chi = 0; chi < chngrp.channels_.size(); chi++ )
@@ -789,19 +910,8 @@ void tdm_reaper::print_group(std::string &id, const char* filename, bool include
             fou<<allchns.at(chi).at(row);
           }
         }
-        else
-        {
-          if ( sep == ' ' )
-          {
-            fou<<std::setw(width)<<std::left<<"";
-          }
-          else
-          {
-            fou<<sep;
-          }
-        }
 
-        if ( chi+1 < chngrp.channels_.size() ) fou<<sep;
+        if ( chi+1 < chngrp.channels_.size() && sep != ' ' ) fou<<sep;
       }
       fou<<"\n";
     }
@@ -811,7 +921,7 @@ void tdm_reaper::print_group(std::string &id, const char* filename, bool include
   }
 }
 
-void tdm_reaper::check_filename_path(const char* filename)
+void tdm_termite::check_filename_path(const char* filename)
 {
   // declare filesystem path instance from filename
   std::filesystem::path pt(filename);
@@ -821,13 +931,13 @@ void tdm_reaper::check_filename_path(const char* filename)
 
   if ( !std::filesystem::is_directory(pt) )
   {
-    throw std::runtime_error(std::string("directory does not exist: ") + pt.c_str() );
+    throw std::runtime_error( std::string("directory does not exist: ") + pt.u8string() );
   }
 }
 
 // -------------------------------------------------------------------------- //
 
-void tdm_reaper::check_local_datatypes()
+void tdm_termite::check_local_datatypes()
 {
   std::cout<<"\nmachine's C++ datatypes:\n";
   std::cout<<std::setw(25)<<std::left<<"char:"
@@ -858,7 +968,7 @@ void tdm_reaper::check_local_datatypes()
            <<std::setw(5)<<std::left<<sizeof(long double)<<"byte(s)\n\n";
 }
 
-void tdm_reaper::check_datatype_consistency()
+void tdm_termite::check_datatype_consistency()
 {
   // check datatype consistency, i.e. "local" representation of datatypes
   for ( tdm_datatype el: tdm_datatypes )
@@ -901,7 +1011,7 @@ void tdm_reaper::check_datatype_consistency()
 // -------------------------------------------------------------------------- //
 
 template<typename datatype>
-void tdm_reaper::convert_data_to_type(std::vector<unsigned char> &buffer,
+void tdm_termite::convert_data_to_type(std::vector<unsigned char> &buffer,
                                       std::vector<tdmdatatype> &channel)
 {
   // check number of elements of type "datatype" in buffer

lib/tdm_termite.hpp

@@ -1,7 +1,7 @@
 // ------------------------------------------------------------------------- //
 
-#ifndef TDM_REAPER
-#define TDM_REAPER
+#ifndef TDM_TERMITE
+#define TDM_TERMITE
 
 #include <iostream>
 #include <fstream>
@@ -24,7 +24,7 @@
 
 // -------------------------------------------------------------------------- //
 
-class tdm_reaper
+class tdm_termite
 {
   // .tdm and .tdx paths/filenames
   std::string tdmfile_;
@@ -39,13 +39,13 @@ class tdm_reaper
   // tdm meta-data
   tdm_meta meta_data_;
 
-  // blocks of data in .tdx file
-  std::map<std::string,block> tdx_blocks_;
-
   // resconstruct "tdm_datatype.hpp: tdm_datatypes" as map to quickly map
   // "valueType"/"channel_datatype" to full datatype
   std::map<std::string,tdm_datatype> tdmdt_name_, tdmdt_chan_;
 
+  // blocks of data in .tdx file
+  std::map<std::string,block> tdx_blocks_;
+
   // tdm root
   tdm_root tdmroot_;
 
@@ -54,13 +54,15 @@ class tdm_reaper
 
   // tdm channels
   std::map<std::string,tdm_channel> tdmchannels_;
+  std::map<std::string,std::vector<tdm_datatype>> tdmchannels_data_;
 
   // submatrices and local_columns
   std::map<std::string,submatrix> submatrices_;
   std::map<std::string,localcolumn> localcolumns_;
 
-  // binary data container
+  // binary data container/file stream
   std::vector<unsigned char> tdxbuffer_;
+  std::ifstream *tdx_ifstream_;
 
   // extract list of identifiers from e.g. "#xpointer(id("usi12") id("usi13"))"
   std::vector<std::string> extract_ids(std::string idstring)
@@ -114,12 +116,15 @@ class tdm_reaper
 public:
 
   // encoding
-  // tdm_reaper(std::vector<std::string> csvfile);
+  // tdm_termite(std::vector<std::string> csvfile);
 
   // decoding
-  tdm_reaper();
-  tdm_reaper(std::string tdmfile, std::string tdxfile = std::string(""),
+  tdm_termite();
+  tdm_termite(std::string tdmfile, std::string tdxfile = std::string(""),
              bool showlog = false);
+  ~tdm_termite();
+  tdm_termite(const tdm_termite& other);
+  tdm_termite& operator=(const tdm_termite& other);
 
   // provide (tdm,tdx) files
   void submit_files(std::string tdmfile, std::string tdxfile = std::string(""),
@@ -179,10 +184,10 @@ public:
   template<typename tdmelement>
   std::string get_overview(format formatter);
 private:
-  void summarize_member(tdm_channelgroup chp, std::string& summary, format& formatter);
-  void summarize_member(submatrix sbm, std::string& summary, format& formatter);
-  void summarize_member(localcolumn lcc, std::string& summary, format& formatter);
-  void summarize_member(block blk, std::string& summary, format& formatter);
+  void summarize_member(tdm_channelgroup &chp, std::string& summary, format& formatter);
+  void summarize_member(submatrix &sbm, std::string& summary, format& formatter);
+  void summarize_member(localcolumn &lcc, std::string& summary, format& formatter);
+  void summarize_member(block &blk, std::string& summary, format& formatter);
 public:
 
   // get list of channelgroup ids
@@ -238,7 +243,8 @@ public:
 
   // dump a single channel/entire group (identified by id) to file
   void print_channel(std::string &id, const char* filename, bool include_meta = true);
-  void print_group(std::string &id, const char* filename, bool include_meta = true, char sep = ' ');
+  void print_group(std::string &id, const char* filename, bool include_meta = true,
+                   char sep = ' ', std::string column_header = std::string(""));
   void check_filename_path(const char* filename);
 
   // check machine's datatypes

makefile

@@ -1,10 +1,12 @@
 # --------------------------------------------------------------------------- #
 
+SHELL := /bin/bash
+
 # declare name of executable
-EXE = tdmreaper
+EXE = tdmtermite
 
 # sources and headers
-SRC := tdm_reaper
+SRC := tdm_termite
 HPP = $(wildcard lib/*.hpp)
 
 # compiler and C++ standard
@@ -13,18 +15,38 @@ CC = g++ -std=c++17
 # compiler options and optimization flags
 OPT = -O3 -Wall -Werror -Wunused-variable -Wsign-compare
 
-# include library path
-LIB = pugixml/
+# include 3rd party libraries paths
+LIBB := 3rdparty/
+LIB := $(foreach dir,$(shell ls $(LIBB)),-I $(LIBB)$(dir))
 
 # determine git version/commit tag
-GTAG := $(shell git tag | head -n1)
+GTAG := $(shell git tag | tail -n1)
+GTAGV := $(shell git tag | tail -n1 | tr -d 'v')
 GHSH := $(shell git rev-parse HEAD | head -c8)
+PIPYVS := $(shell grep "version" pip/setup.py | tr -d 'version=,\#\" ')
+PICGVS := $(shell grep "version" pip/setup.cfg | tr -d 'version=,\#\" ')
+
+# current timestamp
+TMS = $(shell date +%Y%m%dT%H%M%S)
 
 # define install location
 INST := /usr/local/bin
 
-# platform
+# platform and current working directory
 OST := $(shell uname)
+CWD := $(shell pwd)
+
+# --------------------------------------------------------------------------- #
+# version/tag check
+
+checkversion:
+	@echo "git tag:                "$(GTAG)
+	@echo "git head:               "$(GHSH)
+	@echo "pip setup.py version:   "$(PIPYVS)
+	@echo "pip setup.cfg version:  "$(PICGVS)
+
+$(GTAGV):
+	@echo "check consistent versions (git tag vs. setup.py): "$(GTAG)" <-> "$(PIPYVS)" "
 
 # --------------------------------------------------------------------------- #
 # CLI tool
@@ -44,19 +66,27 @@ main.o : src/main.cpp lib/$(SRC).hpp $(HPP)
 	@if [ $(OST) = "Linux" ]; then\
 		sed -i 's/TAGSTRING/$(GTAG)/g' $<.cpp; \
 		sed -i 's/HASHSTRING/$(GHSH)/g' $<.cpp; \
+		sed -i 's/TIMESTAMPSTRING/$(TMS)/g' $<.cpp; \
 	fi
 	@if [ $(OST) = "Darwin" ]; then\
 		sed -i '' 's/TAGSTRING/$(GTAG)/g' $<.cpp; \
 		sed -i '' 's/HASHSTRING/$(GHSH)/g' $<.cpp; \
+		sed -i '' 's/TIMESTAMPSTRING/$(TMS)/g' $<.cpp; \
 	fi
-	$(CC) -c $(OPT) -I $(LIB) -I lib/ $<.cpp -o $@
+	$(CC) -c $(OPT) $(LIB) -I lib/ $<.cpp -o $@
 	@rm $<.cpp
 
 $(SRC).o : lib/$(SRC).cpp lib/$(SRC).hpp $(HPP)
-	$(CC) -c $(OPT) -I $(LIB) $< -o $@
+	$(CC) -c $(OPT) $(LIB) $< -o $@
 
-clean-cpp :
-	rm -f $(EXE) *.o src/main.cpp.cpp
+tdmtest : tdmtest.o
+	$(CC) $(OPT) $^ -o $@
+
+tdmtest.o : src/test.cpp lib/$(SRC).hpp $(HPP)
+	$(CC) -c $(OPT) $(LIB) -I lib/ $< -o $@
+
+cpp-clean :
+	rm -f $(EXE) *.o src/main.cpp.cpp tdmtest
 
 # --------------------------------------------------------------------------- #
 # check process
@@ -77,20 +107,44 @@ cython-help : cython/setup.py
 cython-list : cython/setup.py
 	python3 $< --name --description --author --author-email --url
 
-cython-build : cython/setup.py cython/tdm_reaper.pxd cython/py_tdm_reaper.pyx $(HPP) lib/tdm_reaper.cpp
+cython-build : cython/setup.py cython/tdm_termite.pxd cython/py_tdm_termite.pyx $(HPP) lib/tdm_termite.cpp
 	python3 $< build_ext --inplace
-	cp -v tdm_reaper.cpython-*.so python/
+	cp -v tdm_termite.cpython-*.so python/
 
-cython-install : cython/setup.py cython/tdm_reaper.pxd cython/py_tdm_reaper.pyx $(HPP) lib/tdm_reaper.cpp
+cython-install : cython/setup.py cython/tdm_termite.pxd cython/py_tdm_termite.pyx $(HPP) lib/tdm_termite.cpp
 	python3 $< install
 
-clean-cython :
-	rm -vf cython/py_tdm_reaper.cpp
-	rm -vf tdm_reaper.cpython-*.so python/tdm_reaper.cpython-*.so
+cython-clean :
+	rm -vf cython/py_tdm_termite.c* cython/tdm_termite.c*
+	rm -vf tdm_termite.cpython-*.so python/tdm_termite.cpython-*.so
 	rm -rf build
 
 # --------------------------------------------------------------------------- #
+# docker
 
-clean : clean-cpp clean-cython
+docker-build:
+	docker build . --tag tdmtermite:latest
+
+docker-run:
+	mkdir -pv data/{input,output}
+	docker run -it --rm --volume $(CWD)/data:/home/data tdmtermite:latest /bin/bash
+
+docker-clean:
+	rm -rv data
+	docker image remove tdmtermite
+
+# --------------------------------------------------------------------------- #
+# pip
+
+pip-publish: $(PIPYVS) cython-build
+	cd pip/ && make pip-publish
+
+pip-test:
+	cd pip/ && make pip-test
+
+# --------------------------------------------------------------------------- #
+
+clean : cpp-clean cython-clean
+	cd pip/ && make pip-clean
 
 # --------------------------------------------------------------------------- #

pip/Dockerfile

@@ -0,0 +1,15 @@
+
+FROM debian:bullseye
+
+RUN DEBIAN_FRONTEND=noninteractive \
+  apt-get update && apt-get upgrade -y && apt-get install -y \
+  python3 \
+  python3-pip \
+  python3-setuptools
+
+# RUN python3 -m pip install -i https://test.pypi.org/simple/ TDMtermite-RecordEvolution==0.6.7 && \
+#   python3 -m pip install --upgrade TDMtermite-RecordEvolution
+RUN python3 -m pip install TDMtermite
+
+# CMD ["sleep","3600"]
+CMD ["python3","-m","pip","freeze"]

pip/Dockerfile-manylinux

@@ -0,0 +1,25 @@
+
+FROM quay.io/pypa/manylinux2010_x86_64
+
+RUN yum -y install \
+    git \
+    python34-devel \
+    gcc-c++ \
+    python34-tkinter \
+    make \
+    python34-pip \
+    
+RUN sudo yum -y install rh-python36-python-devel.x86_64 rh-python35-python-devel.x86_64 python27-python-devel.x86_64
+
+ENV PATH /opt/rh/python27/root/usr/bin:/opt/rh/rh-python35/root/usr/bin/:/opt/rh/rh-python36/root/usr/bin/:$PATH
+
+RUN  python3.6 -m pip install --user --upgrade cython wheel twine auditwheel
+
+WORKDIR /home/${USERNAME}/neuron-yale
+
+RUN  git checkout setuppy \
+     && python3.6 setup.py bdist_wheel
+
+# repair wheel? see : https://malramsay.com/post/perils_of_packaging/
+RUN python3.6 -m pip install --user --upgrade auditwheel
+RUN LD_LIBRARY_PATH=`pwd`/_install/lib auditwheel repair dist/NEURON-7.8-cp36-cp36m-linux_x86_64.whl

pip/MANIFEST.in

@@ -0,0 +1,4 @@
+include *.hpp
+include *.cpp
+include *.pyx
+include *.pxd

pip/bkup.pyproject.toml

@@ -0,0 +1,4 @@
+[build-system]
+requires = [
+    "setuptools"
+]

pip/makefile

@@ -0,0 +1,62 @@
+# --------------------------------------------------------------------------- #
+
+pip-publish: pip-sdist pip-upload
+
+pip-sdist: ../cython/py_tdm_termite.pyx ../cython/tdm_termite.pxd ../cython/py_tdm_termite.cpp
+	cp -v ../cython/py_tdm_termite.pyx ../cython/tdm_termite.pxd ./
+	cp -v ../cython/py_tdm_termite.cpp ./
+	cp -v ../lib/*.hpp ../lib/*.cpp ./
+	cp -v ../3rdparty/pugixml/* ./
+	cat ../README.md | grep '^# TDMtermite' -A 50000 > ./README.md
+	cp -v ../LICENSE ./
+	# cython py_tdm_termite.pyx -o py_tdm_termite.cpp
+	python3 setup.py sdist
+
+pip-setup:
+	apt-get install -y python3-setuptools \
+	                   python3-pip \
+	                   python3-venv
+	python3 -m pip install --upgrade build
+	python3 -m pip install twine wheel auditwheel cython
+	python3 -m pip install --user --upgrade twine
+
+pip-build:
+	#python3 -m build
+	# python3 setup.py sdist bdist_wheel
+	python3 setup.py bdist_wheel
+
+# actually it seems we have to use CentOS container
+# docker run -i -t -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /bin/bash
+# see https://github.com/neuronsimulator/nrn/issues/329 for setup of the container
+
+pip-audit:
+	auditwheel repair $(shell find dist/ -name "*-cp38-cp38-linux_x86_64.whl")
+
+# username: __token__
+# password: API-token including "pypi-"
+# !! RUN AS ROOT!! (only for bdist_wheel + auditwheel)
+pip-upload-test:
+	python3 -m twine upload --repository testpypi dist/$(shell ls -t dist/ | head -n1)
+
+pip-upload:
+	python3 -m twine upload dist/$(shell ls -t dist/ | head -n1)
+
+pip-test-install:
+	python3 -m pip install --index-url https://test.pypi.org/simple --no-deps TDMtermite-RecordEvolution
+	# python3 -m pip install -i https://test.pypi.org/simple/ TDMtermite-RecordEvolution==0.5
+
+pip-test:
+	docker build . --tag tdmtermite-piptest
+	docker run -it --rm tdmtermite-piptest
+
+pip-clean:
+	rm -rvf dist/
+	rm -rvf *.egg-info
+	rm -rvf build/
+	rm -rvf cython/
+	rm -rvf wheelhouse/
+	rm -vf *.pyx *.pxd
+	rm -vf *.cpp *.c *.hpp
+	rm -vf README.md LICENSE
+
+# --------------------------------------------------------------------------- #

pip/setup.cfg

@@ -0,0 +1,21 @@
+[metadata]
+name = TDMtermite-RecordEvolution
+version = 1.0.1
+author = Record Evolution GmbH
+author_email = mario.fink@record-evolution.de
+maintainer = Record Evolution GmbH
+license = MIT
+description = Extract and read data from National Instruments LabVIEW tdx/tdm files and export them as csv files
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://github.com/RecordEvolution/TDMtermite.git
+project_urls =
+    Bug Tracker = https://github.com/RecordEvolution/TDMtermite/issues
+classifiers =
+    Programming Language :: Python :: 3
+    License :: OSI Approved :: MIT License
+    Operating System :: OS Independent
+
+[options]
+packages = find:
+python_requires = >=3.6

pip/setup.py

@@ -0,0 +1,49 @@
+
+from setuptools import setup, Extension
+import sys
+
+print("building on platform: "+sys.platform)
+
+if sys.platform == "linux" or sys.platform == "darwin" :
+	cmpargs = ['-std=c++17','-Wno-unused-variable']
+	lnkargs = ['-std=c++17','-Wno-unused-variable']
+elif sys.platform == "win32" :
+	cmpargs = ['/EHsc','/std:c++17']
+	lnkargs = []
+else :
+	raise RuntimeError("unknown platform")
+
+with open("README.md", "r", encoding="utf-8") as fh:
+    long_description = fh.read()
+
+setup(
+    name="TDMtermite",
+    version="1.0.3",
+    author="Record Evolution GmbH",
+    author_email="mario.fink@record-evolution.de",
+    maintainer="Record Evolution GmbH",
+    license="MIT",
+    description="Extract and read data from National Instruments LabVIEW tdx/tdm files and export them as csv files",
+    keywords="TDM/TDX NationalInstruments LabVIEW decode",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/RecordEvolution/TDMtermite.git",
+    project_urls={
+        "Bug Tracker": "https://github.com/RecordEvolution/TDMtermite/issues",
+    },
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+    ],
+    ext_modules=[Extension("tdm_termite",
+                           ["py_tdm_termite.cpp"],
+                           # libraries_dirs = ["cython/"],
+                           # include_dirs = ["3rdparty/pugixml/","lib/"],
+                           # depends = ["../lib/tdm_termite.hpp"]
+                           language = 'c++',
+                           extra_compile_args = cmpargs,
+                           extra_link_args = lnkargs,
+                           )
+	],
+)

python/custom.py

@@ -0,0 +1,50 @@
+
+import tdm_termite
+import json
+import re
+
+# create 'tdm_termite' instance object
+try :
+    jack = tdm_termite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')
+except RuntimeError as e :
+    print("failed to load/decode TDM files: " + str(e))
+
+# list ids of channelgroups
+grpids = jack.get_channelgroup_ids()
+
+# iterate through group-ids
+for grp in grpids[0:2] :
+
+    # obtain meta data of channelgroups
+    grpinfo = jack.get_channelgroup_info(grp)
+    print( json.dumps(grpinfo,sort_keys=False,indent=4) )
+
+    # get channel meta-data and compose header
+    column_header = ""
+    chns = grpinfo['channels'].split(' ')
+    for chn in chns :
+        # obtain channel's meta-data as dictionary (JSON)
+        chninfo = jack.get_channel_info(chn.encode())
+        # print( json.dumps(chninfo,sort_keys=False,indent=4) )
+        # choose e.g. channel-id and name to compose column header
+        chnhead = ( str(chninfo['channel-id']) + "("
+                  + str(chninfo['name']).replace(',',' ') + ")" )
+        # append to full header
+        column_header = column_header + chnhead + ","
+    # remove last comma
+    column_header = column_header[0:-1]
+
+    print("column_header:\n"+column_header+"\n")
+
+    # write channelgroup to file
+    try :
+        grpname = re.sub('[^A-Za-z0-9]','',grpinfo['name'])
+        grpfile = str("./channelgroup_") + str(grp.decode()) + "_" + str(grpname) + str(".csv")
+        jack.print_channelgroup(grp,                   # id of group to be written
+                                grpfile.encode(),      # filename
+                                False,                 # no meta-data header
+                                ord(','),              # csv separator character
+                                column_header.encode() # provide column header
+                                )
+    except RuntimeError as e :
+        print("failed to print channelgroup: " + str(grp) + " : " + str(e))

python/minimal.py

@@ -1,15 +1,20 @@
 
-import tdm_reaper
+import tdm_termite
 
-# create 'tdm_reaper' instance object
+# create 'tdm_termite' instance object
 try :
-    jack = tdm_reaper.tdmreaper(b'samples/SineData.tdm',b'samples/SineData.tdx')
+    jack = tdm_termite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')
     # list ids of channelgroups
     grpids = jack.get_channelgroup_ids()
     # iterate through groups
     for grp in grpids :
         # write channelgroup to file (prepend './' for local directory!!)
         grpfile = "./channelgroup_" + str(grp.decode()) + ".csv"
-        jack.print_channelgroup(grp,grpfile.encode(),True,ord(' '))
+        jack.print_channelgroup(grp,              # id of group to be written
+                                grpfile.encode(), # filename
+                                False,            # include meta-data fileheader?
+                                ord(','),         # csv separator character
+                                b""               # (empty=default) column header
+                                )
 except RuntimeError as e :
     print("failed to load/decode/write TDM files: " + str(e))

python/usage.py

@@ -1,12 +1,12 @@
 
-import tdm_reaper
+import tdm_termite
 # import numpy as np
 import json
 import re
 
-# create 'tdm_reaper' instance object
+# create 'tdm_termite' instance object
 try :
-    jack = tdm_reaper.tdmreaper(b'samples/SineData.tdm',b'samples/SineData.tdx')
+    jack = tdm_termite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')
 except RuntimeError as e :
     print("failed to load/decode TDM files: " + str(e))
 
@@ -24,8 +24,8 @@ for grp in grpids[0:2] :
     # write channelgroup to file
     try :
         grpname = re.sub('[^A-Za-z0-9]','',grpinfo['name'])
-        grpfile = "channelgroup_" + str(grp) + "_" + str(grpname) + ".csv"
-        jack.print_channelgroup(grp.encode(),grpfile.encode(),True,ord(' '))
+        grpfile = "./channelgroup_" + str(grp) + "_" + str(grpname) + ".csv"
+        jack.print_channelgroup(grp.encode(),grpfile.encode(),True,ord(' '),b'')
     except RuntimeError as e :
         print("failed to print channelgroup: " + str(grp) + " : " + str(e))
 
@@ -49,7 +49,7 @@ for chn in chnids[0:2] :
     print(str(chndata[0:6]) + " ...")
 
     # write channel to file
-    chnfile = "channel_" + str(chn) + ".csv"
+    chnfile = "./channel_" + str(chn) + ".csv"
     try :
         jack.print_channel(chn.encode(),chnfile.encode(),True)
     except RuntimeError as e :

src/main.cpp

@@ -1,6 +1,6 @@
 // ------------------------------------------------------------------------- //
 
-#include "tdm_reaper.hpp"
+#include "tdm_termite.hpp"
 
 #include <filesystem>
 #include <regex>
@@ -11,16 +11,17 @@
 
 const std::string gittag("TAGSTRING");
 const std::string githash("HASHSTRING");
+const std::string timestamp("TIMESTAMPSTRING");
 
 void show_usage()
 {
   std::cout<<"\n"
-           <<"tdmreaper ["<<gittag<<"-g"<<githash<<"] (github.com/RecordEvolution/tdm_ripper.git)"
+           <<"tdmtermite ["<<gittag<<"-g"<<githash<<"-"<<timestamp<<"] (https://github.com/RecordEvolution/TDMtermite.git)"
            <<"\n\n"
            <<"Decode TDM/TDX files and dump data as *.csv"
            <<"\n\n"
            <<"Usage:\n\n"
-           <<" tdmreaper <tdm-file> <tdx-file> [options]"
+           <<" tdmtermite <tdm-file> <tdx-file> [options]"
            <<"\n\n"
            <<"Options:"
            <<"\n\n"
@@ -55,7 +56,7 @@ void show_usage()
 typedef std::map<std::string,std::string> optkeys;
 
 const std::string argmsg = std::string("both .tdm and .tdx file must be provided!");
-const std::string arguse = std::string("see $ tdmreaper --help for usage");
+const std::string arguse = std::string("see $ tdmtermite --help for usage");
 
 optkeys parse_args(int argc, char* argv[], bool showargs = false)
 {
@@ -78,7 +79,7 @@ optkeys parse_args(int argc, char* argv[], bool showargs = false)
     else if ( std::string(argv[1]) == std::string("--version")
            || std::string(argv[1]) == std::string("-v") )
     {
-      std::cout<<"tdmreaper "<<gittag<<"-g"<<githash<<"\n";
+      std::cout<<"tdmtermite "<<gittag<<"-g"<<githash<<"-"<<timestamp<<"\n";
     }
     else
     {
@@ -270,8 +271,8 @@ int main(int argc, char* argv[])
     std::string csvsep = cfgopts.count("csvsep") == 1 ? cfgopts.at("csvsep")
                                                       : std::string(",");
 
-    // declare and initialize tdm_reaper instance
-    tdm_reaper jack;
+    // declare and initialize tdm_termite instance
+    tdm_termite jack;
     try {
       jack.submit_files(cfgopts.at("tdm"),cfgopts.at("tdx"),false);
     } catch (const std::exception& e) {
@@ -284,13 +285,13 @@ int main(int argc, char* argv[])
     if ( showroot ) std::cout<<"\n"<<jack.get_root().get_info()<<"\n";
 
     // get complete channel(-group) overview
-    format grpformatter(26,false,false,' ');
+    format grpformatter(16,false,false,' ');
     if (listgroups) std::cout<<"\n"<<jack.get_overview<tdm_channelgroup>(grpformatter)<<"\n";
-    format chformatter(14,false,false,' ');
+    format chformatter(16,false,false,' ');
     if (listchannels) std::cout<<"\n"<<jack.get_channel_overview(chformatter)<<"\n";
 
     // get complete submatrix/localcolumns overview
-    format formatter(18,false,false,' ');
+    format formatter(16,false,false,' ');
     if (listblocks) std::cout<<jack.get_overview<block>(formatter)<<"\n";
     if (listsubmatrices) std::cout<<jack.get_overview<submatrix>(formatter)<<"\n";
     if (listlocalcolumns) std::cout<<jack.get_overview<localcolumn>(formatter)<<"\n";

src/test.cpp

@@ -0,0 +1,33 @@
+// ------------------------------------------------------------------------- //
+
+#include "tdm_termite.hpp"
+
+#include <filesystem>
+#include <regex>
+#include <thread>
+#include <chrono>
+
+// ------------------------------------------------------------------------- //
+
+int main(int argc, char* argv[])
+{
+  std::string tdmfile(argv[1]);
+  std::string tdxfile(argv[2]);
+
+  std::cout<<tdmfile<<"\n"<<tdxfile<<"\n";
+
+  pugi::xml_document xml_doc;
+  pugi::xml_parse_result xml_result;
+
+  // load XML document from stream
+  std::ifstream fin(tdmfile.c_str());
+  xml_result = xml_doc.load(fin);
+  fin.close();
+
+  std::cout<<"\nloading "<<tdmfile<<": "<<xml_result.description()<<"\n";
+  std::cout<<"encoding: "<<(pugi::xml_encoding)xml_result.encoding<<"\n\n";
+
+  std::this_thread::sleep_for(std::chrono::milliseconds(4000));
+
+  return 0;
+}