Compare commits

..

34 Commits

Author SHA1 Message Date
538b6f4659 README.md: add famosfilekey reference 2021-06-29 11:45:50 +02:00
Mario Fink
08e2589dfd README: fix typo 2021-06-28 16:38:23 +02:00
Mario Fink
d098273816 README.md: udpate to new CLI options 2021-06-28 16:37:16 +02:00
Mario Fink
2f9c550323 bump pip/cython version 1.2.5 2021-06-28 16:33:25 +02:00
Mario Fink
234876c5a9 * cython/*_imc_termite.*: introduce both print_channel/print_channels
methods featuring csv delimiter option
* imc_raw.hpp: add print_channel()
* python/{example.py,usage.py}: adjust to existing sample file names,
  add print_channel() example
* src/main.cpp: add --delimiter option, additional CLI option checks
2021-06-28 16:26:07 +02:00
Mario Fink
bafc018566 CLI version: add option for delimiter token of csv output 2021-06-28 12:51:09 +02:00
fdd107fbb3 README.md: fix typo 2021-05-06 18:11:40 +02:00
cd4ce55e7e badge starts 2021-05-06 16:01:07 +02:00
27683cbbd6 license badge 2021-05-06 15:59:43 +02:00
25a51215a0 adjust README.md header 2021-05-06 15:50:38 +02:00
36c02089aa clean up README.md header 2021-05-06 15:47:31 +02:00
cd75689d32 lgtm code stats 2021-05-06 11:46:43 +02:00
57c7f6021b makefile: cppcheck target 2021-05-05 15:58:10 +02:00
83922c343f * imc_channel, imc_raw: optimize with pass by reference
* imc_object: asc_time, localtime: make threadsafe
* imc_datatype: satisfy 'rule of two'
* python: remove all unused imports
2021-05-05 13:28:11 +02:00
72378877ec add some pipy release documentation 2021-05-02 19:57:13 +02:00
e7094d0125 * py_imc_termite.pyx: remove json loads parse_float option
* imc_raw.hpp: catch empty/invalid file to avoid seg fault
2021-05-02 19:34:08 +02:00
16a77ecf1e bump version 1.2.4 for pip/cython 2021-04-30 11:10:41 +02:00
798fc22d7a imc_conversion.hpp: more comprehensive error message, imc_datatype: imc_Ulongint,imc_Slongint typedef for x86_64 vs. arm 2021-04-27 12:42:24 +02:00
2b2e69f0e5 * add support for datatype 11 = 2-byte-word digital
* usage.py: raise proper exception
* usage_adv.py: use different example file
2021-04-27 11:38:43 +02:00
Mario Fink
8708d2d008 * cython/py_imc_termite.pyx: json loads take care of floats, fix column assignment
* imc_channel.hpp: fix floating precision for get_channel()
* extend/include advanced usage in examples
2021-04-26 19:41:56 +02:00
9feadb50c1 py_imc_termite.pyx: fix special character encoding 2021-04-16 09:37:58 +02:00
dbeb1b862a py_imc_termite.pyx: print_table: encoding, xdata 2021-04-15 23:51:44 +02:00
bda799832c fix Dockerfile, cython: print_table to print all channels in single table 2021-04-15 20:17:02 +02:00
97c2195f98 ignore hexshow.hpp, version 1.2.3 2021-03-05 09:19:06 +01:00
ea7ab9fa85 gitignore: add pip files, pip: add proper README 2021-03-04 18:48:59 +01:00
c281aeae19 clean: include build directory 2021-03-04 11:48:06 +01:00
0403f2bc64 add pip package build/upload, check version tags 2021-03-04 11:47:15 +01:00
Mario Fink
0969d626b8 Merge pull request #5 from ZornitsaD/patch-2
Update README.md
2021-02-18 11:38:42 +01:00
ZornitsaD
37cbc2d45f Update README.md 2021-02-18 10:50:25 +01:00
8c8ed10a90 README.md: some minor tweak 2021-02-17 10:23:53 +01:00
Mario Fink
ecc1cd8005 Merge pull request #4 from ZornitsaD/patch-1
Update README.md
2021-02-16 12:39:45 +01:00
ZornitsaD
8746519d1a Update README.md
Hi Mario, I've had a look at the file and made some minor adjustments (punctuation mostly). Marko has asked us to mention the platform in the README files of the open-source projects with a brief explanation of where TDMtermite comes from and how we use it in the platform. I have to say, however, that I'm a bit out of my depth here.
2021-02-16 12:33:36 +01:00
c9f8097c9b README: fix link 2021-02-12 13:32:05 +01:00
5ae6b3f5ba add references 2021-02-12 13:29:53 +01:00
21 changed files with 444 additions and 146 deletions

9
.gitignore vendored
View File

@@ -20,3 +20,12 @@ cython/*.cpp
*.parquet
src/*.cpp.cpp
pip/*.cpp
pip/*.hpp
pip/*.pyx
pip/*.pxd
pip/README.md
pip/LICENSE
pip/*egg-info
pip/dist/

View File

@@ -4,8 +4,9 @@ FROM debian:bullseye-20210111
USER root
RUN apt-get update && apt-get install -y \
build-essential git \
build-essential git vim \
python3 python3-pip
RUN python3 -m pip install cython
RUN g++ -v
@@ -15,6 +16,6 @@ COPY ./ /IMCtermite/
RUN cd /IMCtermite && ls -lh && make install && ls -lh /usr/local/bin/imctermite
# install Python module
# RUN cd /tdm_ripper && ls -lh && make cython-requirements && make cython-install
RUN cd /IMCtermite && ls -lh && make cython-install
CMD ["sleep","inifity"]
CMD ["sleep","infinity"]

103
README.md
View File

@@ -1,25 +1,25 @@
<p align="center">
<a href="https://record-evolution.de/reswarm">
<img
alt="imctermite.svg"
src="assets/imctermite.svg"
width="400"
/>
</a>
</p>
[![Total alerts](https://img.shields.io/lgtm/alerts/g/RecordEvolution/IMCtermite.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/RecordEvolution/IMCtermite/alerts/)
[![Language grade: C/C++](https://img.shields.io/lgtm/grade/cpp/g/RecordEvolution/IMCtermite.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/RecordEvolution/IMCtermite/context:cpp)
[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/RecordEvolution/IMCtermite.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/RecordEvolution/IMCtermite/context:python)
[![LICENSE](https://img.shields.io/github/license/RecordEvolution/IMCtermite)](https://img.shields.io/github/license/RecordEvolution/IMCtermite)
[![STARS](https://img.shields.io/github/stars/RecordEvolution/IMCtermite)](https://img.shields.io/github/stars/RecordEvolution/IMCtermite)
# IMCtermite
_IMCtermite_ provides access to the proprietary data format
_IMC Bus Format_ with file extension _.raw_ introduced and developed by
_IMC Bus Format_ with the file extension _.raw_ introduced and developed by
[imc Test & Measurement GmbH](https://www.imc-tm.de/). This data format is
employed i.a. by the measurement hardware
[imc CRONOSflex](https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/)
to dump and store data and the software packages
[imc Studio](https://www.imc-tm.de/produkte/messtechnik-software/imc-studio/ueberblick/)
and [imc FAMOS](https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/)
for measurement data control and analysis. The extracted measurement data can
be stored in various open source file formats like _csv_, _json_, _parquet_ or
_hdf5_.
& [imc FAMOS](https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/)
for measurement data control and analysis. Thanks to the integrated Python module,
the extracted measurement data can be stored in any open-source file format
accessible by Python like i.a. _csv_, _json_ or _parquet_.
On the [Record Evolution Platform](https://www.record-evolution.de/en/home-en/), the library can be used both as a command line tool for interactive usage and as a Python module to integrate the _.raw_ format into any ETL workflow.
## Overview
@@ -28,16 +28,15 @@ _hdf5_.
* [Usage and Examples](#Usage)
* [References](#References)
## Fileformat
## File format
A data file of _IMC Bus Format_ type with extension _.raw_ is a _mixed text/binary
A data file of the _IMC Bus Format_ type with the extension _.raw_ is a _mixed text/binary
file_ featuring a set of markers (keys) that indicate the start of various blocks
of data providing meta information and the actual measurement data. Every single
marker is introduced by character `"|" = 0x 7c` followed by two uppercase letters,
which characterize the type of marker. Each block is further divided into several
of data that provide meta information and the actual measurement data. Every single
marker is introduced by the character `"|" = 0x 7c` followed by two uppercase letters that characterize the type of marker. Each block is further divided into several
parameters separated by commata `"," = 0x 2c` and terminated by a semicolon
`";" = 0x 3b`. For instance, the header - first 600 bytes - of a raw file may
look like this (in UTF-8 encoding)
look like this (in UTF-8 encoding):
```
|CF,2,1,1;|CK,1,3,1,1;
@@ -49,8 +48,8 @@ look like this (in UTF-8 encoding)
|CS,1, 9619, 1,<2C>oD <09>nD6<44>nD)<29>nD<6E>
```
where line breaks where introduced for readability. Most of the markers introduce
blocks of text, while only the last one identified by `|CS` contains binary data.
Line breaks are introduced for readability. Most of the markers introduce
blocks of text, while only the last block identified by `|CS` contains binary data.
The format supports the storage of _multiple data sets (channels)_ in a single
file. The channels may be ordered in _multiplex_ mode (ordering w.r.t. time) or
_block_ mode (ordering w.r.t. to channels).
@@ -61,7 +60,7 @@ letters. There are _two types_ of markers distinguished by the first letter:
1. _critical_ markers: introduced by `|C` featuring uppercase `C`
1. _noncritical_ markers: introduced by `|N` featuring uppercase `N`
The second letter represents further details of the specific key. Note, that
The second letter represents further details of the specific key. Note that
while the _noncritical_ keys are optional, any _.raw_ file _cannot be_ correctly
decoded if any of the _critical_ markers are misinterpreted, invalid or damaged.
The second uppercase letter is followed by the first comma and the _version_
@@ -70,9 +69,9 @@ representation) specifies the length of the entire block, i.e. the number of
bytes between the following comma and the block-terminating semicolon. The further
structure of a block is not defined and may feature different numbers of additional
parameters. The format allows for any number of carriage returns (`CR = 0x0d`)
and line feeds (`LF = 0x 0a`) between keys, i.e. the block terminating semicolon
and line feeds (`LF = 0x 0a`) between keys, i.e. the block-terminating semicolon
and the vertical bar (pipe) of the next key. The following _critical markers_
are defined
are defined:
| marker | description |
@@ -118,8 +117,8 @@ module.
### CLI tool
To build the CLI tool locally use the default target `make` resulting
in the binary `imctermite`. To ensure system-wide availability the installation
To build the CLI tool locally, use the default target `make` resulting
in the binary `imctermite`. To ensure system-wide availability, the installation
of the tool (in the default location `/usr/local/bin`) is done via
```
@@ -130,27 +129,40 @@ which may require root permissions.
### Python
To integrate the library into a customized ETL toolchain several cython targets
are available. For a local build that enables you to run the examples use:
To integrate the library into a customized ETL toolchain, several cython targets
are available. For a local build that enables you to run the examples, use:
```
make cython-build
```
However, in a production environment a proper installation of the module with
However, in a production environment, a proper installation of the module with
`make cython-install` is recommended for system-wide availability of the module.
#### Installation with pip
The package is also available in the [Python Package Index](https://pypi.org)
at [IMCtermite](https://pypi.org/project/IMCtermite/).
To install the latest version simply do
```Shell
python3 -m pip install IMCtermite
```
Note, that _python3_setuptools_ and _gcc version >= 10.2.0_ are required to
successfully install and use it.
## Usage
### CLI
The usage of the `imctermite` binary looks like this
The usage of the `imctermite` binary looks like this:
```
imctermite <raw-file> [options]
```
since you have to provide a single _raw_ files and any option to specify what
You have to provide a single _raw_ file and any option to specify what
to do with the data. All available options can be listed with `imctermite --help`:
```
@@ -159,19 +171,23 @@ Options:
-c, --listchannels list channels
-b, --listblocks list IMC key-blocks
-d, --output output directory to print channels
-s, --delimiter csv delimiter/separator char for output
-h, --help show this help message
-v, --version display version
```
For instance, to show a list of all channels included in `sample-data.raw`, you
do `imctermite sample-data.raw --listchannels`. By default, no output files are
written but only when an existing (!) directory is provided as argument to
the `--output` option.
do `imctermite sample-data.raw --listchannels`. No output files are
written by default. Output files are written only when an existing (!) directory
is provided as argument to the `--output` option. By default, every output file
is written using a `,` delimiter. You may provide any custom separator with the
option `--delimiter`. For example, in order to use `|`, the binary is called with
options `imctermite sample-data.raw -b -c -s '|'`.
### Python
Given the `imctermite` module is available we can import it and declare an instance
of it by passing a _raw_ file to the constructor
Given the `imctermite` module is available, we can import it and declare an instance
of it by passing a _raw_ file to the constructor:
```Python
import imc_termite
@@ -195,8 +211,15 @@ channels = imcraw.get_channels(False)
print(channels)
```
A more complete example including the methods for obtaining the channels including
their data and or directly printing them to files can be found at
[Usage](python/usage.py).
A more complete [example](python/usage.py), including the methods for obtaining the
channels, i.a. their data and/or directly printing them to files, can be found
in the Python folder.
## References
- https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/funktionen/im-und-export/
- https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/
- https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html
- https://github.com/Apollo3zehn/ImcFamosFile
- https://apollo3zehn.github.io/ImcFamosFile/api/ImcFamosFile.FamosFileKeyType.html
- https://pypi.org/help/#apitoken

View File

@@ -14,5 +14,7 @@ cdef extern from "imc_raw.hpp" namespace "imc":
void set_file(string rawfile) except +
# get JSON list of channels
vector[string] get_channels(bool json, bool data) except +
# print all channels
void print_channels(string outputdir) except +
# print single channel/all channels
void print_channel(string channeluuid, string outputdir, char delimiter) except +
void print_channels(string outputdir, char delimiter) except +
void print_table(string outputfile) except +

View File

@@ -3,6 +3,7 @@
from imc_termite cimport imc_termite
import json as jn
import decimal
# import numpy as np
cdef class imctermite:
@@ -21,9 +22,23 @@ cdef class imctermite:
# get JSON list of channels
def get_channels(self, bool data):
chnlst = self.cpp_imc.get_channels(True,data)
chnlstjn = [jn.loads(chn.decode()) for chn in chnlst]
chnlstjn = [jn.loads(chn.decode(errors="ignore")) for chn in chnlst]
return chnlstjn
# print a channels
def print_channels(self, string outputdir):
self.cpp_imc.print_channels(outputdir)
# print single channel/all channels
def print_channel(self, string channeluuid, string outputfile, char delimiter):
self.cpp_imc.print_channel(channeluuid,outputfile,delimiter)
def print_channels(self, string outputdir, char delimiter):
self.cpp_imc.print_channels(outputdir,delimiter)
# print table including channels
def print_table(self, string outputfile):
chnlst = self.cpp_imc.get_channels(True,True)
chnlstjn = [jn.loads(chn.decode(errors="ignore")) for chn in chnlst]
with open(outputfile.decode(),'w') as fout:
for chn in chnlstjn:
fout.write('#' +str(chn['xname']).rjust(19)+str(chn['yname']).rjust(20)+'\n')
fout.write('#'+str(chn['xunit']).rjust(19)+str(chn['yunit']).rjust(20)+'\n')
for n in range(0,len(chn['ydata'])):
fout.write(str(chn['xdata'][n]).rjust(20)+
str(chn['ydata'][n]).rjust(20)+'\n')

View File

@@ -14,11 +14,11 @@ extensions = Extension(
)
setup(
version='0.1',
name="imc_termite",
version='1.2.5',
description='IMCtermite cython extension',
author='Record Evolution GmbH',
author_email='mario.fink@record-evolution.de',
url='https://github.com/RecordEvolution/IMCtermite.git',
name="imc_termite",
ext_modules=cythonize(extensions,force=True)
)

View File

@@ -95,7 +95,7 @@ namespace imc
ss<<"[";
if ( myvec.size() <= limit )
{
for ( dt el: myvec ) ss<<el<<",";
for ( dt el: myvec ) ss<<std::setprecision(10)<<el<<",";
}
else
{
@@ -104,7 +104,7 @@ namespace imc
ss<<"...";
for ( unsigned long int i = myvec.size()-heals; i < myvec.size(); i++ )
{
ss<<myvec[i]<<",";
ss<<std::setprecision(10)<<myvec[i]<<",";
}
}
std::string sumstr = ss.str();
@@ -147,9 +147,11 @@ namespace imc
std::string group_uuid_, group_name_, group_comment_;
// constructor takes channel's block environment
channel(channel_env chnenv, std::map<std::string,imc::block>* blocks,
channel(channel_env &chnenv, std::map<std::string,imc::block>* blocks,
std::vector<unsigned char>* buffer):
chnenv_(chnenv), blocks_(blocks), buffer_(buffer), group_index_(-1)
chnenv_(chnenv), blocks_(blocks), buffer_(buffer),
factor_(1.), offset_(0.),
group_index_(-1)
{
// declare list of block parameters
std::vector<imc::parameter> prms;
@@ -292,6 +294,11 @@ namespace imc
{
imc::convert_data_to_type<imc_double>(CSbuffer,ydata_);
}
// ...
else if ( datatp_ == 11 )
{
imc::convert_data_to_type<imc_digital>(CSbuffer,ydata_);
}
else
{
throw std::runtime_error(std::string("unsupported/unknown datatype") + std::to_string(datatp_));
@@ -374,7 +381,7 @@ namespace imc
}
// print channel
void print(std::string filename, const char sep = ' ', int width = 25)
void print(std::string filename, const char sep = ',', int width = 25)
{
std::ofstream fou(filename);

View File

@@ -17,7 +17,11 @@ namespace imc
// check number of elements of type "datatype" in buffer
if ( subbuffer.size() != channel.size()*sizeof(datatype) )
{
throw std::runtime_error("size mismatch between subbuffer and datatype");
throw std::runtime_error( std::string("size mismatch between subbuffer (")
+ std::to_string(subbuffer.size())
+ std::string(") and datatype (")
+ std::to_string(channel.size()) + std::string("*")
+ std::to_string(sizeof(datatype)) + std::string(")") );
}
// extract every single number of type "datatype" from buffer
@@ -36,6 +40,8 @@ namespace imc
// save number in channel
channel[i] = df;
}
// for ( auto el: channel ) std::cout<<el<<"\n";
}
}

View File

@@ -10,13 +10,27 @@ namespace imc
// map datatype to machine's datatypes
typedef unsigned char imc_Ubyte;
typedef signed char imc_Sbyte;
//
typedef unsigned short imc_Ushort;
typedef signed short imc_Sshort;
//
// e.g. ARM Cortex-A72 armv7l gcc version 10.2.0 (Ubuntu 10.2.0-13ubuntu1)
// #ifdef __arm__
typedef unsigned long int imc_Ulongint;
typedef signed long int imc_Slongint;
// e.g. Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz x86_64 gcc version 10.2.0 (Ubuntu 10.2.0-13ubuntu1)
// #ifdef i386 __i386 __i386__
// typedef unsigned int imc_Ulongint;
// typedef signed int imc_Slongint;
//
typedef float imc_float;
typedef double imc_double;
// TODO remaining types are not yet supported
//
// TODO not all remaining types are supported yet
// typedef <whatever that is ->... > "imc Devices Transitional Recording"
// typedf <sometimestamptype> "Timestamp Ascii"
typedef char16_t imc_digital;
// typedef < > imc_sixbyte "6byte unsigned long"
class datatype
{
@@ -29,12 +43,14 @@ namespace imc
imc_Slongint slint_; // 5
imc_float sfloat_; // 6
imc_double sdouble_; // 7
short int dtidx_; // \in \{0,...,7\}
imc_digital sdigital_; // 10
short int dtidx_; // \in \{0,...,7,10\}
public:
datatype(): ubyte_(0), sbyte_(0),
ushort_(0), sshort_(0),
ulint_(0.0), slint_(0.0),
sfloat_(0.0), sdouble_(0.0),
sdigital_(0),
dtidx_(0) { };
// every supported datatype gets its own constructor
datatype(imc_Ubyte num): ubyte_(num), dtidx_(0) {};
@@ -46,11 +62,29 @@ namespace imc
datatype(imc_float num): sfloat_(num), dtidx_(6) {};
datatype(imc_double num): ubyte_(0), sbyte_(0), ushort_(0), sshort_(0),
ulint_(0.0), slint_(0.0), sfloat_(0.0), sdouble_(num),
dtidx_(7) {};
sdigital_(0), dtidx_(7) {};
datatype(imc_digital num): ubyte_(0), sbyte_(0), ushort_(0), sshort_(0),
ulint_(0.0), slint_(0.0), sfloat_(0.0), sdouble_(num),
sdigital_(num), dtidx_(10) {};
// identify type
short int& dtype() { return dtidx_; }
// copy constructor
datatype(const datatype &num)
{
this->ubyte_ = num.ubyte_;
this->sbyte_ = num.sbyte_;
this->ushort_ = num.ushort_;
this->sshort_ = num.sshort_;
this->ulint_ = num.ulint_;
this->slint_ = num.slint_;
this->sfloat_ = num.sfloat_;
this->sdouble_ = num.sdouble_;
this->sdigital_ = num.sdigital_;
this->dtidx_ = num.dtidx_;
}
// overall assignment operator
datatype& operator=(const datatype &num)
{
@@ -64,6 +98,7 @@ namespace imc
this->slint_ = num.slint_;
this->sfloat_ = num.sfloat_;
this->sdouble_ = num.sdouble_;
this->sdigital_ = num.sdigital_;
this->dtidx_ = num.dtidx_;
}
@@ -119,6 +154,12 @@ namespace imc
this->dtidx_ = 7;
return *this;
}
datatype& operator=(const imc_digital &num)
{
this->sdigital_ = num;
this->dtidx_ = 10;
return *this;
}
// obtain number as double
double as_double()
@@ -132,6 +173,7 @@ namespace imc
else if ( dtidx_ == 5 ) num = (double)slint_;
else if ( dtidx_ == 6 ) num = (double)sfloat_;
else if ( dtidx_ == 7 ) num = (double)sdouble_;
else if ( dtidx_ == 10 ) num = static_cast<double>(sdigital_);
return num;
}
@@ -146,6 +188,7 @@ namespace imc
else if ( num.dtidx_ == 5 ) out<<num.slint_;
else if ( num.dtidx_ == 6 ) out<<num.sfloat_;
else if ( num.dtidx_ == 7 ) out<<num.sdouble_;
else if ( num.dtidx_ == 10 ) out<<static_cast<double>(num.sdigital_);
return out;
}

View File

@@ -501,16 +501,16 @@ namespace imc
second_ = std::stod( get_parameter(buffer,&parameters[7]) );
time_t rawtime;
struct tm* ts;
struct tm ts;
time(&rawtime);
ts = localtime(&rawtime);
ts->tm_mday = day_;
ts->tm_mon = month_-1;
ts->tm_year = year_-1900;
ts->tm_hour = hour_;
ts->tm_min = minute_;
ts->tm_sec = (int)second_;
timestamp_ = asctime(ts);
localtime_r(&rawtime,&ts);
ts.tm_mday = day_;
ts.tm_mon = month_-1;
ts.tm_year = year_-1900;
ts.tm_hour = hour_;
ts.tm_min = minute_;
ts.tm_sec = (int)second_;
asctime_r(&ts,&timestamp_[0]);
// timestamp_ = std::to_string(year_) + std::string("-") + std::to_string(month_)
// + std::string("-") + std::to_string(day_)
// + std::string("T") + std::to_string(hour_)

View File

@@ -7,7 +7,7 @@
#include <filesystem>
#include <iostream>
#include "hexshow.hpp"
// #include "hexshow.hpp"
#include "imc_key.hpp"
#include "imc_block.hpp"
#include "imc_datatype.hpp"
@@ -174,7 +174,7 @@ namespace imc
// check consistency of blocks
void check_consistency()
{
for ( unsigned long int b = 0; b < this->rawblocks_.size()-1; b++ )
for ( unsigned long int b = 0; b < this->rawblocks_.size()-1 && this->rawblocks_.size() > 0; b++ )
{
if ( this->rawblocks_[b].get_end() >= this->rawblocks_[b+1].get_begin() )
{
@@ -300,7 +300,7 @@ namespace imc
}
// list a particular type of block
std::vector<imc::block> list_blocks(imc::key mykey)
std::vector<imc::block> list_blocks(const imc::key &mykey)
{
std::vector<imc::block> myblocks;
for ( imc::block blk: this->rawblocks_ )
@@ -332,14 +332,38 @@ namespace imc
return channels;
}
// print all channels in directory
void print_channels(std::string output)
// print single specific channel
void print_channel(std::string channeluuid, std::string outputfile, const char sep)
{
// check for given parent directory of output file
std::filesystem::path pdf = outputfile;
if ( !std::filesystem::is_directory(pdf.parent_path()) )
{
throw std::runtime_error(std::string("required directory does not exist: ")
+ pdf.parent_path().u8string() );
}
// find channel with given name
if ( channels_.count(channeluuid) == 1 )
{
channels_.at(channeluuid).print(outputfile,sep);
}
else
{
throw std::runtime_error(std::string("channel does not exist:")
+ channeluuid);
}
}
// print all channels into given directory
void print_channels(std::string output, const char sep)
{
// check for given directory
std::filesystem::path pd = output;
if ( !std::filesystem::is_directory(pd) )
{
throw std::runtime_error("given directory does not exist");
throw std::runtime_error(std::string("given directory does not exist: ")
+ output);
}
for ( std::map<std::string,imc::channel>::iterator it = channels_.begin();
@@ -352,7 +376,7 @@ namespace imc
std::filesystem::path pf = pd / filenam;
// and print the channel
it->second.print(pf);
it->second.print(pf,sep);
}
}

View File

@@ -1,5 +1,7 @@
#-----------------------------------------------------------------------------#
SHELL := /bin/bash
# name of executable and CLI tool
EXE = imctermite
@@ -17,9 +19,11 @@ CC = g++ -std=c++17
#OPT = -O3 -Wall -mavx -mno-tbm -mf16c -mno-f16c
OPT = -O3 -Wall -Werror -Wunused-variable -Wsign-compare
# determine git version/commit tag
# determine git version/commit and release tag
GTAG := $(shell git tag | tail -n1)
GHSH := $(shell git rev-parse HEAD | head -c8)
RTAG := v$(shell cat pip/setup.py | grep version | grep -oP "([0-9]\.){2}[0-9]")
CTAG := v$(shell cat cython/setup.py | grep version | grep -oP "([0-9]\.){2}[0-9]")
# define install location
INST := /usr/local/bin
@@ -28,8 +32,8 @@ INST := /usr/local/bin
# C++ and CLI tool
# build exectuable
$(EXE) : main.o
$(CC) $(OPT) $^ -o $@
$(EXE) : check-vtag $(RTAG) main.o
$(CC) $(OPT) main.o -o $@
# build main.cpp and include git version/commit tag
main.o : src/main.cpp $(HPP)
@@ -45,10 +49,28 @@ install : $(EXE)
uninstall : $(INST)/$(EXE)
rm $<
clean :
cpp-clean :
rm -vf $(EXE)
rm -vf *.o
#-----------------------------------------------------------------------------#
# linter and code check
check-code:
cppcheck --enable=all -I lib/ src/main.cpp
#-----------------------------------------------------------------------------#
# check version consistency of git tags and version string in package.json
$(GTAG) :
@echo "consistent versions check successful: building $(GTAG)"
check-vtag:
@echo "git tag version: "$(GTAG)
@echo "git commit hash: "$(GHSH)
@echo "release version: "$(RTAG)
@echo "module version: "$(CTAG)
#-----------------------------------------------------------------------------#
# Docker
@@ -61,48 +83,28 @@ docker-run:
#-----------------------------------------------------------------------------#
# python
cython-build : $(CYT)setup.py $(CYT)imc_termite.pxd $(CYT)py_imc_termite.pyx $(HPP)
python3 $< build_ext --inplace
cython-build : check-vtag $(CTAG) $(CYT)setup.py $(CYT)imc_termite.pxd $(CYT)py_imc_termite.pyx $(HPP)
python3 $(CYT)setup.py build_ext --inplace
cp -v imc_termite.cpython-*.so $(PYT)
cython-install : $(CYT)setup.py $(CYT)imc_termite.pxd $(CYT)py_imc_termite.pyx $(HPP)
python3 $< install --record files_imctermite.txt
cython-install : check-vtag $(CTAG) $(CYT)setup.py $(CYT)imc_termite.pxd $(CYT)py_imc_termite.pyx $(HPP)
python3 $(CYT)setup.py install --record files_imctermite.txt
cython-clean :
rm -vf imc_termite.cpython-*.so
rm -vf $(PYT)imc_termite.cpython-*.so
rm -rvf build/
#-----------------------------------------------------------------------------#
# Python (to be removed)
# pip
# build python module
py : $(CYT)setup_raw_eater.py $(CYT)raw_eater.pyx $(CYT)raw_eater.pxd $(LIB)raweat.hpp \
$(CYT)setup_raw_meat.py $(CYT)raw_meat.pyx $(CYT)raw_meat.pxd $(LIB)rawmerge.hpp \
output
python3 $(CYT)setup_raw_eater.py build_ext --inplace
python3 $(CYT)setup_raw_meat.py build_ext --inplace
cp raw_eater.cpython-*.so $(PYT)
cp raw_meat.cpython-*.so $(PYT)
rm *.so
py-install: $(CYT)setup_raw_eater.py $(CYT)raw_eater.pyx $(CYT)raw_eater.pxd $(LIB)raweat.hpp \
$(CYT)setup_raw_meat.py $(CYT)raw_meat.pyx $(CYT)raw_meat.pxd $(LIB)rawmerge.hpp
python3 $(CYT)setup_raw_eater.py install --record files_raw_eater.txt
python3 $(CYT)setup_raw_meat.py install --record files_raw_meat.txt
py-clean :
rm -f raw_eater.cpython-*.so
rm -f $(PYT)raw_eater.cpython-*.so
rm -f $(CYT)raw_eater.cpp
rm -f raw_meat.cpython-*.so
rm -f $(PYT)raw_meat.cpython-*.so
rm -f $(CYT)raw_meat.cpp
rm -rf build/
rm -f *.txt
rm -rf output/
# prepare directory for test output
output :
mkdir -pv output/
pip-release: check-vtag $(RTAG) cython-build
cd ./pip/ && make publish
#-----------------------------------------------------------------------------#
# clean
clean: cpp-clean cython-clean
cd ./pip/ && make clean
#-----------------------------------------------------------------------------#

View File

@@ -1,6 +1,5 @@
#-----------------------------------------------------------------------------#
import glob
from pathlib import Path
# find source files

4
pip/MANIFEST.in Normal file
View File

@@ -0,0 +1,4 @@
include *.hpp
include *.cpp
include *.pyx
include *.pxd

4
pip/bkup.pyproject.toml Normal file
View File

@@ -0,0 +1,4 @@
[build-system]
requires = [
"setuptools"
]

29
pip/makefile Normal file
View File

@@ -0,0 +1,29 @@
# --------------------------------------------------------------------------- #
SHELL := /bin/bash
publish: sdist upload
sdist: ../cython/py_imc_termite.pyx ../cython/imc_termite.pxd ../cython/py_imc_termite.cpp
cp -v $? ./
cp -v $(shell ls ../lib/imc_*.hpp) ./
tail -n 212 ../README.md > ./README.md
cp -v ../LICENSE ./
python3 setup.py sdist
# authentication:
# - username: __token__
# - password: <token value including pypi-prefix>
upload:
python3 -m twine upload dist/$(shell ls -t dist/ | head -n1)
clean:
rm -rvf dist/
rm -rvf *.egg-info
rm -rvf build/
rm -rvf cython/
rm -vf *.pyx *.pxd
rm -vf *.cpp *.c *.hpp
rm -vf README.md LICENSE
# --------------------------------------------------------------------------- #

36
pip/setup.py Normal file
View File

@@ -0,0 +1,36 @@
from setuptools import setup, Extension
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
setup(
name="IMCtermite",
version="1.2.5",
author="Record Evolution GmbH",
author_email="mario.fink@record-evolution.de",
maintainer="Record Evolution GmbH",
license="MIT",
description="Enables extraction of measurement data from binary files with extension 'raw' used by proprietary software imcFAMOS/imcSTUDIO and facilitates its storage in open source file formats",
keywords="IMC raw imcFAMOS imcSTUDIO imcCRONOS",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/RecordEvolution/IMCtermite.git",
project_urls={
"Bug Tracker": "https://github.com/RecordEvolution/IMCtermite/issues",
},
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
ext_modules=[Extension("imc_termite",
["py_imc_termite.cpp"],
# libraries_dirs=["cython/"],
# include_dirs=["3rdparty/pugixml/","lib/"],
# depends=["../lib/tdm_termite.hpp"]
language='c++',
extra_compile_args=['-std=c++17','-Wno-unused-variable'],
extra_link_args=['-std=c++17'],
)],
)

View File

@@ -5,19 +5,18 @@ import raw_eater
import raw_meat
import pyarrow as pa
import pyarrow.parquet as pq
import glob
from pathlib import Path
fileobj1 = Path("smp/Rangerover_Evoque_F-RR534_2019-05-07/").rglob("*.raw")
fileobj1 = Path("samples/datasetA/").rglob("*.raw")
rawlist1 = [str(fl) for fl in fileobj1]
fileobj2 = Path("smp/Mercedes_E-Klasse-2019-08-08/").rglob("*.raw")
fileobj2 = Path("samples/datasetB/").rglob("*.raw")
rawlist2 = [str(fl) for fl in fileobj2]
rawlist = rawlist1 #[rawlist1[0],rawlist1[4],rawlist2[0],rawlist2[6]]
for fil in rawlist2 :
rawlist.append(fil)
rawlist.append("smp/pressure_Vacuum.asc")
rawlist.append("./README.md")
print("")
print(rawlist)

View File

@@ -1,12 +1,13 @@
import imc_termite
import json
import os
# declare and initialize instance of "imctermite" by passing a raw-file
try :
imcraw = imc_termite.imctermite(b"samples/sampleA.raw")
imcraw = imc_termite.imctermite(b"samples/exampleB.raw")
except RuntimeError as e :
print("failed to load/parse raw-file: " + str(e))
raise Exception("failed to load/parse raw-file: " + str(e))
# obtain list of channels as list of dictionaries (without data)
channels = imcraw.get_channels(False)
@@ -22,4 +23,16 @@ print(len(chnydata))
print(len(chnxdata))
# print the channels into a specific directory
imcraw.print_channels(b"./")
imcraw.print_channels(b"./data",ord(','))
# print all channels separately
idx = 0
for chn in channels :
print(str(idx)+" : "+chn['name']+" : "+chn['uuid'])
filname = os.path.join("./data",str(idx) + "_" + chn['name']+".csv")
print(filname)
imcraw.print_channel(chn['uuid'].encode(),filname.encode(),ord(','))
idx = idx + 1
# print all channels in single file
# imcraw.print_table(b"./data/allchannels.csv")

30
python/usage_adv.py Normal file
View File

@@ -0,0 +1,30 @@
import imc_termite
import json
import os
# list files in sample directory
# fileobj1 = Path("samples/").rglob("*.raw")
# rawlist1 = [str(fl) for fl in fileobj1]
rawlist1 = ["samples/datasetB/datasetB_29.raw"]
print(rawlist1)
for fl in rawlist1:
print("converting " + str(fl) + " : " + str(os.path.basename(fl)) )
# declare and initialize instance of "imctermite" by passing a raw-file
try :
imcraw = imc_termite.imctermite(fl.encode())
except RuntimeError as e :
raise Exception("failed to load/parse raw-file: " + str(e))
# obtain list of channels as list of dictionaries (without data)
channels = imcraw.get_channels(False)
print(json.dumps(channels,indent=4, sort_keys=False))
# print the channels into a specific directory
imcraw.print_channels(b"./")
# print all channels in single file
imcraw.print_table(("./"+str(os.path.basename(fl).split('.')[0])+"_allchannels.csv").encode())

View File

@@ -32,25 +32,37 @@ optkeys parse_args(int argc, char* argv[], bool list_args = false)
// parse all CLI arguments
for ( int i = 1; i < argc; i++ )
{
if ( std::string(argv[i]) == std::string("--showmeta")
|| std::string(argv[i]) == std::string("-m") )
{
prsdkeys.insert(std::pair<std::string,std::string>("showmeta",argv[i]));
}
// if ( std::string(argv[i]) == std::string("--showmeta")
// || std::string(argv[i]) == std::string("-m") )
// {
// prsdkeys.insert(std::pair<std::string,std::string>("showmeta",argv[i]));
// }
// else if ( std::string(argv[i]) == std::string("--listgroups")
// || std::string(argv[i]) == std::string("-g") )
// {
// prsdkeys.insert(std::pair<std::string,std::string>("listgroups",argv[i]));
// }
else if ( std::string(argv[i]) == std::string("--listchannels")
if ( std::string(argv[i]) == std::string("--listchannels")
|| std::string(argv[i]) == std::string("-c") )
{
prsdkeys.insert(std::pair<std::string,std::string>("listchannels",argv[i]));
if ( i+1 < argc ) {
if ( argv[i+1][0] != '-' ) {
std::cerr<<"option --list-channels does not take any argument\n";
prsdkeys.insert(std::pair<std::string,std::string>("invalid","channels"));
}
}
}
else if ( std::string(argv[i]) == std::string("--listblocks")
|| std::string(argv[i]) == std::string("-b") )
{
prsdkeys.insert(std::pair<std::string,std::string>("listblocks",argv[i]));
if ( i+1 < argc ) {
if ( argv[i+1][0] != '-' ) {
std::cerr<<"option --list-blocks does not take any argument\n";
prsdkeys.insert(std::pair<std::string,std::string>("invalid","listblocks"));
}
}
}
else if ( std::string(argv[i]) == std::string("--output")
|| std::string(argv[i]) == std::string("-d") )
@@ -66,6 +78,20 @@ optkeys parse_args(int argc, char* argv[], bool list_args = false)
i = i + 1;
}
}
else if ( std::string(argv[i]) == std::string("--delimiter")
|| std::string(argv[i]) == std::string("-s") )
{
if ( i+1 == argc || argv[i+1][0] == '-' )
{
std::cerr<<"invalid or missing --delimiter argument\n";
prsdkeys.insert(std::pair<std::string,std::string>("invalid","delimiter"));
}
else
{
prsdkeys.insert(std::pair<std::string,std::string>("delimiter",argv[i+1]));
i = i + 1;
}
}
else if ( std::string(argv[i]) == std::string("--help")
|| std::string(argv[i]) == std::string("-h") )
{
@@ -84,13 +110,13 @@ optkeys parse_args(int argc, char* argv[], bool list_args = false)
std::cerr<<"invalid or unkown argument: "<<argv[i]<<"\n";
prsdkeys.insert(std::pair<std::string,std::string>("invalid",argv[i]));
}
// or missing filenames
else if ( std::string(argv[i]).find(".raw") == std::string::npos )
{
std::cerr<<"doesn't look like a .raw file (make sure to include extension in filename!): "
<<argv[i]<<"\n";
prsdkeys.insert(std::pair<std::string,std::string>("invalid",argv[i]));
}
// // or missing filenames
// else if ( std::string(argv[i]).find(".raw") == std::string::npos )
// {
// std::cerr<<"doesn't look like a .raw file (make sure to include extension in filename!): "
// <<argv[i]<<"\n";
// prsdkeys.insert(std::pair<std::string,std::string>("invalid",argv[i]));
// }
}
}
@@ -121,9 +147,13 @@ void show_usage()
<<" -c, --listchannels list channels\n"
<<" -b, --listblocks list IMC key-blocks\n"
<<" -d, --output output directory to print channels\n"
<<" -s, --delimiter csv delimiter/separator char for output\n"
<<" -h, --help show this help message \n"
<<" -v, --version display version\n"
<<"\n";
<<"\n"
<<"Example:"
<<" $ ./imctermite sample/data_A.raw -c -b -d ./data -s ','"
<<"\n\n";
}
//---------------------------------------------------------------------------//
@@ -178,6 +208,13 @@ int main(int argc, char* argv[])
return 1;
}
// catch invalid or empty ".raw" file
if ( imcraw.blocks().size() == 0 )
{
std::cerr<<"this appears to be an empty/invalid '.raw' file since no blocks were found"<<"\n";
return 1;
}
// list blocks
if ( cfgopts.count("listblocks") == 1 )
{
@@ -202,7 +239,22 @@ int main(int argc, char* argv[])
if ( cfgopts.count("output") == 1 )
{
try {
imcraw.print_channels(cfgopts.at("output"));
// check and use desired delimiter
char delim;
if ( cfgopts.count("delimiter") == 1 )
{
if ( cfgopts.at("delimiter").size() > 1 )
{
throw std::runtime_error("invalid delimiter comprised of more than a single char");
}
delim = cfgopts.at("delimiter")[0];
}
// use comma by default
else
{
delim = ',';
}
imcraw.print_channels(cfgopts.at("output"),delim);
} catch (const std::exception& e) {
std::cerr<<"failed to print channels for "<<rawfile<<": "<<e.what()<<"\n";
}