One of the early decisions I have to make when working on a new project is how to store any data that we capture. Often this is National Instrument's TDMS format. This article will explain why I like to use it and how to access your data in a TDMS file.
What is TDMS?
TDMS is a structured, binary file format developed by National Instruments.
Structured means that it defines measurements in channels. Each channel belongs to a channel group and a file contains multiple channel groups.
At each of these levels (file, channel group and channel) I can assign properties which we use to store meta-data about the measurements taken. For example, the file level may contain a property about what version of the software captured the data. The channel level may contain the scale factors and the serial number of sensors used.
Example TDMS Structure
These structural elements provide traceability to your measurements to ensure that you can search for relevant data and we can track any problems discovered in the system. So if we discover that a particular sensor developed a fault, we can find all channels that were captured with that sensor.
The actual measurement data is stored in a binary format. What this means is that it tends to be very fast to read and write with no loss of precision in the data. This is great to performance but means you need special applications to read them.
How to Read a TDMS File
Excel Plugin
National Instruments provide a plugin so that TDMS files can be pulled into Excel. This creates a front page with all the structural information and the properties for each channel etc. And then creates a new sheet for each channel group with the channel measurements.
This is very popular as everyone has Excel and the plugin is free. You do still hit the limit of rows in Excel though so for large files the plugin will ask you what row to start importing to and it will cut off the rest of the file if it is too large.
First Sheet After Excel Import - This shows the structure of the file. The other sheets contain the data.
DIAdem
DIAdem is a data management and inspection tool from National Instruments. It is designed around the structure of TDMS files which make it a natural partner. The big advantages it has over Excel are:
- Ability to load files of all sizes.
- Much easier to navigate and inspect data than Excel (this is primarily what we use it for at Wiresmith Technology).
- You are able to search and load multiple files for easier data searching and side-by-side comparison.
- Built-in engineering analysis tools.
DIAdem is our go-to tool for TDMS files at Wiresmith Technology but it does require the purchase of a license from NI.
DIAdem Interactive Data View
MATLAB/Python/Other Environments
Sometimes you will have users that want to analyse the data in other data-driven environments.
Obviously, LabVIEW has excellent support for TDMS but what about other common environments?
For MATLAB, National Instruments provide a wrapper around their C DLL although I've heard that some community libraries may function better. If you search the MATLAB file exchange there are a couple of projects such as https://www.mathworks.com/matlabcentral/fileexchange/44206-converttdms-v10 and https://www.mathworks.com/matlabcentral/fileexchange/30023-tdms-reader
Similarly for Python there are a couple of community projects such as pyTDMS and npTDMS for loading data into Python.
Please note I have not tested these libraries ourselves and you should contact the maintainers for support with them.
Case Study
In my project working with Lontra, I had to make this decision. Excel was determined to be the easiest interface to the data and we had to capture high-speed and low-speed data.
In this case we kept the low speed as simple CSV data since the logs were short and slow.
However, once we needed to add high-speed logging we moved to TDMS and the Excel plugin so we could get the write performance coupled with the ease of access for their engineers.
You can read more in Advancing the design of the Blade Compressor®