TsFile: A Standard Format for IoT Time Series Data

The TsFile project has reached 1.0 as committers work toward making it an independent project within the Apache Software Foundation.

TsFile is a columnar storage file format designed for time series data, featuring advanced compression to minimize storage, high throughput of read and write, and deep integration with processing and analysis tools such as Apache projects Spark and Flink.

With the industrial Internet of Things, equipment such as a single wind turbine, for example, produces an incredible amount of data.

“Especially when IoT dives into industrial internet, intelligent equipment produces one to two orders of magnitudes of data more than consumer-oriented IoT,” and it becomes much more complicated to get actionable insights, according to the project’s GitHub page.

It says TsFile is designed to support a “high ingestion rate up to tens of million data points per second and rare updates only for the correction of low-quality data; compact data packaging and deep compression for long-live historical data; traditional sequential and conditional query, complex exploratory query, signal processing, data mining and machine learning.”

Underlying Format in IoTDB

TsFile is the underlying storage file format for the Apache IoTDB time-series database. IoTDB represents more than a decade of work at China’s Tsinghua University School of Software. It became a top-level project with the Apache Software Foundation in 2020.

“Before TsFile, there was a lack of a standard file format for time series data, leading to complications in data collection and processing. TsFile aims to simplify this by providing a unified format …,” Pengcheng Zheng, a spokesman for the project committee, said in an email.

“With TsFile, users can perform portable unloading and loading of data in IoTDB, making the management and migration of underlying data more flexible. Even without a database, users can directly read data from a TsFile using the SDK, making some lightweight data read/write scenarios possible.”

Users can write data into a TsFile inside end devices or gateway, then send it to the cloud to IoTDB or other unified management systems. It’s not a database itself, but a format that, through compression and efficient storage, reduces network transmission and computing resource consumption in the cloud.

TsFile can store time series from a single or from multiple devices. Though data from multiple devices is stored together in TsFiles, each has an independent storage engine, so is physically isolated as in a traditional database. The data is indexed with time dimensions to accelerate query performance, enabling fast filtering and retrieval of time series data.

In IoTDB, it supports both online transaction processing (OLTP) and online analytical processing (OLAP) without reloading data to different stores.

Using Fewer Cloud Resources

An IoT native data model organizes time series from devices and sensors in an adapted log-structured merge tree for delayed data arrivals in write-intensive workloads. For short delays, the data are first cached in MemTables and then flushed to TsFiles.

TsFile allows users to directly write data with or without pre-defining schema, with or without filters and the new release adds support for more data types and algorithms.

Though originally written in Java, demand for is growing for TsFile implementation in multiple languages, such as C++, Go and Rust, Zheng said. Its users generally work in scenarios where efficient data storage, fast access, and analysis are critical, such as IoT, smart control systems, financial analytics and log analysis.

He said TsFile distinguishes itself with its focus on time series data’s unique requirements.

“Companies used to write time series data in various user-defined file formats without unification, or use general columnar file format such as [Apache projects] Parquet and ORC, which makes data collection and processing complicated without a standard,” he said.

“TsFile offers advantages like deep compression for long-lived historical data, high ingestion rates and the ability to handle rare updates. Its integration capabilities with IoTDB and other systems for unified data management further set it apart. Users could write data in TsFile on embedded devices or gateways, then directly transfer TsFile to the cloud without any traditional ETL [extract, transform, load] processes. In this way, the requirements of network transmission and computing resources in the cloud are decreased.”

Going forward the committee wants to make TsFile an independent project that has its own SDK and documentation that is easier to use, add support for more languages, integrate more encoding and compression methods in TsFile and provide more tools, such as visualization, parsing and repair tools.

“However, those plans are not irrevocable, since we are collaborating in the Apache way and every discussion with new insights could contribute to modifications and optimizations,” Zheng said.

Susan Hall is the Sponsor Editor for The New Stack. Her job is to help sponsors attain the widest readership possible for their contributed content. She has written for The New Stack since its early days, as well as sites…

Read more from Susan Hall