Datareduction/Downsampling in InfluxDB

Description

InfluxDB [0] is used extensively for monitoring of metric of several infrastructures operated by SCC like the GridKa Tier-1 Center and the Large Scale Data Facility (LSDF). Grafana is used to display the metrics from InfluxDB [2]. In contrast to other time series DBs, InfluxDB does not automatically downsample data to a coarse time resolution after a while. Since storage space is limited, data has to be downsampled or removed from the database eventually. The goal of the project is to prepare an overview of the possibilities for data reduction/downsampling in InfluxDB, test the downsampling, and convert existing Grafana dashboards to use the downsampled metrics.

Tasks

• familiarization with InfluxDB, Kapacitor [3] and Grafana
• investigation of possibilities to develop helper tools to facilitate downsampling of generic metrics
• implementation of data reduction strategies in a test environment
• implementation of conversion scripts for existing Grafana dashboards

Requirements

• basic understanding of time series databases
• good knowledge of Python

Supervisors

Andreas Petzold petzold@kit.edu

References

[0] InfluxDB

[1] Downsampling

[2] Grafana

[3] Kapacitor