As System Insight gathers a large amount of data per second in the form of metrics, using InfluxDB to handle this data, the growing amount of storage required for this data is addressed by InfluxDB. The InfluxDB solution downsamples the data so that high precision raw data is kept for only a limited period of time, and lower precision data is kept for a longer period of time. There are two features which automate the process of downsampling data and expiring old data: Continuous Queries (CQ) and Retention Policies (RP).

For details on this solution, see https://docs.influxdata.com/influxdb/v1.2/query_language/continuous_queries/ and https://docs.influxdata.com/influxdb/v1.2/guides/downsampling_and_retention/.

If you have installed InfluxDB using the script provided, the default setup includes three predefined retention policies and one predefined continuous query.

The predefined retention policies are:

• one_week - this is set as the default retention policy for the database
• six_months
• one_year

The predefined continuous query is named cq_six_months. This is a generic continuous query, downsampling all metrics as mean values over a period of 10 minutes from the default retention policy of one week into the retention policy of six months.

You must implement the data compaction solution provided by InfluxDB to store the data produced using System Insight. See the examples below.

Metrics Compaction Examples

Implementing Default Metrics Compaction

In this scenario, you have installed System Insight using the scripts provided so that you have the default setup of InfluxDB. This means you have a database named "mz" and a default retention policy named "one_week", and InfluxDB is up and running.

The steps below are required to create a retention policy that stores data for one month and a continuous query that runs every five minutes and calculates the mean of idle_cpu of the measurements during that time, and to store the new measurement with the name host.compute in the retention policy created. 

1. Create a retention policy for one month. Measurements with this retention policy are stored for a month:
   

    CREATE RETENTION POLICY "one_month" ON "mz" DURATION 30d REPLICATION 1

2. In this instance, you are sampling a metric named host.compute with  tags={time,host_name} and the values {idle_cpu , user_time_cpu , idle_proc , sleep_proc , sys_cpu , total_proc , up_time, user_cpu , wait_cpu} to InfluxDB with the default retention policy ( one_week ):
   
   

   CREATE CONTINUOUS QUERY "5min_cq" ON "mz" BEGIN SELECT mean("idle_cpu") as "idle_cpu" INTO "mz"."one_month"."host.compute" FROM "host.compute" GROUP BY time(5m) END

   This new measurement is stored in the database for a month, after that they are removed.

Removing Default Metrics Compaction

Depending on how you want to use System Insight, the predefined continuous query cq_six_months might store too much data.

To disable the query, remove the predefined continuous query as follows:

 DROP CONTINUOUS QUERY "cq_six_months" ON "mz"