Protecting Against Textfile Collector Script Failure

Monitoring bespoke metrics with Prometheus textfile collector scripts can fail in such a way that all your monitors look fine but actually there is a problem. Here's how to prevent that.

When you want to monitor a metric with the Prometheus monitoring system one approach is to use the node exporter with the flag, this allows us to add additional custom metrics easily and quickly.

Textfile Collector Cron Jobs

The recommeneded approach is to use a cronjob to generate these text files, here's an example based on the node exporter README:

scriptToOutputMetrics > metrics.prom.$$
mv metrics.prom.$$ metrics.prom

There is an alternative approach recommended in the community node exporter textfile collector scripts repo:

scriptToOutputMetrics | sponge metrics.prom

Both of these approaches solve the problem of the metrics.prom file from being read while it's being written to (so-called "atomic" changes), but what happens if scriptsToOutputMetrics fails to run?

The Problem

In both the scripts above if scriptToOutputMetrics exits without outputting to stdout then the previous copy of metrics.prom isn't overwritten! What this means is that when the node exporter is run and reads the file, as far as it's concerned the values in metrics.prom have just remained the same!

The Solution

All that needs to change to mitigate this problem is to delete the metrics.prom file if the metrics.prom.$$ doesn't exist:

scriptToOutputMetrics > metrics.prom.$$
mv metrics.prom.$$ metrics.prom || rm metrics.prom

 This will cause those metrics to disappear which can then be detected and cause an alert.

Popular Reads


Keep up to date

Please provide your email address
Please provide your name
Please provide your name
No thanks