Filtering Metrics With the observIQ Distro for OpenTelemetry Collector

On this tutorial, we are going to handle the widespread monitoring use case of filtering metrics inside observIQ’s distribution of the OpenTelemetry (OTEL) collector. Whether or not the metrics are deemed pointless, or they’re filtered for safety considerations, the method is pretty simple.

For our pattern setting, we are going to use MySQL on Pink Hat Enterprise Linux 8. The vacation spot exporter shall be to Google Cloud Operations, however the course of is exporter agnostic. We’re utilizing this exporter to supply the visible charts displaying the metric earlier than and after filtering.

Atmosphere Conditions

• Appropriate working system
• observIQ Distro for OTEL Collector put in
• MySQL put in
• MySQL Least Privilege Person (LPU) setup
• OTEL configured to gather metrics from MySQL

Sources

Preliminary Metrics

As soon as configured utilizing the LPU I created, MySQL metrics ought to be flowing. For our functions, we are going to concentrate on the precise metric mysql.buffer_pool.restrict. Presently, our config.yaml MySQL part appears to be like like this:

mysql:
    endpoint: localhost:3306
    username: otel
    password: otelPassword
    collection_interval: 60s

After ready for a minimum of 5 minutes to get a superb quantity of information, metrics will look one thing like this in Google’s Metrics Explorer:

Filtering

Now that metrics are flowing, we will filter them. First, allow us to focus on the explanations to filter this particular metric. The reply is straightforward: It isn’t actually all that helpful or vital. It’s going to, barring a configuration change by the DBA, be a flat line. Even after a configuration change, it will merely step that flat line up or down.

To do the filtering, we first want to have a look at the metadata file for the MySQL receiver. On this file, we discover a itemizing of the attributes and metrics related to this receiver. If we go to the metrics part of the file, and discover our pool restrict metric, we be taught it appears to be like like this:

mysql.buffer_pool.restrict:
	enabled: true
	description: The configured dimension of the InnoDB buffer pool.
	unit: By
	sum:
  	value_type: int
  	input_type: string
  	monotonic: false
  	aggregation: cumulative

This lets us know that it’s enabled by default, provides an outline, and another vital information in regards to the metric. As these are the defaults, we will interpret from it that if we set the enabled parameter to false, then it ought to disable — aka filter — this metric. It is not going to be collected, and because it isn’t collected, it additionally is not going to be despatched to the exporter.

To realize this in our configuration file, we make the next adjustments:

 mysql:
    endpoint: localhost:3306
    username: otel
    password: otelPassword
    collection_interval: 60s
    metrics:
      mysql.buffer_pool.restrict:
        enabled: false

This replicates the construction from the metadata file, however with the whole lot else trimmed aside from the naked minimal variety of traces wanted to attain our purpose.

As soon as this has been modified, and the collector restarted, I once more wait a minimum of 5 minutes and verify Google’s Metrics Explorer to see what has modified:

As proven within the screenshot, information was final despatched to Google at 10:48, and it’s now 11:13.

Conclusion

Whereas the knowledge wanted is positioned in a couple of completely different locations, filtering may be very straightforward to do. Don’t neglect that the metadata we checked out additionally gives different data that may be helpful in understanding your information.