Reading JMX Metrics, MIMs and Prometheus Agent Metrics from Execution Context Endpoint(3.0)
Each Execution Context exposes an endpoint that can be scraped by Prometheus at:
http(s)://<ec_host>:<ec_webport>/metrics
This endpoint publishes both the JMX exporter metrics and the cached metrics from the Prometheus agents (see Prometheus Agent). The MBeans are the same as JMX exporter publishes, see Setting up Prometheus(3.0).
This means that Prometheus can find the same metrics published by JMX exporter in this endpoint. So JMX exporter is not needed for the Execution Context and the Prometheus Autodiscovery functionality can be used to point to the endpoint.
Properties
It is possible to disable the publishing of metrics in the Execution Context Properties:
Property | Description |
|---|---|
mz.metrics.export.jmx.disable | Set this parameter to true to disable JMX metrics from being exported and published. Default is false. |
It is also possible to configure how many metrics to store and for how long a cache can hold the metrics. This is configured via System Properties:
Property | Description |
|---|---|
mz.metrics.cache.size.max | Maximum number of records in the metrics cache. This property is ignored if set to less than or equal to zero, meaning that there will be no limit in cache size. The default value is 10000. |
mz.metrics.cache.expire | Maximum time in seconds before a metric is removed from the cache. This property is ignored if set to less than zero, meaning that there will be no expiration time. The default value is 300. |
The cache is shared by all workflows running in the Execution Context so its size has to be set accordingly. Each Prometheus scrape empties the cache so the cache size should be set to minimum
<number_of_metrics_expected_per_second> * <prometheus_scrape_interval>.
Note
Cache Overflow
If the metrics cache overflows the oldest metrics will be dropped in favor for the most recent. In order to make sure this doesn't happen, set the cache size to an appropriate size.