Prometheus data in ElasticSearch via Metricbeats

[Jalpesh1 (Jalpesh Shelar)]

Hi Team,

We are scaling Prometheus data in Elastic via Metricbeats .
But there are lot of metrics exposed in Prometheus .
We want only few metrics in Elastic , is there any filtering possible to restrict metrics?

Currently we are getting around 1,50,000 metrics per minute , we only want 100-200 metrics in Elastic for long term storage .

Please suggest.

Thanks & Regards,
Jalpesh

[Jalpesh1 (Jalpesh Shelar)]

Hi Team,

We have below Sample doc having multiple prometheus metrics and we want only 3 metrics from this doc

        "elasticsearch_thread_pool_threads_count": 1,
        "elasticsearch_thread_pool_largest_count": 2,
        "elasticsearch_thread_pool_completed_count": 43,

Sample doc :

{
  "_index": "metricbeat-7.6.0",
  "_type": "_doc",
  "_id": "-WcQP1l",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-03-04T10:04:46.588Z",
    "service": {
      "type": "prometheus",
      "address": "localhost:9114"
    },
    "event": {
      "duration": 45004100,
      "dataset": "prometheus.collector",
      "module": "prometheus"
    },
    "prometheus": {
      "metrics": {
        "elasticsearch_thread_pool_threads_count": 1,
        "elasticsearch_thread_pool_queue_count": 0,
        "elasticsearch_thread_pool_largest_count": 2,
        "elasticsearch_thread_pool_completed_count": 43,
        "elasticsearch_thread_pool_active_count": 0,
        "elasticsearch_thread_pool_rejected_count": 0
      },
      "labels": {
        "job": "prometheus",
        "cluster": "elasticsearch",
        "es_client_node": "true",
        "name": "vsvsdvdf",
        "es_master_node": "true",
        "type": "ml_utility",
        "instance": "localhost:9114",
        "host": "xyz",
        "es_data_node": "true",
        "es_ingest_node": "true"
      }
    },
    "ecs": {
      "version": "1.4.0"
    },
    "host": {
      "architecture": "x64",
      "name": "ABC",
      "os": {
        "kernel": "107)",
        "build": "1443",
        "platform": "win",
        "version": "10.0",
        "family": "windows",
        "name": "Win"
      },
      "id": "50cf7",
      "hostname": "D1"
    },
    "agent": {
      "hostname": "D1",
      "id": "9a",
      "version": "7.6.0",
      "type": "metricbeat",
      "ephemeral_id": "c23"
    },
    "metricset": {
      "name": "collector",
      "period": 10000
    }
  },
  "fields": {
    "@timestamp": [
      "2020-03-04T10:04:46.588Z"
    ]
  },
  "sort": [
    1583316286588
  ]
}

If there is NO metric available out of that 3 metrics then this doc should drop.

Regards,
Jalpesh

[Jalpesh1 (Jalpesh Shelar)]

Anyone ?