Is it possible to get only the types of fields I want from metricbeat?

Elastic Stack Beats
,
1

My metricbeat default field brings in too much unnecessary data. (For example, kubernetes. Kubernetes-related data such as pod.name, uid, namespace.., etc. There are only a few data I need. Can I specify and import this? Rather than importing it and processing it there. I keep getting an error because too many Docs count and too much storage disk has been used.

{
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": "10000"
        }
      },
      "refresh_interval": "30s",
      "blocks": {
        "read_only_allow_delete": "true"
      },
      "provided_name": "<pororometrics-{now/d}-000001>",
      "query": {
        "default_field": [
          "message",
          "tags",
          "agent.ephemeral_id",
          "agent.id",
          "agent.name",
          "agent.type",
          "agent.version",
          "as.organization.name",
          "client.address",
          "client.as.organization.name",
          "client.domain",
          "client.geo.city_name",
          "client.geo.continent_name",
          "client.geo.country_iso_code",
          "client.geo.country_name",
          "client.geo.name",
          "client.geo.region_iso_code",
          "client.geo.region_name",
          "client.mac",
          "client.registered_domain",
          "client.top_level_domain",
          "client.user.domain",
           ...
2

There are a few options for you, but the main challenge is that most of them aim to exclude fields, rather than only include the fields you want.

For example you can use a Metricbeat processor to drop fields Drop fields from events | Metricbeat Reference [8.9] | Elastic or you can use an Ingest Pipeline in Elasticsearch to shape the document before it's saved Remove processor | Elasticsearch Guide [8.9] | Elastic.

If you're using Elastic Agent, soon you'll have access to TSDS which can perform down sampling Downsampling a time series data stream | Elasticsearch Guide [8.9] | Elastic.
Until then, your best options is probably to use an ILM policy to manage the size of your data Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.9] | Elastic.
You might also consider creating rolls up of data if you don't need the same granularity but still want to retain the data Rollup overview | Elasticsearch Guide [8.9] | Elastic.

3

Thank you so much!! I'll include your feedback!

2023년 9월 12일 (화) 오후 4:44, Milton Hultgren via Discuss the Elastic Stack <notifications@elastic.discoursemail.com>님이 작성:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.