Elastic agent (metricbeat logs): Cannot index event publisher.Event

Elastic Stack Beats
, ,
1

Hi,
I have recently installed a fleet-managed Elastic Agent on my servers.
my metricbeat_monitor-json.log is flooded with this message:

{"log.level":"warn","@timestamp":"2021-08-24T15:26:57.009+0430","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event
{Content:beat.Event{Timestamp:time.Time{wall:0xc04151fdfa1f75b2, ext:100618413248, loc:(*time.Location)(0x55f1d31de540)}, Meta:{\"raw_index\":\"metrics-elastic_agent.elastic_agent-default\"},
 Fields:{\"agent\":{\"ephemeral_id\":\"d58f26d0-b6d5-460b-b58c-f9537f6fa6b9\",\"hostname\":\"WebSite\",\"id\":\"70a0303b-52c6-4545-8255-e792c1874e46\",\"name\":\"WebSite\",\"type\":\"metricbeat\",\"version\":\"7.14.0\"},
\"data_stream\":{\"dataset\":\"elastic_agent.elastic_agent\",\"namespace\":\"default\",\"type\":\"metrics\"},\"ecs\":{\"version\":\"1.10.0\"},\"elastic_agent\":{\"id\":\"70a0303b-52c6-4545-8255-e792c1874e46\",\"process\":\"elastic-agent\",\"snapshot\":false,\"version\":\"7.14.0\"},
\"event\":{\"dataset\":\"elastic_agent.elastic_agent\",\"duration\":3218409,\"module\":\"http\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"WebSite\",\"id\":\"c56358f080224ce980088fdfb18ab45c\",\"ip\":[\"192.168.100.22\",\"fe80::7403:c2ff:fe62:de39\"],\"mac\":[\"76:04:c8:62:dc:39\"],
\"name\":\"WebSite\",\"os\":{\"codename\":\"Core\",\"family\":\"redhat\",\"kernel\":\"3.10.0-1160.25.1.el7.x86_64\",\"name\":\"CentOS Linux\",\"platform\":\"centos\",\"type\":\"linux\",\"version\":\"7 (Core)\"}},\"metricset\":{\"name\":\"json\",\"period\":10000},\"service\":{\"address\":\"http://unix/stats\",\"type\":\"http\"},
\"system\":{\"process\":{\"cpu\":{\"system\":{\"ticks\":90040.000000,\"time\":{\"ms\":90044.000000}},\"total\":{\"ticks\":281690.000000,\"time\":{\"ms\":281699.000000},\"value\":281690.000000},\"user\":{\"ticks\":191650.000000,\"time\":{\"ms\":191655.000000}}},\"fd\":{\"limit\":{\"hard\":4096.000000,\"soft\":1024.000000},\"open\":19.000000},
\"memory\":{\"size\":75580424.000000}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [metrics-elastic_agent.elastic_agent-0.0.7] does not exist\"}","service.name":"metricbeat","event.dataset":"metricbeat_monitor-json.log","ecs.version":"1.6.0"}

I ended up disabling all Metricbeats related logs in "Linux" and "System" integrations.
Any suggestions would be appreciated.

2

Which version of Elastic Agent and Kibana are you on? Is it possible that you are on 7.13? Even though the issue should not be seen on 7.13 it should resolve itself when you upgrade to 7.14. Please make sure the Elastic Agent integration on the Integrations page in Kibana is on version 1.0 or newer.

3

Elastic Agent and Kibana are 7.14.0 and Elastic Agent in integration page is version 1.0.0 . but I haven't add Elastic Agent integration to any of my policies.

4

I'm asking because of this error in the logs:

status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [metrics-elastic_agent.elastic_agent-0.0.7] does not exist\"}

Did you upgrade this setup from 7.13? I'm trying to figure out why this pipeline would still be used. If th elastic_agent package 1.0 is installed, this pipeline should not be in use anymore. The package is installed by default as elastic_agent ships logs and metrics (if enabled) to these data streams.

Assuming you upgraded from 7.13, do you see it only in the "old" logs or this keeps happening? If it keeps happening, I'm curious to see what the mapping of the metrics-elastic_agent.elastic_agent-defaultlooks like. Especially what the ingest pipeline setting is inside. Any chance you could take a look there? You should see this in Kibana under Index Management -> Data Streams.

5

Sorry for the late response.
I used to have Filebeat and Metricbeat on these servers (version 7.9) but I uninstalled those instances and installed elastic agent 7.14 instead. Elasticsearch and Kibana were upgraded from version 7.13.

I keep receiving these logs.

I hope this is what you meant.

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "metrics"
        },
        "codec": "best_compression",
        "mapping": {
          "total_fields": {
            "limit": "10000"
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "1",
        "final_pipeline": ".fleet_final_pipeline-1",
        "query": {
          "default_field": [
            "cloud.account.id",
            "cloud.availability_zone",
            "cloud.instance.id",
            "cloud.instance.name",
            "cloud.machine.type",
            "cloud.provider",
            "cloud.region",
            "cloud.project.id",
            "cloud.image.id",
            "container.id",
            "container.image.name",
            "container.name",
            "host.architecture",
            "host.domain",
            "host.hostname",
            "host.id",
            "host.mac",
            "host.name",
            "host.os.family",
            "host.os.kernel",
            "host.os.name",
            "host.os.platform",
            "host.os.version",
            "host.os.build",
            "host.os.codename",
            "host.type",
            "elastic_agent.id",
            "elastic_agent.process",
            "elastic_agent.version",
            "system.process.cgroup.id",
            "system.process.cgroup.path",
            "system.process.cgroup.cpu.id",
            "system.process.cgroup.cpu.path",
            "system.process.cgroup.cpuacct.id",
            "system.process.cgroup.cpuacct.path",
            "system.process.cgroup.memory.id",
            "system.process.cgroup.memory.path",
            "system.process.cgroup.blkio.id",
            "system.process.cgroup.blkio.path"
          ]
        },
        "number_of_routing_shards": "30"
      }
    },
    "mappings": {
      "dynamic": "false",
      "_meta": {
        "package": {
          "name": "elastic_agent"
        },
        "managed_by": "ingest-manager",
        "managed": true
      },
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "availability_zone": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "image": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "provider": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "container": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "image": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "labels": {
              "type": "object"
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },

...
...
...
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}
6

The part I'm looking for is the final mapping. I assume what you shared here is the content of the template? Get mapping API | Elasticsearch Guide [7.14] | Elastic

If it is the final mapping, it seems only the final ingest pipeline is inside which would be an issue.

7

sorry, here is the result of GET metrics-elastic_agent.elastic_agent-default/_mapping:

{
  ".ds-metrics-elastic_agent.elastic_agent-default-2021.08.14-000001" : {
    "mappings" : {
      "dynamic" : "false",
      "_meta" : {
        "package" : {
          "name" : "elastic_agent"
        },
        "managed_by" : "ingest-manager",
        "managed" : true
      },
      "_data_stream_timestamp" : {
        "enabled" : true
      },
      "dynamic_templates" : [
        {
          "strings_as_keyword" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "cloud" : {...},
	"container" : {...},
	"data_stream" : {...},
	"elastic_agent" : {...},
	"host" : {...},
	 "system" :{...}
	  }
    }
  }
}
8

I'm really sorry, I sent you down the wrong path. I thought the settings are also listed under the _mappings API call but they are not :man_facepalming: Instead there is a separate settings call GET metrics-elastic_agent.elastic_agent-default/_settings. The part I'm looking for is if there is any default_pipeline defined there besides the final pipeline. For metrics, no pipeline should be defined.

9

No worries, here is my metrics settings.
default_pipeline is defined, should I remove it?

{
  ".ds-metrics-elastic_agent.elastic_agent-default-2021.08.14-000001" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "hidden" : "true",
        "provided_name" : ".ds-metrics-elastic_agent.elastic_agent-default-2021.08.14-000001",
        "query" : {
          "default_field" : [
            "cloud.account.id",
            "cloud.availability_zone",
            "cloud.instance.id",
            "cloud.instance.name",
            "cloud.machine.type",
            "cloud.provider",
            "cloud.region",
            "cloud.project.id",
            "cloud.image.id",
            "container.id",
            "container.image.name",
            "container.name",
            "host.architecture",
            "host.domain",
            "host.hostname",
            "host.id",
            "host.mac",
            "host.name",
            "host.os.family",
            "host.os.kernel",
            "host.os.name",
            "host.os.platform",
            "host.os.version",
            "host.os.build",
            "host.os.codename",
            "host.type",
            "elastic_agent.id",
            "elastic_agent.process",
            "elastic_agent.version",
            "system.process.cgroup.id",
            "system.process.cgroup.path",
            "system.process.cgroup.cpu.id",
            "system.process.cgroup.cpu.path",
            "system.process.cgroup.cpuacct.id",
            "system.process.cgroup.cpuacct.path",
            "system.process.cgroup.memory.id",
            "system.process.cgroup.memory.path",
            "system.process.cgroup.blkio.id",
            "system.process.cgroup.blkio.path"
          ]
        },
        "creation_date" : "1628955030431",
        "number_of_replicas" : "1",
        "uuid" : "pSLp9eZITpy1f9Vy4iSm1w",
        "version" : {
          "created" : "7130299"
        },
        "lifecycle" : {
          "name" : "metrics"
        },
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_hot"
            }
          }
        },
        "number_of_shards" : "1",
        "default_pipeline" : "metrics-elastic_agent.elastic_agent-0.0.7"
      }
    }
  }
}
10

Here we have the problem:

 "default_pipeline" : "metrics-elastic_agent.elastic_agent-0.0.7"

As you are on 7.14 and have Elastic Agent package version 1.0.0, the question is why this is still there.

@nchaulet @joshdover During upgrade of the package, if there is no conflict mapping change, we just apply the new settings and mappings. I'm now wondering if there is maybe an issue with updating the settings.

@Gomeisa I would expect, that a rollover of the data stream should solve your issue: Rollover API | Elasticsearch Guide [master] | Elastic This assumes that in the settings of the template for Elastic Agent is the correct pipeline. @nchaulet Is there a pipeline for metrics?

11

@ruflin thank you :pray:

using POST metrics-elastic_agent.elastic_agent-default/_rollover solved my problem.

12

Great to hear it solved your issue. On our end we still need to figure out how you ended up in this situation (which should not happen).

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.