Filebeat Apache module not showing data on Dashboards when using Logstash

Elastic Stack Elasticsearch
8

If I'm not wrong, this is a breaking change in 8.X related to ecs fields.

From 8.0 the pipeline.ecs_compatibility setting is on by default, so Logstash will output ecs fields and in this case the event.original field is created, in version 7.X the pipeline.ecs_compatibility setting was disabled by default.

For example, if you test the following pipeline in Logstash:

input {
    stdin {}
}
output {
    stdout {}
}

And you set pipeline.ecs_compatibility: v8 in logstash.yml

and you run:

echo "sample message" | /opt/logstash/bin/logstash --path.settings /opt/logstash/config -f /opt/logstash/pipelines/config.conf

You will get this output:

{
         "event" => {
        "original" => "sample message"
    },
       "message" => "sample message",
          "host" => {
        "hostname" => "server"
    },
      "@version" => "1",
    "@timestamp" => 2022-11-10T21:46:56.615Z
}

If you set pipeline.ecs_compatibility as disabled you get this output:

{
      "@version" => "1",
       "message" => "sample message",
          "host" => "server",
    "@timestamp" => 2022-11-10T21:50:08.861Z
}

So if someone is using Logstash with Filebeat or Elastic Agent modules and Elasticsearch Ingest pipelines, they may have some issues because some of those pipelines will try to copy message into event.original, which may already exist.

I think that this proposed change to use @metadata.original instead of event.original would be better.

1 Like