Logstash 8.0 is adding extra field

[kannan_raj]

logstash consuming JSON messages from Kafka and indexing them into Elasticsearch. we found that logstash version 8.0 is adding the additional field event with the value of the actual message.

could you please let me know how to ignore the field event without using mutate filter?

actual message is

{
   "name": "test"
}

Doc indexed into Elasticsearch from logstash version 8.0

{
    "_index" : "test-2022.02.15-000001",
    "_id" : "3-eHC38BJKmKsTb9eGMU",
    "_score" : 1.0,
    "_source" : {
        "name" : "test",
        "@version" : "1",
        "event" : {
            "original" : "{\"name\": \"test\"}"
        },
    "@timestamp" : "2022-02-18T06:31:41.467701Z"
}

Doc indexed into Elasticsearch from logstash version 7.16.2

{
    "_index" : "test-2022.02.15-000001",
    "_id" : "4OeKC38BJKmKsTb9xWPN",
    "_score" : 1.0,
    "_source" : {
        "@timestamp" : "2022-02-18T06:35:18.018Z",
        "name" : "test",
        "@version" : "1"
    }
}

configuration

input
{
	kafka
        {
            bootstrap_servers => ["hostname:9093"]
            topics => ["acl"]
            ssl_keystore_password => "changeme"
            ssl_truststore_password => "changeme"
            ssl_keystore_location => "/opt/logstash/ssl/keystore.jks"
            ssl_truststore_location => "/opt/logstash/ssl/truststore.jks"
            security_protocol => "SSL"
            client_id => "client"
            group_id => "group"
            enable_auto_commit => "true"
            auto_offset_reset => "latest"
            check_crcs => "false"
            decorate_events => true
            consumer_threads => "1"
            codec => "json"
	}
}

output {
      elasticsearch {
		hosts => ["https://hostname:9200"]
		user => "user"
		password => "password"
		ilm_enabled => true
		ilm_rollover_alias => "alias"
		ilm_policy => "ilm"
		ssl => true
                cacert => "ca.pem"
		ssl_certificate_verification => true
		manage_template => false
      }
}

logstash.yml

node.name: hostname
path.data: /data/logstash
config.reload.automatic: True
config.reload.interval: 30
queue.type: persisted
path.queue: /data/logstash/queue
queue.page_capacity: 500mb
queue.max_events: 0
queue.max_bytes: 1gb
queue.checkpoint.acks: 0
queue.checkpoint.writes: 0
queue.checkpoint.interval: 0
http.host: 127.0.0.1
http.port: 9600
log.level: info
path.logs: /data/logs/logstsah

[leandrojmp (Leandro Pereira)]

It seems to be related to the ecs compatibility as event.original is an ecs field.

Try to set pipeline.ecs_compatibility: disabled for your pipeline in pipelines.yml or for all pipelines putting the setting in logstash.yml.

[grants]

Leaving pipeline.ecs_compatibility as default will enable this in version 8 of logstash. Older versions this wasn't enabled by default.

If you set pipeline.ecs_compatibility: disabled globally it seems to fix this issue, but if it's left default and disabled for all plugins at the pipeline, it still seems to populate this value.

I wish there was some documentation on this, why it's here, and what other fields logstash adds by default.

Tested:
logstash-6.8.23 (off by default)
logstash-7.17.1 (off by default)
logstash-8.0.1 (on by default)

To further test how this gets added or what plugin is to blame, I disabled the setting globally and enabled it on each plugin but this field didn't show up. Honestly pretty strange behavior.

[leandrojmp (Leandro Pereira)]

The pipeline.ecs_compatiblity being enabled as default in version 8 is on the list of breaking changes.

• Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS). This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting. In Logstash 8, this compatibility mode will be on-by-default for all pipelines. If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8, you can set pipeline.ecs_compatibility: disabled to its definition in pipelines.yml (or globally in logstash.yml ).

Also, some plugins have the information of the names of the fields it creates with the ecs compatibility on and off, for example the file input plugin and the http input plugin, but I'm not sure if the documentation for every plugin has this information yet.