Warning 'Relying on default value of `pipeline.ecs_compatibility' after updating to logstash 7.16.1

[yaauie (Ry Biesemeyer)]

If you want the entire pipeline to behave in 8.x EXACTLY as it does with 7.13, your best path forward is to set pipeline.ecs_compaitibility for the pipeline.

Since you are using a configuration that gives you exactly one pipeline per Logstash process, doing so in the logstash.yml is reasonable. If you were running multiple pipelines, then doing so for each applicable pipeline in its pipelines.yml entry would be best.

If, for some reason, you want to avoid the easy path above (hint: you probably don't), it is possible to resolve this deprecation warning by adjusting your pipeline definition so that each and every plugin definition has an ecs_compatibility => disabled directive. This entails defining your codecs in block form, so codec => json becomes codec => json { ecs_compatibility => disabled }.

I encourage you to take the easy path, to lock in the behaviour you want for the entire pipeline.

[chferng]

@yaauie I was already using the individual plugin definition as shown in my earlier post here and that did not stop the deprecation messages from being generated.

Since I am using the helm chart and values yaml from Bitnami, I suppose I will have to turn to them to understand how the logstash.yml can be configured in the values yaml

[robert_sahakyan]

Hi @yaauie

I seem to be getting a similar issue with excessive log messages about ecs_compatibility & 'target' option after upgrading to the latest version of logstash (v8.1), specifically when using the http plugin.

The log message I want to get rid of is:

[INFO] 2022-03-23 14:02:34.265 [[metrics]-pipeline-manager] json - ECS compatibility is enabled but 'target' option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the 'target' option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)

I have narrowed it down to only be an issue with the http input plugins which seems to be causing this excessive logging across all pipelines using this plugin.

I have tried setting ecs_compatibility: disabled across all three levels (logstash.yml, pipelines.yml & individual plugins) but none seem to get rid of this log.

What confuses me is that when the pipeline starts, it tells me that the ecs_compatibility mode is set to disabled, but yet, a few lines later, it still complains about it being enabled.

That is, I first get:

... Pipeline 'metrics' is configured with 'pipeline.ecs_compatibility: disabled' setting. All plugins in this pipeline will default to 'ecs_compatibility: disabled' ...

followed by:

[INFO] 2022-03-23 14:02:34.265 [[metrics]-pipeline-manager] json - ECS compatibility is enabled but 'target' option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the 'target' option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)

Here is my pipeline config:

input {

    # commented out other inputs to isolate the issue

    http { 
        port => "8111"
        type => http
    }
}

filter {
 # some code here, but it is commented out and still produces the log.
}

output { 
    pipeline {
        send_to => ["rollup_process"]
    }
}

pipelines.yml:

- pipeline.id metrics
  path.config: "${UAM_HOME}/pipelines/metrics.cfg"
  pipeline.workers: 1
  pipeline.ecs_compatibility: disabled

Please advise what I could try, or perhaps this is a bug?