JSON codec plugin - target option (http input)

Elastic Stack Logstash
1

I am having some issues using the JSON codec plugin in my Logstash (v8.1.2) pipeline.

pipeline.conf:

input {
    http {
        codec => json
        port => 5046
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

input:

[
    {"message":"message 1"},
    {"message":"message 2"},
    {"message":"message 3"}
]

The plugin itself generates multiple events if the data sent is a JSON array, so this configuration will generate events like the following:

{ "message" => "message 1" }
{ "message" => "message 2" }
{ "message" => "message 3" }

This also produces the following log message in Logstash:

[INFO][logstash.codecs.json][main][060a60e30dc5b27b453c31a99b7494beee922048c3d0b8261b9894d53753ba46] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)

Setting the target option does not seem to have any effect. Neither the log message nor the structure of the generated events has changed.
The documentation states that this option is used to define the target field for placing the parsed data. By changing this option to [document] I expected the plugin to generate events like this, but that is not the case:

{ "document" => { "message" => "message 1" }}
{ "document" => { "message" => "message 2" }}
{ "document" => { "message" => "message 3" }}

Is there a way to place the parsed data in a target field?
What is the target option used for? Did I misunderstand something here?
Is there a way to suppress the log message?

2

You can append

logger.randomname.name = logstash.codec.json
logger.randomname.level = ERROR

to your log4j2.properties file. When I set the target option on a json codec the fields are moved under the target. For example,

input { generator { count => 1 lines => [ '{ "a": 1 }' ] codec => json { target => "[document]" } } }
filter {}
output { stdout { codec => rubydebug { metadata => false } } }

produces

  "document" => {
    "a" => 1
},
1 Like
3

I can confirm that this works when using the input generator plugin.
However, when using the http input plugin the target option seems to be ignored. For example,

input { 
    http {
        port => 5046
        codec => json {
            target => "[document]"
        }
    }
}
filter {}
output {
    stdout {
        codec => rubydebug {
            metadata => false 
        }
    }
}

+ request

curl -d '{"a":1}' -H "Content-Type: application/json" -X POST http://localhost:5046/

produces

{
    "a" => 1,
    "http" => ...
    "@timestamp" => ...
}
4

That feels like a bug to me. The input applies a codec itself before seeing what codec the user configured. See here. You could try

additional_codecs => {}
codec => json { target => "[document]" }
1 Like
5

Many thanks, that worked!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.