Tags not applied by json filter when source sets HTTP header Content-Type: application/json

[nnet]

Hi, I set up an http listener, and json filter and discovered tag application inconsistencies.
ELK components 5.6.16.

Input:

input {
  http {
  port => "5048"
  tags => [ "CURL" ]
  type => "curl"
  }
}

Filter:

filter {
  if [type] == "curl" {
    json {
      source => "message"
      id => "curl"
      add_tag => [ "JSON","HTTP_INPUT" ]
    }
  }
}

Via curl I send a json object:

curl -XPOST http://localhost:5048 -d '{"dynamic": { "ipaddress": "192.168.17.100", "hostname": "ddns-192-168-17-100", "dnsdomainname": "dhcp.ldev", "fqdn": "ddns-192-168-17-100.dhcp.ldev" }}'

Input is parsed and filter tags are applied.

If I add:
-H 'Content-Type: application/json'
to curl commandline, the object is parsed, but filter tags are not applied.

If I set tags in the input conf, those tags remain set using the above curl arg.

I'm aware that http input defaults to application/json but what I found was data in the form of
[{"key1": "val1"},{"key2": "val2"}]
won't parse as two separate json objects unless -H 'Content-Type: application/json' is added to curl commandline.
This is how I discovered tags weren't being added by the filter. Whether a single object, or multiple objects in one blob, no tags applied from filter.

The expectation is to see all configured tags from each step of the pipeline applied when successfully parsed.

Can anyone shed any light on why the HTTP header Content-Type: application/json prevents tag application in json filter?

Thanks

[Badger]

If you have application/json, is the http input doing the parsing? If you remove the json filter does it still get parsed?

[nnet]

Hmmm very interesting...

What I have found after disabling the json filter is that all fields are parsed when Content-Type: application/json header is explicitly set.

If that header isn't passed, the only fields parsed are the headers.* fields presumably provided by the http input plugin itself. Also a message field is created that contains the whole raw json object.

Below is a screenshot from kibana showing the top event as fully parsed, and the second event as not:

Note the headers.content_type field in each and the static.content_type_defined field I added to indicate when I added Content-Type: application/json on curl commandline.

I believe I am wrong in believing the default Content-Type was application/json.
I also believe I don't need to have an additional filter to parse json.

Do you concur?

[Badger]

Yeah, I don't understand the details of the code, but it is clear that the http input switches codecs based on the Content-Type header.