Logstash 8.10.4 breaking changes

Elastic Stack Logstash
1 

 "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'xxxxxxx'.

added this to resolve the above

mutate {
    rename => { "[host]" => "[host][name]" }
  }

output: 
 "host"=>{"name"=>{"name"=>"brmop-dff102.site-mpp.local
 I expected host field like host => "brmop-dff102.site-mpp.local" or host.name => "brmop-dff102.site-mpp.local"

"host": {
"properties": {
  "name": {
    "type": "text",
    "fields": {
      "keyword": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
}
},

All the configurations are same. I recieve the below error

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host.name] of type [text] in document with id 'qZYfkIsBrdR7-jvdNETu'.

After using latest logstash version 8.10.4. So many behavior seems to be breaking.
Althought ecs_compatibilty =>disabled.

Having same configuration between all the different logstash should behave the same.

If I upgrade the existing from logstash 7.17.6 to 8.10.4 I see all mapping issues. If I bring up the new setup I don't see the mapping exception. Please help me understand what's going on?

2

Hello,

It is not clear what is your issue, you didn't share the full error you got, also it is confusing what you are trying to do, what you expect, and what is not working as expected.

Can you provide more context about your issue?

Are they on the same version and have the same configurations in logstash.yml? If their versions are diferent, like one on version 7.X and other in version 8.X, and not all configurations are the same in logstash.yml then it is expected that they can behave differently, there are a couple of breaking changes from version 7 to version 8.

It is not clear what you mention with this, which new setup? What version of Logstash?

3

Might be here is a problem:

mutate {
    rename => { "[host]" => "[host][name]" }
  }

The "host" is a JSON nested field, not a string with ECS enabled.

Can you show a full message which cause the problem?

4

This is a mapping error, it means that in your index the mapping for the field host is a text and logstash probably tryind to send an object.

The following mutate you shared would not solve this issue, so not sure what else you did as you didn't provide much context.

You didn't shared what is your input, but your input probably add a field named host.name with the value brmop-dff102.site-mpp.local, you will have this in your event.

{ "host": { "name": "brmop-dff102.site-mpp.local" } }

Renaming host to host.name will rename the top level field, and you will end up with:

{ "host": { "name": { "name": "brmop-dff102.site-mpp.local" } } }

Which again, will not match the mapping you shared as it expect the host.name field to be a text field, not an object.

It is not clear what you did between your first error and the second one, but while they are exactly the same kind of error, the mapping was changed.

1 Like
5

I have been using logstash 7.17.6. Now I have upgraded logstash to 8.10.4 (latest)
Although I have set ecs_compatibilty =>disabled. in the output section

I received the below error:

hash"=>"2c497c34ca7f0c8704e095a24856dce9fa830e34"}, "@timestamp"=>2023-10-30T23:59:55.777381206Z, "env"=>"dim0", "node"=>"dim0-ope-mum-server_eti", "productCode"=>"dim0-ope", "type"=>"messages", "host"=>{"name"=>"poe.st.019181.site-dim0.local"}, "@version"=>"1"}], :response=>{"index"=>{"_index"=>"dim0-ope-mum-server_eti-2023.10.30", "_id"=>"JMAHg4sBOPcF7zZGnYSo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'JMAHg4sBOPcF7zZGnYSo'. Preview of field's value: '{name=poe.st.019181.site-od2.local}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:184751"}}}}}

To Resolve the above error
Added this to the filter

mutate {
    rename => { "[host]" => "[host][name]" }
  }

But After this It started complaining about host.name field

{:status=>400, :action=>["index", {:_id=>nil, :_index=>"mppp.local-2023.11.02", :routing=>nil}, {"@timestamp"=>2023-11-02T13:00:46.074416724Z, "message"=>"type=PROCTITLE msg=audit(1698930045.556:1846): proctitle=2F7573722F7362696E2F6368726F6E7964002D460032002D75006368726F6E79", "podName"=>"ppe", "@version"=>"1", "tags"=>["_grokparsefailure"], "log"=>{"file"=>{"path"=>"/var/log/audit/audit.log"}}, "type"=>"audit", "productCode"=>"ppe-ope", "nodeRole"=>"mppp.local", "event"=>{"hash"=>"91be6a0655505bf7452edb46fbf5a4a73ec26bf1", "original"=>"type=PROCTITLE msg=audit(1698930045.556:1846): proctitle=2F7573722F7362696E2F6368726F6E7964002D460032002D75006368726F6E79"}, "host"=>{"name"=>{"name"=>"ope-rb-016c28e4.site-ppe.local"}}}], :response=>{"index"=>{"_index"=>"mppp.local-2023.11.02", "_id"=>"qZYfkIsrbdR7-jvdNETu", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host.name] of type [text] in document with id 'qZYfkIsrbdR7-jvdNETu'.
Preview of field's value: '{name=ope-rb-016c28e4.site-ppe.local}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:579"}}}}}

If ecs_compatibilty => disabled this mapping should work the way it was working previously.

This issues are seen after my upgrade to 8.10.4

6

response=>{"index"=>{"_index"=>"dim0-ope-mum-server_eti-2023.10.30", "_id"=>"JMAHg4sBOPcF7zZGnYSo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'JMAHg4sBOPcF7zZGnYSo'. Preview of field's value: '{name=poe.st.019181.site-od2.local}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:184751"}}}}}

As mentioned before, this is a mapping error, the mapping for the field host in the index dim0-ope-mum-server_eti-2023.10.30 is of the type text, but logstash is trying to send a json object.

You need to fix your template with the correct mapping and create a new index.

response=>{"index"=>{"_index"=>"mppp.local-2023.11.02", "_id"=>"qZYfkIsrbdR7-jvdNETu", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host.name] of type [text] in document with id 'qZYfkIsrbdR7-jvdNETu'.
Preview of field's value: '{name=ope-rb-016c28e4.site-ppe.local}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:579"}}}}}

Also, as mentioned in the previous answer, this is also a mapping error, in this case you have a completely different index with a different mapping, it seems that in the index mppp.local-2023.11.02 you have the field host mapped as a json object with a nested field name, mapped as text, but logstash is trying to send a json object in the format host.name.name: value, which will not work.

This is probably caused by this mutate you added

mutate {
    rename => { "[host]" => "[host][name]" }
  }

You need to provide your Logstash configuration, you have two different indices, with different mappings, it is not clear if you have just one pipeline, or multiple pipelines, please provide context.

Where did you set this? The ecs_compatibility can be configured in the input, in filters, in logstash.yml and in pipelines.yml as explained in the documentation.

There is no option to set the ecs_compatibility in the output, after all the parse was already finished using ecs fields.

Please share where you set this.

This is expected, it is a breaking change and it is in the documentation.

The issues you shared are mapping issues caused by the ecs compatibility in Logstash, you need to fix your mappings or disable ecs compatibility in the correct place.

7

@leandrojmp Thank you for your time.
added

pipeline.ecs_compatibility: disabled in logstash.yml

started logstash with the below command

ExecStart=/usr/share/logstash/logstash-8.10.4/bin/logstash --path.settings /usr/share/logstash/logstash-8.10.4/config/logstash.yml "-f" "/usr/share/logstash/logstash-8.10.4/conf.d"

error message:


"type"=>"messages", "fingerprint"=>"18a58969db3853ae3eceab89b41791f5329e7c6a", "Code"=>"tem-pen0", "@version"=>"1", "@timestamp"=>2023-11-02T18:29:16.795575949Z, "host"=>"pen0-st-020727bd.local-tem.local", "path"=>"/var/log/messages", "nodeRole"=>"tem-pen0-fox0-ote_server", "podName"=>"tem"}], :response=>{"index"=>{"_index"=>"tem-pen0-fox0-ote_server-2023.11.02", "_id"=>"udlMkYsBIDW79JzaEkAf", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field[host] as object, but found a concrete value"}}}

Please help me. Looks like this is a bug.
Although I disabled pipeline.ecs_compatibility: disabled it is still having mapping exception

8

This is not a bug, the issue is not with Logstash, you have an issue with your Elasticsearch mapping that does not match your data, you need to validate your mapping and change it.

With ecs disabled logstash is sending the host field as a string, you can validate this in the same error message you shared.

"host"=>"pen0-st-020727bd.local-tem.local"

The host field is a string now, but the mapping for your indice is expecting the host field as a json object.

"reason"=>"object mapping for [host] tried to parse field[host] as object, but found a concrete value"

This mean that for that index the host field is mapped as a json object, but logstash is now sending a string (concrete value).

Do you have templates for your indices? It looks like you do not have any template for your indices defining the mapping of the fields, in this case Elasticsearch will create the mappings based on the first value it receives for a field.

Since you had ecs_compatibility enabled, the mapping was based on the host field as an object, after disabling it you need to use a new index.

Not having a template with a mapping can lead to the issue you are getting.

Do you have Kibana? If Yes, go into Dev Tools and run this request:

GET tem-pen0-fox0-ote_server-2023.11.02/_mapping

This will show you the mapping for this indice, which by the way is the third different index you shared.

9

Basically when using Logstash 8.10, with ecs compatibility enabled and without any template with the correct mapping, Elasticsearch create the mapping for some issues were the host field is a json object.

But since you disabled ecs compatibility, the mapping for the indices already created won't change, and now Logstash is sending the host field as a string, which will not be accepted by Elasticsearch.

1 Like
10

@leandrojmp
I really appreciate you for explaining the things with error message, this shows your in depth knowledge in Elastic search and logstash. Thank you.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.