# Logstash 8.10.4 breaking changes

**URL:** <https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312>\
**Category:** Logstash\
**Created:** [November 2, 2023, 5:18pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312 "2023-11-02T17:18:02Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [November 2, 2023, 5:18pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/1 "2023-11-02T17:18:02Z")

</div>

```auto

 "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'xxxxxxx'. 

```

added this to resolve the above

```auto
mutate {
    rename => { "[host]" => "[host][name]" }
  }

```

```auto
output: 
 "host"=>{"name"=>{"name"=>"brmop-dff102.site-mpp.local
 I expected host field like host => "brmop-dff102.site-mpp.local" or host.name => "brmop-dff102.site-mpp.local"

```

```auto
"host": {
"properties": {
  "name": {
    "type": "text",
    "fields": {
      "keyword": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
}
},

```

All the configurations are same. I recieve the below error

```auto
"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host.name] of type [text] in document with id 'qZYfkIsBrdR7-jvdNETu'.

```

After using latest logstash version 8.10.4. So many behavior seems to be breaking.  
Althought ecs\_compatibilty =\>disabled.

Having same configuration between all the different logstash should behave the same.

If I upgrade the existing from logstash 7.17.6 to 8.10.4 I see all mapping issues. If I bring up the new setup I don't see the mapping exception. Please help me understand what's going on?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 2, 2023, 5:31pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/2 "2023-11-02T17:31:40Z")

</div>

Hello,

It is not clear what is your issue, you didn't share the full error you got, also it is confusing what you are trying to do, what you expect, and what is not working as expected.

Can you provide more context about your issue?

> [@ranjini](#):
>
> Having same configuration between all the different logstash should behave the same.

Are they on the same version and have the same configurations in `logstash.yml`? If their versions are diferent, like one on version 7.X and other in version 8.X, and not all configurations are the same in `logstash.yml` then it is expected that they can behave differently, there are a couple of breaking changes from version 7 to version 8.

> [@ranjini](#):
>
> If I bring up the new setup I don't see the mapping exception.

It is not clear what you mention with this, which new setup? What version of Logstash?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [November 2, 2023, 5:49pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/3 "2023-11-02T17:49:09Z")

</div>

Might be here is a problem:

```auto
mutate {
    rename => { "[host]" => "[host][name]" }
  }

```

The "[host](https://www.elastic.co/guide/en/ecs/current/ecs-host.html)" is a JSON nested field, not a string with ECS enabled.

Can you show a full message which cause the problem?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 2, 2023, 5:53pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/4 "2023-11-02T17:53:55Z")

</div>

> [@ranjini](#):
>
> `"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'xxxxxxx'. `

This is a mapping error, it means that in your index the mapping for the field `host` is a `text` and logstash probably tryind to send an object.

The following mutate you shared would not solve this issue, so not sure what else you did as you didn't provide much context.

> [@ranjini](#):
>
> ```auto
> output: 
> "host"=>{"name"=>{"name"=>"brmop-dff102.site-mpp.local
> I expected host field like host => "brmop-dff102.site-mpp.local" or host.name => "brmop-dff102.site-mpp.local"
> 
> ```

You didn't shared what is your input, but your input probably add a field named `host.name` with the value `brmop-dff102.site-mpp.local`, you will have this in your event.

```auto
{ "host": { "name": "brmop-dff102.site-mpp.local" } }

```

Renaming `host` to `host.name` will rename the top level field, and you will end up with:

```auto
{ "host": { "name": { "name": "brmop-dff102.site-mpp.local" } } }

```

Which again, will not match the mapping you shared as it expect the `host.name` field to be a text field, not an object.

It is not clear what you did between your first error and the second one, but while they are exactly the same kind of error, the mapping was changed.

---

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [November 2, 2023, 5:58pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/5 "2023-11-02T17:58:46Z")

</div>

I have been using logstash 7.17.6. Now I have upgraded logstash to 8.10.4 (latest)  
Although I have set ecs\_compatibilty =\>disabled. in the output section

I received the below error:

```auto
hash"=>"2c497c34ca7f0c8704e095a24856dce9fa830e34"}, "@timestamp"=>2023-10-30T23:59:55.777381206Z, "env"=>"dim0", "node"=>"dim0-ope-mum-server_eti", "productCode"=>"dim0-ope", "type"=>"messages", "host"=>{"name"=>"poe.st.019181.site-dim0.local"}, "@version"=>"1"}], :response=>{"index"=>{"_index"=>"dim0-ope-mum-server_eti-2023.10.30", "_id"=>"JMAHg4sBOPcF7zZGnYSo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'JMAHg4sBOPcF7zZGnYSo'. Preview of field's value: '{name=poe.st.019181.site-od2.local}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:184751"}}}}}

```

To Resolve the above error  
Added this to the filter

```auto
mutate {
    rename => { "[host]" => "[host][name]" }
  }

```

But After this It started complaining about host.name field

```auto
{:status=>400, :action=>["index", {:_id=>nil, :_index=>"mppp.local-2023.11.02", :routing=>nil}, {"@timestamp"=>2023-11-02T13:00:46.074416724Z, "message"=>"type=PROCTITLE msg=audit(1698930045.556:1846): proctitle=2F7573722F7362696E2F6368726F6E7964002D460032002D75006368726F6E79", "podName"=>"ppe", "@version"=>"1", "tags"=>["_grokparsefailure"], "log"=>{"file"=>{"path"=>"/var/log/audit/audit.log"}}, "type"=>"audit", "productCode"=>"ppe-ope", "nodeRole"=>"mppp.local", "event"=>{"hash"=>"91be6a0655505bf7452edb46fbf5a4a73ec26bf1", "original"=>"type=PROCTITLE msg=audit(1698930045.556:1846): proctitle=2F7573722F7362696E2F6368726F6E7964002D460032002D75006368726F6E79"}, "host"=>{"name"=>{"name"=>"ope-rb-016c28e4.site-ppe.local"}}}], :response=>{"index"=>{"_index"=>"mppp.local-2023.11.02", "_id"=>"qZYfkIsrbdR7-jvdNETu", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host.name] of type [text] in document with id 'qZYfkIsrbdR7-jvdNETu'.
Preview of field's value: '{name=ope-rb-016c28e4.site-ppe.local}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:579"}}}}}

```

If ecs\_compatibilty =\> disabled this mapping should work the way it was working previously.

This issues are seen after my upgrade to 8.10.4

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 2, 2023, 6:16pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/6 "2023-11-02T18:16:45Z")

</div>

> response=\>{"index"=\>{"\_index"=\>"dim0-ope-mum-server\_eti-2023.10.30", "\_id"=\>"JMAHg4sBOPcF7zZGnYSo", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [host] of type [text] in document with id 'JMAHg4sBOPcF7zZGnYSo'. Preview of field's value: '{name=poe.st.019181.site-od2.local}'", "caused\_by"=\>{"type"=\>"illegal\_state\_exception", "reason"=\>"Can't get text on a START\_OBJECT at 1:184751"}}}}}

As mentioned before, this is a mapping error, the mapping for the field `host` in the index `dim0-ope-mum-server_eti-2023.10.30` is of the type `text`, but logstash is trying to send a json object.

You need to fix your template with the correct mapping and create a new index.

> response=\>{"index"=\>{"\_index"=\>"mppp.local-2023.11.02", "\_id"=\>"qZYfkIsrbdR7-jvdNETu", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [host.name] of type [text] in document with id 'qZYfkIsrbdR7-jvdNETu'.  
> Preview of field's value: '{name=ope-rb-016c28e4.site-ppe.local}'", "caused\_by"=\>{"type"=\>"illegal\_state\_exception", "reason"=\>"Can't get text on a START\_OBJECT at 1:579"}}}}}

Also, as mentioned in the previous answer, this is also a mapping error, in this case you have a completely different index with a different mapping, it seems that in the index `mppp.local-2023.11.02` you have the field `host` mapped as a json object with a nested field `name`, mapped as text, but logstash is trying to send a json object in the format `host.name.name: value`, which will not work.

This is probably caused by this mutate you added

```auto
mutate {
    rename => { "[host]" => "[host][name]" }
  }

```

You need to provide your Logstash configuration, you have two different indices, with different mappings, it is not clear if you have just one pipeline, or multiple pipelines, please provide context.

> [@ranjini](#):
>
> Although I have set ecs\_compatibilty =\>disabled. in the output section

Where did you set this? The `ecs_compatibility` can be configured in the `input`, in `filters`, in `logstash.yml` and in `pipelines.yml` as explained in the [documentation](https://www.elastic.co/guide/en/logstash/current/ecs-ls.html).

There is no option to set the `ecs_compatibility` in the output, after all the parse was already finished using ecs fields.

Please share where you set this.

> [@ranjini](#):
>
> This issues are seen after my upgrade to 8.10.4

This is expected, it is a breaking change and it is in the [documentation](https://www.elastic.co/guide/en/logstash/current/breaking-8.0.html#bc-ecs-compatibility).

The issues you shared are mapping issues caused by the ecs compatibility in Logstash, you need to fix your mappings or disable ecs compatibility in the correct place.

---

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [November 2, 2023, 6:38pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/7 "2023-11-02T18:38:30Z")

</div>

@leandrojmp Thank you for your time.  
added

```auto
pipeline.ecs_compatibility: disabled in logstash.yml

```

started logstash with the below command

```auto
ExecStart=/usr/share/logstash/logstash-8.10.4/bin/logstash --path.settings /usr/share/logstash/logstash-8.10.4/config/logstash.yml "-f" "/usr/share/logstash/logstash-8.10.4/conf.d"

```

error message:

```auto

"type"=>"messages", "fingerprint"=>"18a58969db3853ae3eceab89b41791f5329e7c6a", "Code"=>"tem-pen0", "@version"=>"1", "@timestamp"=>2023-11-02T18:29:16.795575949Z, "host"=>"pen0-st-020727bd.local-tem.local", "path"=>"/var/log/messages", "nodeRole"=>"tem-pen0-fox0-ote_server", "podName"=>"tem"}], :response=>{"index"=>{"_index"=>"tem-pen0-fox0-ote_server-2023.11.02", "_id"=>"udlMkYsBIDW79JzaEkAf", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field[host] as object, but found a concrete value"}}}

```

Please help me. Looks like this is a bug.  
Although I disabled pipeline.ecs\_compatibility: disabled it is still having mapping exception

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 2, 2023, 7:25pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/8 "2023-11-02T19:25:54Z")

</div>

> [@ranjini](#):
>
> Please help me. Looks like this is a bug.  
> Although I disabled pipeline.ecs\_compatibility: disabled it is still having mapping exception

This is not a bug, the issue is not with Logstash, you have an issue with your Elasticsearch mapping that does not match your data, you need to validate your mapping and change it.

With ecs disabled logstash is sending the `host` field as a string, you can validate this in the same error message you shared.

> "host"=\>"pen0-st-020727bd.local-tem.local"

The `host` field is a string now, but the mapping for your indice is expecting the `host` field as a json object.

> "reason"=\>"object mapping for [host] tried to parse field[host] as object, but found a concrete value"

This mean that for that index the host field is mapped as a json object, but logstash is now sending a string (concrete value).

Do you have templates for your indices? It looks like you do not have any template for your indices defining the mapping of the fields, in this case Elasticsearch will create the mappings based on the **first** value it receives for a field.

Since you had ecs\_compatibility enabled, the mapping was based on the `host` field as an object, after disabling it you need to use a new index.

Not having a template with a mapping can lead to the issue you are getting.

Do you have Kibana? If Yes, go into Dev Tools and run this request:

```auto
GET tem-pen0-fox0-ote_server-2023.11.02/_mapping

```

This will show you the mapping for this indice, which by the way is the third different index you shared.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 2, 2023, 7:52pm UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/9 "2023-11-02T19:52:08Z")

</div>

Basically when using Logstash 8.10, with ecs compatibility enabled and without any template with the correct mapping, Elasticsearch create the mapping for some issues were the `host` field is a json object.

But since you disabled ecs compatibility, the mapping for the indices already created won't change, and now Logstash is sending the `host` field as a string, which will not be accepted by Elasticsearch.

---

<div class="post-metadata">

**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\
**Post date:** [November 3, 2023, 4:55am UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/10 "2023-11-03T04:55:19Z")

</div>

@leandrojmp  
I really appreciate you for explaining the things with error message, this shows your in depth knowledge in Elastic search and logstash. Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 1, 2023, 4:55am UTC](https://discuss.elastic.co/t/logstash-8-10-4-breaking-changes/346312/11 "2023-12-01T04:55:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.