# Prevent collision of field types of structured logs

**URL:** <https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806>\
**Category:** Logstash\
**Created:** [November 25, 2022, 9:58am UTC](https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806 "2022-11-25T09:58:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![katexochen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katexochen/32/113761_2.png) [@katexochen](https://discuss.elastic.co/u/katexochen)\
**Post date:** [November 25, 2022, 9:58am UTC](https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806/1 "2022-11-25T09:58:07Z")

</div>

I'm collecting logs to filebeat, sending them to logstash and from there to elastic. As the logs are structured, I'm using the json filter in logstash to parse the message.

Let's say I'm having two messages/logs in json format:

`message => "{"foo": "bar"}"`

and

`message => "{"foo.addr": "127.0.0.1"}"`

And my filter is

```auto
filter {
    json {
            source => "message"
            target => "mylogs"
            skip_on_invalid_json => true
    }
}

```

I get the following error:

> Could not dynamically add mapping for field `[foo.addr]`. Existing mapping for `[mylogs.foo]` must be of type object but found `[text]`.

I need a solution where I don't need to know the field names of the log message in advanced. I thought to the de\_dot filter to get something like `mylogs.foo` and `mylogs.foo_addr`, but logstash's de\_dot filter requires a concrete filed name to work with:

> Sub-fields must be manually specified in the array.

How can I prevent this type collision for any field that might be parsed from the message?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 27, 2022, 1:30pm UTC](https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806/2 "2022-11-27T13:30:58Z")

</div>

Can you share the error you got and the output document?

Because in Logstash having a document with:

```auto
"{"foo": "bar"}"

```

And another with:

```auto
"{"foo.addr": "127.0.0.1"}"

```

Would not result in a mapping error, in the first case you would end up with the `food` field, and the second one would result in a field named `foo.addr` which has a literal dot in its name.

---

<div class="post-metadata">

**Author:** ![katexochen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katexochen/32/113761_2.png) [@katexochen](https://discuss.elastic.co/u/katexochen)\
**Post date:** [November 29, 2022, 5:01pm UTC](https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806/3 "2022-11-29T17:01:09Z")

</div>

Sorry for the late reply, and the missing information.

Here are some more details and things to reproduce:

logstash.conf:

```auto
input {
    http {
        host => "0.0.0.0"
        port => 5044
    }
}
filter {
        json {
            source => "message"
            target => "logs"
            skip_on_invalid_json => true
        }
}
output {
    opensearch {
        ...
    }
}

```

Doing this:

```auto
curl -XPOST --insecure 'localhost:5044' -H 'Content-Type: application/json' -d '{"foo": "bar"}'
curl -XPOST --insecure 'localhost:5044' -H 'Content-Type: application/json' -d '{"foo.addr": "bar"}'

```

I get the following error (from opensearch/elasticsearch, not from logstash):

```auto
[2022-11-29T16:47:53,144][WARN][logstash.outputs.opensearch][main][51178d5a010d5db4ecb8fd69e0f46c9f336d6e30941d744f1e9f6ce89320ac40] Could not index event to OpenSearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my-first-index", :routing=>nil}, {"headers"=>{"http_version"=>"HTTP/1.1", "request_path"=>"/", "content_type"=>"application/json", "content_length"=>"19", "http_user_agent"=>"curl/7.85.0", "http_accept"=>"*/*", "request_method"=>"POST", "http_host"=>"localhost:5044"}, "host"=>"10.88.0.1", "@version"=>"1", "@timestamp"=>2022-11-29T16:47:52.968Z, "foo.addr"=>"bar"}], :response=>{"index"=>{"_index"=>"my-first-index", "_type"=>"_doc", "_id"=>"TkdJxIQBApPBqyr9qzwW", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Could not dynamically add mapping for field [foo.addr]. Existing mapping for [foo] must be of type object but found [text]."}}}}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2022, 5:01pm UTC](https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806/4 "2022-11-29T17:01:09Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 27, 2022, 5:01pm UTC](https://discuss.elastic.co/t/prevent-collision-of-field-types-of-structured-logs/319806/5 "2022-12-27T17:01:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.