# Logs drop on field conflict

**URL:** https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259
**Category:** Elasticsearch
**Created:** [April 24, 2018, 8:31am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259 "2018-04-24T08:31:12Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![slavik57](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slavik57/32/22870_2.png) [@slavik57](https://discuss.elastic.co/u/slavik57)
#### Post date: [April 24, 2018, 8:31am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/1 "2018-04-24T08:31:12Z")

</div>

I am using `ELK 5.5.0`  
When I send a log containing a field with conflicted type it doesn't show up in Kibana but I cannot find any error in the logs.

Example:

```auto
curl -H "content-type: application/json" -XPOST 'http://localhost:9191' -d '{
"message": "This is a log line",
"extrafield": 1
}'

curl -H "content-type: application/json" -XPOST 'http://localhost:9191' -d '{
"message": "This is a log line",
"extrafield": "no i am string"
}'

```

The first log sets the `extrafield` type to be a number.  
The second log is simply ignored without any errors.

Am I missing a place where the errors can be written?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 8:44am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/2 "2018-04-24T08:44:54Z")

</div>

It seems like you are missing the index and type in your request. If I run the following:

```auto
curl -H "content-type: application/json" -XPOST 'http://localhost:9200/test/doc?pretty' -d '{
  "message": "This is a log line",
  "extrafield": 1
}'

curl -H "content-type: application/json" -XPOST 'http://localhost:9200/test/doc?pretty' -d '{
  "message": "This is a log line",
  "extrafield": "no i am string"
}'

```

I get an error in the response for the second request:

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [extrafield]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse [extrafield]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "For input string: \"no i am string\""
    }
  },
  "status" : 400
}

```

---

<div class="post-metadata">

### Author: ![slavik57](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slavik57/32/22870_2.png) [@slavik57](https://discuss.elastic.co/u/slavik57)
#### Post date: [April 24, 2018, 10:10am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/3 "2018-04-24T10:10:34Z")

</div>

Changed to:

```auto
curl -H "content-type: application/json" -XPOST 'http://localhost:9191/logs' -d '{
"message": "This is a log line",
"extrafield": 1
}'

curl -H "content-type: application/json" -XPOST 'http://localhost:9191/logs' -d '{
"message": "This is a log line",
"extrafield": "no i am string"
}'

```

Still same behaviour.

---

<div class="post-metadata">

### Author: ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)
#### Post date: [April 24, 2018, 10:15am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/4 "2018-04-24T10:15:57Z")

</div>

This should show up in Elasticsearch logs as an error.

To get around this you have to set `index.mapping.ignore_malformed=true` for `extrafield`

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 10:18am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/5 "2018-04-24T10:18:51Z")

</div>

You are still not specifying the type, which is causing the requests to fail. Please try running the example I provided.

---

<div class="post-metadata">

### Author: ![slavik57](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slavik57/32/22870_2.png) [@slavik57](https://discuss.elastic.co/u/slavik57)
#### Post date: [April 24, 2018, 10:27am UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/6 "2018-04-24T10:27:31Z")

</div>

I am using logstash so it hid the message for me.  
When i sent the request directly to elastic it showed the error

---

<div class="post-metadata">

### Author: ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)
#### Post date: [April 24, 2018, 12:10pm UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/7 "2018-04-24T12:10:36Z")

</div>

One more thing. If you can avoid having different types of data for the same key, do it. I don't have much control of what format logs are sent to our Elastic Stack and over time this has become a serious pain.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 12:13pm UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/8 "2018-04-24T12:13:41Z")

</div>

If you are using Logstash, you may want to enable the [dead letter queue](https://www.elastic.co/guide/en/logstash/6.2/dead-letter-queues.html#dead-letter-queues) as I believe it would catch this scenario.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 22, 2018, 12:13pm UTC](https://discuss.elastic.co/t/logs-drop-on-field-conflict/129259/9 "2018-05-22T12:13:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
