Logs drop on field conflict

Elastic Stack Elasticsearch
1

I am using ELK 5.5.0
When I send a log containing a field with conflicted type it doesn't show up in Kibana but I cannot find any error in the logs.

Example:

curl -H "content-type: application/json" -XPOST 'http://localhost:9191' -d '{
"message": "This is a log line",
"extrafield": 1
}'

curl -H "content-type: application/json" -XPOST 'http://localhost:9191' -d '{
"message": "This is a log line",
"extrafield": "no i am string"
}'

The first log sets the extrafield type to be a number.
The second log is simply ignored without any errors.

Am I missing a place where the errors can be written?

2

It seems like you are missing the index and type in your request. If I run the following:

curl -H "content-type: application/json" -XPOST 'http://localhost:9200/test/doc?pretty' -d '{
  "message": "This is a log line",
  "extrafield": 1
}'

curl -H "content-type: application/json" -XPOST 'http://localhost:9200/test/doc?pretty' -d '{
  "message": "This is a log line",
  "extrafield": "no i am string"
}'

I get an error in the response for the second request:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [extrafield]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse [extrafield]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "For input string: \"no i am string\""
    }
  },
  "status" : 400
}
3

Changed to:

curl -H "content-type: application/json" -XPOST 'http://localhost:9191/logs' -d '{
"message": "This is a log line",
"extrafield": 1
}'

curl -H "content-type: application/json" -XPOST 'http://localhost:9191/logs' -d '{
"message": "This is a log line",
"extrafield": "no i am string"
}'

Still same behaviour.

4

This should show up in Elasticsearch logs as an error.

To get around this you have to set index.mapping.ignore_malformed=true for extrafield

5

You are still not specifying the type, which is causing the requests to fail. Please try running the example I provided.

6

I am using logstash so it hid the message for me.
When i sent the request directly to elastic it showed the error

7

One more thing. If you can avoid having different types of data for the same key, do it. I don't have much control of what format logs are sent to our Elastic Stack and over time this has become a serious pain.

8

If you are using Logstash, you may want to enable the dead letter queue as I believe it would catch this scenario.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.