Issue with Elasticsearch mapping

[arisbanach]

I've been getting these errors nonstop all day and haven't been able to figure out a solution. From investigating the issue, it looks like our data input is giving us a JSON object that I want to parse with the JSON filter, but the issue (I think) is that one of the object's properties is another object, which has a property that is sometimes set to null. The parent object is the field [extras] and then that contains a child field called [visitors] and that contains a child object called [user_data].

[2017-12-21T22:31:48,253][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metrics-buffalonewscom-2017-09", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x6d9f48f6>], :response=>{"index"=>{"_index"=>"metrics-buffalonewscom-2017-09", "_type"=>"doc", "_id"=>"_cw0e2ABluZYKCJJOIpQ", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [visitor.user_data] tried to parse field [user_data] as object, but found a concrete value"}}}}

My pipeline first uses the JSON filter on [extras] and then on [visitors][user_data], but it shows this warning when it runs into issues with empty or null values for it. I've tried removing the field if [visitor][user_data] == "null" but that evidently isn't working.

Is there a simpler solution? I would love it if the JSON filter just had an option that handled this kind of issue (i.e. if Elasticsearch won't be able to index it correctly then it just removes the field).

What is the best way to handle this?

Thank you.

[guyboertje]

Before you actually send the data to elasticsearch try to experiment with a simple config.
e.g.

input {
  generator {
    message => '{"from_ip": "10.2.3.40", "app": "bazfoo", "amount": 22.95,
         "extras":{"visitors":{"user_data":null}}}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }

  if ! [extras][visitors][user_data] {
    mutate {
      add_tag => ["user_data-empty"]
    }
  }
}

output {
  stdout {
    codec => rubydebug {metadata => true}
  }
}

The if conditional is a kind of not exists test when done like above.

{
           "app" => "bazfoo",
      "sequence" => 0,
        "amount" => 22.95,
    "@timestamp" => 2017-12-22T10:26:37.214Z,
          "host" => "Elastics-MacBook-Pro.local",
      "@version" => "1",
        "extras" => {
        "visitors" => {
            "user_data" => nil
        }
    },
       "message" => "{\"from_ip\": \"10.2.3.40\", \"app\": \"bazfoo\", \"amount\": 22.95, \"extras\":{\"visitors\":{\"user_data\":null}}}",
          "tags" => [
        [0] "user_data-empty"
    ],
       "from_ip" => "10.2.3.40"
}