Nginx useragent mapper_parsing_exception

[stevesimpson]

Hi,

I'm having a bit of trouble understanding why I'm getting the below errors when trying to Index Nginx access logs.

[2019-10-24T12:52:56,876][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-nginx-production-ilm-alias", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x4eb61fdd>], :response=>{"index"=>{"_index"=>"filebeat-nginx-production-2019.10.24-000001", "_type"=>"_doc", "_id"=>"uLbT_W0B8p-jxR3-wxKh", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [user_agent.original] of type [keyword] in document with id 'uLbT_W0B8p-jxR3-wxKh'. Preview of field's value: '{patch=3865, minor=0, os=Windows, major=77, build=, name=Chrome, os_name=Windows, device=Other}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:4686"}}}}}
[2019-10-24T12:52:56,877][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-nginx-production-ilm-alias", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x3d140226>], :response=>{"index"=>{"_index"=>"filebeat-nginx-production-2019.10.24-000001", "_type"=>"_doc", "_id"=>"ubbT_W0B8p-jxR3-wxKj", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [user_agent.original] of type [keyword] in document with id 'ubbT_W0B8p-jxR3-wxKj'. Preview of field's value: '{patch=3865, minor=0, os=Windows, major=77, build=, name=Chrome, os_name=Windows, device=Other}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:4263"}}}}}

We use the Filebeat Nginx module with the Logstash output. We add custom fields to our Nginx access logs so we use Logstash to parse out the correct fields before Indexing into Elasticsearch. Reading the Logstash logs I can see this is related to mapping of user_agent.original. I'm pretty sure that I've used the default mappings provided by Filebeat for this Index, only removing non essential mappings. I'll post the relevant sections of the Logstash filters and outputs below.

Logstash Filter

    grok {
      match => {
        "message" => 
          '%{IP:[source][ip]} %{NOTSPACE:[nginx][request_id]} - %{USERNAME:[user][name]} \[%{HTTPDATE:[nginx][access][time]}] "%{WORD:[http][request][method]} %{NOTSPACE:[url][original]} HTTP/%{NUMBER:[http][version]}" %{NUMBER:[http][response][status_code]} %{NUMBER:[http][response][body][bytes]} "%{DATA:[http][request][referrer]}\" "%{DATA:[nginx][access][agent]}" "%{DATA:[nginx][http_x_forwarded_for]}" "%{DATA:[nginx][http_cookie]}" "%{DATA:[nginx][http_x_ssl_protocol]}" rt=%{NUMBER:[nginx][request_time]} urt=%{NOTSPACE:[nginx][upstream_response_time]}' 
      }
    }
    date {
      match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z"]
      target => "@timestamp"
      timezone => "UTC"
      remove_field => "[nginx][access][time]"
    }
    useragent {
      source => "[nginx][access][agent]"
      target => "[user_agent][original]"
      remove_field => "[nginx][access][agent]"
    }

output

elasticsearch {
    user => redacted
    password => redacted
    ssl => true
    cacert => "path_to_pem"
    ssl_certificate_verification => true
    hosts => ["es-hosts"]
    ilm_enabled => true
    ilm_rollover_alias => "filebeat-nginx-production-ilm-alias"
    ilm_policy => "filebeat-nginx-production-ilm-policy"
    ilm_pattern => "{now/d}-000001"
    manage_template => false
}

Any advise would be appreciated on why this is failing and what steps I can take towards resolution.

Thanks!

[stevesimpson]

I should also point out we are running 7.3.0 for all stack components

[Badger]

@stevesimpson

failed to parse field [user_agent.original] of type [keyword]

That's telling you that elasticsearch is expecting a string in that field. However, the useragent filter creates an object with several subfields, and a field in elasticsearch cannot be an object on some documents and a string on others.

That said, the preview looks odd to me, but I haven't used elasticsearch in a long time, so maybe they changed the format of the error message somewhere after v5

If you are running Kibana then check the mapping there, otherwise use the mapping API to see what elasticsearch expects.