Hi,
I'm having a bit of trouble understanding why I'm getting the below errors when trying to Index Nginx access logs.
[2019-10-24T12:52:56,876][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-nginx-production-ilm-alias", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x4eb61fdd>], :response=>{"index"=>{"_index"=>"filebeat-nginx-production-2019.10.24-000001", "_type"=>"_doc", "_id"=>"uLbT_W0B8p-jxR3-wxKh", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [user_agent.original] of type [keyword] in document with id 'uLbT_W0B8p-jxR3-wxKh'. Preview of field's value: '{patch=3865, minor=0, os=Windows, major=77, build=, name=Chrome, os_name=Windows, device=Other}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:4686"}}}}}
[2019-10-24T12:52:56,877][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-nginx-production-ilm-alias", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x3d140226>], :response=>{"index"=>{"_index"=>"filebeat-nginx-production-2019.10.24-000001", "_type"=>"_doc", "_id"=>"ubbT_W0B8p-jxR3-wxKj", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [user_agent.original] of type [keyword] in document with id 'ubbT_W0B8p-jxR3-wxKj'. Preview of field's value: '{patch=3865, minor=0, os=Windows, major=77, build=, name=Chrome, os_name=Windows, device=Other}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:4263"}}}}}
We use the Filebeat Nginx module with the Logstash output. We add custom fields to our Nginx access logs so we use Logstash to parse out the correct fields before Indexing into Elasticsearch. Reading the Logstash logs I can see this is related to mapping of user_agent.original
. I'm pretty sure that I've used the default mappings provided by Filebeat for this Index, only removing non essential mappings. I'll post the relevant sections of the Logstash filters and outputs below.
Logstash Filter
grok {
match => {
"message" =>
'%{IP:[source][ip]} %{NOTSPACE:[nginx][request_id]} - %{USERNAME:[user][name]} \[%{HTTPDATE:[nginx][access][time]}] "%{WORD:[http][request][method]} %{NOTSPACE:[url][original]} HTTP/%{NUMBER:[http][version]}" %{NUMBER:[http][response][status_code]} %{NUMBER:[http][response][body][bytes]} "%{DATA:[http][request][referrer]}\" "%{DATA:[nginx][access][agent]}" "%{DATA:[nginx][http_x_forwarded_for]}" "%{DATA:[nginx][http_cookie]}" "%{DATA:[nginx][http_x_ssl_protocol]}" rt=%{NUMBER:[nginx][request_time]} urt=%{NOTSPACE:[nginx][upstream_response_time]}'
}
}
date {
match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z"]
target => "@timestamp"
timezone => "UTC"
remove_field => "[nginx][access][time]"
}
useragent {
source => "[nginx][access][agent]"
target => "[user_agent][original]"
remove_field => "[nginx][access][agent]"
}
output
elasticsearch {
user => redacted
password => redacted
ssl => true
cacert => "path_to_pem"
ssl_certificate_verification => true
hosts => ["es-hosts"]
ilm_enabled => true
ilm_rollover_alias => "filebeat-nginx-production-ilm-alias"
ilm_policy => "filebeat-nginx-production-ilm-policy"
ilm_pattern => "{now/d}-000001"
manage_template => false
}
Any advise would be appreciated on why this is failing and what steps I can take towards resolution.
Thanks!