Upgrade to logstash 5, got mapper_parsing_error

Elastic Stack Logstash
1

Hi, I upgraded logstash from v2.2 to 5.4 recently, got mapper_parsing_error, same grok pattern works on previous version, and verified on "http://grokconstructor.appspot.com/" as well, could someone help me out on this issue? Many thanks!

log example:
2017-06-03 09:46:28 [http-/0.0.0.0:8080-4] DEBUG [endpoint=GET /transactions/credit, sourceIp=10.195.182.201, requestId=4c3e702e-d01c-46d6-ba95-7be58c2082c1, customerId=2999, accountNumber=6510, branch=3750, deviceId=g6/wFMM6VCm+/LYXbNrtHEUVHsMwM8HokTbYrm32krMlbQY/FMUO5n1WA4aFbzq7] b.c.b.n.p.c.dao.LoggableJdbcTemplate - Executing prepared SQL query

grok pattern:
(?<syslog_time>20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}))%{SPACE}[%{DATA:thread}]%{SPACE}%{LOGLEVEL:level}%{SPACE}[%{GREEDYDATA:kvpairs}]%{SPACE}%{JAVACLASS:class}%{SPACE}-%{SPACE}%{JAVALOGMESSAGE:java_message}

parsing error in logstash log file:
[2017-06-13T14:00:38,032][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2017.06.13", :_type=>"syslog", :_routing=>nil}, 2017-06-03 09:46:28 [http-/0.0.0.0:8080-4] DEBUG [endpoint=GET /transactions/credit, sourceIp=10.195.182.201, requestId=4c3e702e-d01c-46d6-ba95-7be58c2082c1, customerId=2999, accountNumber=6510, branch=3750, deviceId=g6/wFMM6VCm+/LYXbNrtHEUVHsMwM8HokTbYrm32krMlbQY/FMUO5n1WA4aFbzq7] b.c.b.n.p.c.dao.LoggableJdbcTemplate - Executing prepared SQL query], :response=>{"index"=>{"_index"=>"filebeat-2017.06.13", "_type"=>"syslog", "_id"=>"AVyhwZeNs5aALyvA1py5", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [syslog_time]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"error parsing field [syslog_time], expected an object but got syslog_time"}}}}}

2

It looks like your syslog_time field has been mapped as an object in your index but in the events you're trying to index it's no longer an object. Temporarily replace the elasticsearch output with a stdout { codec => rubydebug } output so that you can see exactly what the events look like. Compare those results to the mapping of the index. Use ES's get mapping API to obtain it.

3

I cleared old mapping, it works now. thanks.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.