System.auth.user grok results in string not object after index readonly event

I had an event where the indices were set to read-only due to diskspace. After increasing space and setting them for read/write, the grok for syslog-auth appears to be parsing the system.auth.user wrong or somehow the index was created with an object type instead of string. I validated that the grok was exactly the same as specified in and I can see that the system.auth.user is a single string and in the entries from the day before. I see multiple errors such as:

[2018-08-23T17:41:55,158][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-6.3.2-2018.08.23", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x4f6ae416], :response=>{"index"=>{"_index"=>"filebeat-6.3.2-2018.08.23", "_type"=>"doc", "_id"=>"-1q8aGUB2ciOGo1Tme43", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [system.auth.user] tried to parse field [user] as object, but found a concrete value"}}}} shows this result using sample data
{ ...
"[system][auth][user]": [
... }

stdout { codec => rubydebug } shows this for same sample data:
"system" => {
"auth" => {
"user" => "git",
... }

I'm suspecting that somehow my index got corrupted and it might be corrected with the next day's index creation, but I don't know how that would have happened.

Although this did correct itself, it was not when the index rolled over as I expected. I was tweaking a different output filter due to duplicate tags on items from filebeats and direct to logstash that shouldn't have been sent to the same index. This was due to a transitional period between the two methods. Once I had the filters the way I wanted them, they were added to config management (puppet) and pushed to the logstash server. This did result in a couple restarts of logstash, and it was resolved after the final restart. Although logstash had been restarted multiple times before this while working on the filters, it was only after the final restart that it resolved. This can be closed automatically, but I wanted the discussion opened early.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.