The issue pertaining to a few of our indices resolved itself when the index rolled over the following day, but we are still seeing this with data originating via filebeat.
Our data originates on a remote machine and is shipped using filebeat into logstash with a beats input. The filebeat prospector is pretty basic and follows the documentation online, aside from the path:
filebeat.prospectors:
- type: log
paths:
- /Library/Log/login.log
In our logstash dead letter queue we see the following:
[����MJ72017-11-27T15:54:06.683Z�dMETA�dbeathfilebeatjip_addressn192.168.43.207dtypeclog�dDATA�foffsetZ�bipn192.168.43.207jinput_typeclogfsourcex/Library/Logs/login.loggmessagex=LOGIN,USERNAME,Mon Nov 27 09:53:35 CST 2017,192.168.43.207dtypeclogdtags�hmacLoginxbeats_input_codec_plain_appliedx19-filter-computerLoginsj@timestampx2017-11-27T15:53:35.000Zkreceived_atx2017-11-27T15:53:43.095Zh@versiona1dbeat�dnamelC2017025-L04hhostnameilocalhostgversione5.6.3�dhostilocalhostieventTimexMon Nov 27 09:53:35 CST 2017factioneLOGINelasticsearch@388aa477a7790c221ae6013b8ba2d79913e2a3b659e75668880e12ccc8f7f1ee�Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"login-2017.11.27", :_type=>"log", :_routing=>nil}, #<LogStash::Event:0x2fbf0773>], response: {"index"=>{"_index"=>"login-2017.11.27", "_type"=>"log", "_id"=>"JLUv_l8BJvIoQpoCY3Zl", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [login-2017.11.27] as the final mapping would have more than 1 type: [log, type1]"}}}
If I look at our index template:
{
"login" : {
"order" : 0,
"index_patterns" : [
"login-*"
],
"settings" : {
"index" : {
"number_of_shards" : "1"
}
},
"mappings" : {
"type1" : {
"_source" : {
"enabled" : false
},
"properties" : {
"host_name" : {
"type" : "keyword"
},
"eventTime" : {
"type" : "date",
"format" : "EEE MMM dd HH:mm:ss z YYYY"
}
}
}
},
"aliases" : { }
}
}
Would I just need to replace type1 in the index template mappings with log for the event to properly store in the index? I am working toward removing types and working on tags within our configuration to be more future proof.