here is logstash log:
[2017-12-22T10:57:58,015][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"FcAbfGABywB1gOlxgvRg", :_index=>"app_101283-2017.12.22", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x5084bdcf>], :response=>{"index"=>{"_index"=>"app_101283-2017.12.22", "_type"=>"doc", "_id"=>"FcAbfGABywB1gOlxgvRg", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"22/Dec/2017:10:43:50 +0800\" is malformed at \"/Dec/2017:10:43:50 +0800\""}}}}}
[2017-12-22T10:57:58,015][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"CzAbfGABM12IwqqD6_bs", :_index=>"app_101283-2017.12.22", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0xd5ac17c>], :response=>{"index"=>{"_index"=>"app_101283-2017.12.22", "_type"=>"doc", "_id"=>"CzAbfGABM12IwqqD6_bs", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"22/Dec/2017:10:44:24 +0800\" is malformed at \"/Dec/2017:10:44:24 +0800\""}}}}}
[2017-12-22T10:57:58,015][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"ocAbfGABywB1gOlx8Pa_", :_index=>"app_101283-2017.12.22", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x2509f0d1>], :response=>{"index"=>{"_index"=>"app_101283-2017.12.22", "_type"=>"doc", "_id"=>"ocAbfGABywB1gOlx8Pa_", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"22/Dec/2017:10:44:44 +0800\" is malformed at \"/Dec/2017:10:44:44 +0800\""}}}}}
[2017-12-22T10:57:58,015][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"n8AbfGABywB1gOlx8Pa_", :_index=>"app_101283-2017.12.22", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x37e00906>], :response=>{"index"=>{"_index"=>"app_101283-2017.12.22", "_type"=>"doc", "_id"=>"n8AbfGABywB1gOlx8Pa_", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"22/Dec/2017:10:44:26 +0800\" is malformed at \"/Dec/2017:10:44:26 +0800\""}}}}}
[2017-12-22T10:57:58,087][INFO ][logstash.pipeline ] Pipeline terminated {"pipeline.id"=>"main"}
and config:
input{
elasticsearch {
hosts => "10.50.5.52:9200"
# hosts => ["localhost:9200"]
index => "app_101283-*"
query => '{ "query": { "match_all": {} } }'
size => 10000
scroll => "5m"
docinfo => true
codec => json
}
}
filter {
if "IAM-" in [fields][service] {
if "access" in [source]{
grok {
patterns_dir => ["./patterns"]
match => {
"message" => "%{IAM_ACCESS_LOG}"
}
}
mutate {
add_field => {"isAccss" => true}
}
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
}
}
else {
grok {
patterns_dir => ["./patterns"]
match => {
"message" => "%{IAMLOG}"
}
}
date {
match => ["timestamp", "MMM dd HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
}
}
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
document_id => "%{[@metadata][_id]}"
index => "%{[@metadata][_index]}"
user => "elastic"
password => "changeme"
}
}
, in my index, there are 2 kind of log, one is spring boot log and another is spring boot tomcat embedded access log, they locate in the same machine, I collect them together. since they have different timestamp format, I use two date plugin for them. the key point here is that, I grok them with the same field name "timestamp", so, this two date plugin thread seems mix up each other's data. when I change the field name to "iso_timestamp" and another to "httpd_timestamp", this issue are not appeared any more. (it is permanent issue before )