Is there any way that I can link Elasticsearch error with my input logline so I can fix it? I'm trying to fix Elasticsearch event errors from large log file(500MB), so testing line by line is impossible.
Here is Eleasticsearch error that I got from Logstash output plugin:
[2020-12-02T14:32:52,582][WARN ][logstash.outputs.elasticsearch][main][892049105f255c3edd9d26bf17571cd57f34b7021cc626620a7f35c34dbf6e1b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash_ver15-2020.12.02", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x68cf9f80], :response=>{"index"=>{"_index"=>"logstash_ver15-2020.12.02", "_type"=>"_doc", "_id"=>"zc7eI3YBes5tK59JyHzv", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [json.context.bindings] of type [text] in document with id 'zc7eI3YBes5tK59JyHzv'. Preview of field's value: '{a=1231}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:808"}}}}}`
Input log line that trigger that error:
{"message":"ddmsg_test11","context":{"bindings":[1,2,"test",{"a":1231}]},"level":200,"level_name":"INFO","channel":"local","datetime":{"date":"2020-12-02 09:32:48.266617","timezone_type":3,"timezone":"America/New_York"},"extra":[],"parid":"31156:0"}
Logstash rubydebug output:
{
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"input" => {
"type" => "log"
},
"log" => {
"offset" => 88542,
"file" => {
"path" => "C:\\xampp\\htdocs\\cb\\storage\\logs\\laravel-2020-12-02.log"
}
},
"agent" => {
"ephemeral_id" => "c7108138-d3cc-428d-b9d2-067d8eb35306",
"version" => "7.10.0",
"id" => "e3807ace-328b-4c1e-a512-a736ddb745be",
"name" => "DESKTOP-F0CFPRL",
"hostname" => "DESKTOP-F0CFPRL",
"type" => "filebeat"
},
"message" => "{\"message\":\"ddmsg_test11\",\"context\":{\"bindings\":[1,2,\"test\",{\"a\":1231}]},\"level\":200,\"level_name\":\"INFO\",\"channel\":\"local\",\"datetime\":{\"date\":\"2020-12-02 09:32:48.266617\",\"timezone_type\":3,\"timezone\":\"America/New_York\"},\"extra\":[],\"parid\":\"31156:0\"}",
"@timestamp" => 2020-12-02T14:32:51.350Z,
"json" => {
"channel" => "local",
"level" => 200,
"message" => "ddmsg_test11",
"context" => {
"bindings" => [
[0] 1,
[1] 2,
[2] "test",
[3] {
"a" => 1231
}
]
},
"parid" => "31156:0",
"level_name" => "INFO",
"datetime" => {
"timezone" => "America/New_York",
"date" => "2020-12-02 09:32:48.266617",
"timezone_type" => 3
},
"extra" => []
},
"@version" => "1",
"ecs" => {
"version" => "1.6.0"
}
}
