Hi Team,
I want to include @metadata field contents in my elasticsearch output.
This is the output when i am using stdout in my output filter-
{
"@timestamp" => 2018-03-08T08:17:42.059Z,
"thread_name" => "SimpleAsyncTaskExecutor-2",
"@metadata" => {
"dead_letter_queue" => {
"entry_time" => 2018-03-08T08:17:50.082Z,
"reason" => "Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"applog-2018.03.08", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x3ab79ab5], response: {"index"=>{"_index"=>"applog-2018.03.08", "_type"=>"doc", "_id"=>"POuwBGIB0PJDPQOoDy1Q", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [message]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:223"}}}}",
"plugin_type" => "elasticsearch",
"plugin_id" => "7ee60ceccc2ef7c933cf5aa718d42f24a65b489e12a1e1c7b67ce82e04ef0d37"
}
},
"@version" => "1",
"beat" => {
"name" => "filebeat-kwjn6",
"version" => "6.0.0"
},
"dateOffset" => 408697,
"source" => "/var/log/applogs/spring-cloud-dataflow/Log.log",
"logger_name" => "decurtis.dxp.deamon.JobConfiguration",
"message" => {
"timeStamp" => "2018-01-30",
"severity" => "ERROR",
"hostname" => "",
"commonUtility" => {},
"offset" => "Etc/UTC",
"messageCode" => "L_9001",
"correlationId" => "ea5b13c3-d395-4fa5-8124-19902e400316",
"componentName" => "dxp-deamon-refdata-country",
"componentVersion" => "1",
"message" => "Unhandled exceptions",
},
"tags" => [
[0] "webapp-log",
[1] "beats_input_codec_plain_applied",
[2] "_jsonparsefailure"
]
}
I want my @metadata field in elasticsearch output.
Below is my conf file:
input {
dead_letter_queue {
path => "/usr/share/logstash/data/dead_letter_queue"
commit_offsets => true
pipeline_id => "main"
}
}
filter {
json {
source => "message"
}
mutate {
rename => { "[@metadata][dead_letter_queue][reason]" => "reason" }
}
}
output {
elasticsearch {
hosts => "elasticsearch"
manage_template => false
index => "deadletterlog-%{+YYYY.MM.dd}"
}
}
Now in my output there is a field called "reason" but without any content. Is there something i am missing.
I got idea to rename that field from Sending @metadata from logstash to elastic search . @magnusbaeck Can you please help me with this? Is there something i am missing.