Hello everyone, I´m having trouble trying to parse simple json data from logstash to elasticsearch . My config is :
filebeat suscribed to some mosquitto topic > logstash mqqt input > elasticsearch > kibana.
@timestamp , mqtt topic and the message json payload structured in fields are my goals .
While I do get information I don´t see it well structured , i played with filebeat json processor , logstash input (json_lines) , logstash json filter , but got the best result with json codec .
I'll post my config ,hoping someone can tell me what I'm missing . If you haven't noticed I´m really a newbie at ELK .
logstash config :
input {
beats {
port => "5044"
codec => "json"
}
}
output {
stdout { codec => rubydebug { metadata => true } }
elasticsearch {
hosts => ["http://localhost:9200"]
index => "block"
}
}
This is the output I´m getting , I see it organized , however ,back in kibana I don´t see the resulting fields . Do you guys validate this output and maybe I´m missing something in Elasticsearch???
{
"input" => {
"type" => "mqtt"
},
"agent" => {
"hostname" => "DESKTOP-564O36T",
"name" => "DESKTOP-564O36T",
"id" => "72b87c44-b538-4a3d-b195-f3f35824acf2",
"type" => "filebeat",
"ephemeral_id" => "5a1658fa-4bd4-4657-ba9d-4e2fba7357dd",
"version" => "7.9.2"
},
"@timestamp" => 2021-02-23T17:22:16.663Z,
"ecs" => {
"version" => "1.5.0"
},
"@metadata" => {
"ip_address" => "0:0:0:0:0:0:0:1",
"beat" => "filebeat",
"type" => "_doc",
"version" => "7.9.2"
},
"mqtt" => {
"retained" => false,
"qos" => 0,
"topic" => "Block/SalaN/Equipamiento",
"message_id" => 0,
"duplicate" => false
},
"@version" => "1",
"host" => {
"name" => "DESKTOP-564O36T"
},
"tags" => [
[0] "beats_input_codec_json_applied"
],
"Equipamiento" => {
"Corriente" => 0.015769,
"Id" => "12010-13402",
"Unit" => "Amp"
}
}
Thanks in any way you can help me .
How are you verifying the data in Kibana?
In Dev Tools run GET block/_search
to see if you have any data.
Also did you create an index pattern? You can only see the data in Visualizations and Discover after you have done that.