Kibana not reading logstash output


(Fredrick Yessaian) #1

Hi,

I have two inputs plugins in logstash.conf 1. beats & 2. file like below. I input JSON string for both.

Logstash config

input {
file{
path => "/logs/ServerLogs/testing-jsonoutput.log"
sincedb_path => "/appl/log/sincedbloc/sincedbfile.txt"
codec => json
}

beats {
port => 6000
tags => "beats"
codec => json
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout{}
file{
path => "/appl/log/TestLogsOutput/LogStash_output.log"
}
}

filebeat config

filebeat.prospectors:

  • paths:

    • /read-write/mnt/addons/ClientLogs/client-jsonoutput.log*

    document_type: log
    json.keys_under_root: true
    json.add_error_key: true

output.logstash:
hosts: ['111.111.111.111:6000']

I am inputing data to logstash simultaniouly and I do see logstash processed data in its output file like below.

String from filebeat after processing:{"logMessage":"Invoke null takes 7 ms","offset":220,"type":"log","input_type":"log","hostName":"VNWQ63213","source":"/read-write/mnt/addons/ClientLogs/client-jsonoutput.log","clientName":"LOT_CONTROL_SIM_2","loglevel":"DEBUG","beat":{"hostname":"VNWQ63213","version":"5.1.2","name":"VNWQ63213"},"@timestamp":"2017-05-11T16:40:35.545Z","loggingTime":"Thu 11 May 2017 12-40-33 474 EDT","applicationName":"LOT_CONTROL_SIM_2","@version":"1","host":"VNWQ63213","tags":["beats","beats_input_raw_event"]}

String from file input after processing: {"loggingTime":"Thu 11 May 2017 08-45-47 505 EDT","applicationName":"RestService","hostName":"HostName1","loglevel":"INFO","threadName":"WebContainer : 0","logMessage":{"hostName":"LOT_CONTROL_SIM_2","ipAddress":"111.111.111.111","port":40000,"routerPort":0,"terminalDescription":"LOT CONTROL SIM 2","locatedProcessPointId":"SIMPP01","divisionId":"SIMDIV01","afTerminalFlag":"Terminal","processPoint":{"processPointId":"SIMPP01","processPointName":"LOT_CONTROL_SIM_2","processPointDescription":"","processPointTypeId":5,"siteName":"SITESIM01","plantName":"SIMPLANT01","divisionId":"SIMDIV01","divisionName":"AF","lineId":"SIMLINE01","lineName":"AF ON","backFillProcessPointId":"","sequenceNumber":0,"trackingPointFlag":0,"recoveryPointFlag":0,"passingCountFlag":1,"createTimestamp":"2017-05-11 08:07:44.752 EDT","updateTimestamp":"2013-01-09 14:43:30.000 EST"},"createTimestamp":"2017-05-11 08:07:44.794 EDT"},"@version":"1","@timestamp":"2017-05-11T12:45:50.419Z","path":"/logs/GALCServerLogs/server-jsonoutput.log","host":"HMIPGRBLAPP502"}

Both the processed json strings are valid too

But when I try to search the logs through kibana. kibana pulls only logs processed through file input.
Looks I have proper data.. but why kibana is not pulling data which was through filebeat

Here is the sample json string data which came through filebeat

{"loggingTime":"Thu 11 May 2017 08-33-06 945 EDT","applicationName":"LOT_CONTROL_SIM_2","hostName":"VNWQ63213.hmin.am.honda.com","clientName":"LOT_CONTROL_SIM_2","loglevel":"DEBUG","logMessage":"Invoke null takes 6 ms"}
{"loggingTime":"Thu 11 May 2017 08-33-07 157 EDT","applicationName":"LOT_CONTROL_SIM_2","hostName":"VNWQ63213.hmin.am.honda.com","clientName":"LOT_CONTROL_SIM_2","loglevel":"DEBUG","logMessage":"Invoke null takes 8 ms"}


(Fredrick Yessaian) #2

Please ignore this post.. I figured out the issue.. its with the data type that I have in the JSON String.. I will create ticket under Kibana

Thanks


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.