Hi,
I have two inputs plugins in logstash.conf 1. beats & 2. file like below. I input JSON string for both.
Logstash config
input {
file{
path => "/logs/ServerLogs/testing-jsonoutput.log"
sincedb_path => "/appl/log/sincedbloc/sincedbfile.txt"
codec => json
}
beats {
port => 6000
tags => "beats"
codec => json
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout{}
file{
path => "/appl/log/TestLogsOutput/LogStash_output.log"
}
}
filebeat config
filebeat.prospectors:
-
paths:
- /read-write/mnt/addons/ClientLogs/client-jsonoutput.log*
document_type: log
json.keys_under_root: true
json.add_error_key: true
output.logstash:
hosts: ['111.111.111.111:6000']
I am inputing data to logstash simultaniouly and I do see logstash processed data in its output file like below.
String from filebeat after processing:{"logMessage":"Invoke null takes 7 ms","offset":220,"type":"log","input_type":"log","hostName":"VNWQ63213","source":"/read-write/mnt/addons/ClientLogs/client-jsonoutput.log","clientName":"LOT_CONTROL_SIM_2","loglevel":"DEBUG","beat":{"hostname":"VNWQ63213","version":"5.1.2","name":"VNWQ63213"},"@timestamp":"2017-05-11T16:40:35.545Z","loggingTime":"Thu 11 May 2017 12-40-33 474 EDT","applicationName":"LOT_CONTROL_SIM_2","@version":"1","host":"VNWQ63213","tags":["beats","beats_input_raw_event"]}
String from file input after processing: {"loggingTime":"Thu 11 May 2017 08-45-47 505 EDT","applicationName":"RestService","hostName":"HostName1","loglevel":"INFO","threadName":"WebContainer : 0","logMessage":{"hostName":"LOT_CONTROL_SIM_2","ipAddress":"111.111.111.111","port":40000,"routerPort":0,"terminalDescription":"LOT CONTROL SIM 2","locatedProcessPointId":"SIMPP01","divisionId":"SIMDIV01","afTerminalFlag":"Terminal","processPoint":{"processPointId":"SIMPP01","processPointName":"LOT_CONTROL_SIM_2","processPointDescription":"","processPointTypeId":5,"siteName":"SITESIM01","plantName":"SIMPLANT01","divisionId":"SIMDIV01","divisionName":"AF","lineId":"SIMLINE01","lineName":"AF ON","backFillProcessPointId":"","sequenceNumber":0,"trackingPointFlag":0,"recoveryPointFlag":0,"passingCountFlag":1,"createTimestamp":"2017-05-11 08:07:44.752 EDT","updateTimestamp":"2013-01-09 14:43:30.000 EST"},"createTimestamp":"2017-05-11 08:07:44.794 EDT"},"@version":"1","@timestamp":"2017-05-11T12:45:50.419Z","path":"/logs/GALCServerLogs/server-jsonoutput.log","host":"HMIPGRBLAPP502"}
Both the processed json strings are valid too
But when I try to search the logs through kibana. kibana pulls only logs processed through file input.
Looks I have proper data.. but why kibana is not pulling data which was through filebeat
Here is the sample json string data which came through filebeat
{"loggingTime":"Thu 11 May 2017 08-33-06 945 EDT","applicationName":"LOT_CONTROL_SIM_2","hostName":"VNWQ63213.hmin.am.honda.com","clientName":"LOT_CONTROL_SIM_2","loglevel":"DEBUG","logMessage":"Invoke null takes 6 ms"}
{"loggingTime":"Thu 11 May 2017 08-33-07 157 EDT","applicationName":"LOT_CONTROL_SIM_2","hostName":"VNWQ63213.hmin.am.honda.com","clientName":"LOT_CONTROL_SIM_2","loglevel":"DEBUG","logMessage":"Invoke null takes 8 ms"}