Hi Guys I am a total noob to ElasticStack and Filebeats.
I have a custom log file in a JSON format, the app we are using will output an 1 entry per file as follows
{"cuid":1,"Machine":"001","cuSize":0,"starttime":"2017-03-19T15:06:48.3402437+00:00","endtime":"2017-03-19T15:07:13.3402437+00:00","rejectcount":47,"fitcount":895,"unfitcount":58,"totalcount":1000,"processedcount":953}
I am trying to ingest this into ElasticSearch. I believe this is possible as I am using ES5.X
I have configured my FileBeat prospector, I have attempted to at least pull out 1 field from the file for now, namely the Cuid
filebeat.prospectors:
- input_type: log
json.keys_under_root : true
paths:- C:\Files\output*-Account-*
tags : ["json"]
- C:\Files\output*-Account-*
output.elasticsearch:
The Logstash hosts
hosts: ["10.1.0.4:9200"]
template.name: "filebeat"
template.path: "filebeat.template.json"
template.overwrite: true
processors:
- decode_json_fields:
fields: ["cuid"]
When I start the FileBeat , it seems to harvest the files, As I get an entry in the FileBeat Registry files
2017-03-20T13:21:08Z INFO Harvester started for file: C:\Files\output\001-Account-20032017105923.json
2017-03-20T13:21:27Z INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=160 publish.events=320 filebeat.harvester.started=160 registrar.states.update=320 registrar.writes=2
However, I can't seem to find the data within Kibana. I am not entirely sure how to find it?
I have ensured the FileBeat templates are loaded in kibana.
I have tried to read the documentation and I think I understand it correctly but I am still very hazy, as I am totally new to the stack.