Beats_input_raw_event

[nabha (Nabha Shirkul)]

I am trying to fetch a .json file through filebeat to send logs to logstash. But it throws an error as
"
"tags" => [
[0] "beats_input_raw_event"
]"

My filebeat.yml has following configuration:
filebeat.prospectors:

• type: log
  enabled: true

  paths:

  • ./test_sample.json
    json.keys_under_root: true
    json.add_error_key: true

My json file has following data
[
{"name": "Rhonda Levine","department": "Development","email": "rhondalevine@anixang.com"},
{"name": "Mcgee Klein","department": "Accounts","email": "mcgeeklein@digique.com" },
{"name": "Dixon Poole","department": "HR","email": "dixonpoole@microluxe.com"},
{"name": "Blackwell Hill","department": "Development","email": "blackwellhill@affluex.com"},
{"name": "Leonor Mack","department": "Accounts","email": "leonormack@isosphere.com"},
{"name": "Corrine Howe","department": "Development","email": "corrinehowe@terrasys.com"},
{"name": "Sondra Flores","department": "HR","email": "sondraflores@jetsilk.com"},
{"name": "Farley Shaw","department": "Development","email": "farleyshaw@comvey.com"},
{"name": "Mcpherson Patterson","department": "HR","email": "mcphersonpatterson@escenta.com"},
{"name": "Preston Lucas","department": "Development","email": "prestonlucas@enormo.com"},
{"name": "Hester Butler","department": "Development","email": "hesterbutler@zillanet.com"},
{"name": "Fay Burris","department": "Accounts","email": "fayburris@suremax.com"},
{"name": "Deena Benton","department": "Development","email": "deenabenton@intradisk.com"},
{"name": "Mclean Carrillo","department": "MEDESIGN","email": "mcleancarrillo@medesign.com"},
{"name": "Daniels Good","department": "KONNECT","email": "danielsgood@konnect.com"},
{"name": "Marietta Evans","department": "Development","email": "mariettaevans@bezal.com"},
{"name": "Angeline Cantu","department": "Development","email": "angelinecantu@sportan.com"},
{"name": "Macdonald Atkins","department": "Development","email": "macdonaldatkins@orbin.com"},
{"name": "Mooney Bennett","department": "Accounts","email": "mooneybennett@interfind.com"}
]

Please help!

[ruflin]

What is your expected outcome? The above looks like a json array with a json doc on each line. Do you expect all to end up as one event in Elasticsearch / Logstash?

[nabha (Nabha Shirkul)]

Hey Ruflin thanks for your reply. Some changes in filebeat.yml file worked fine for me. Results are attached in attachment which is as expected. Now I want to render a graph from Visualize section to show Department wise data but I am not able to find Development field in field list. Please help for the same.

The changes in filebeat.yml are as follows :

filebeat.prospectors:
paths:
- ./test_sample.json
json.keys_under_root: true
json.add_error_key: true

[nabha (Nabha Shirkul)]

Actually I just want to pass data to elastic search from filebeat from JSON file. In above example I would like to treat each object as a record in log file. Can you please suggest a json file format, which can be used for the purpose mentioned above? Thanks

[ruflin]

I thin the problem in your case is that Filebeat sends up all as one event (which is expected) and then you have nested objects in Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

If you want each line to be an event, you probably have to go trough logstash for more complex processing or alternative if you are in control of the logger, log one json even per line.

[nabha (Nabha Shirkul)]

I am not exactly getting it. Please find attachment,

Discover section shows each json object as a record from file and the graph shows count which is equal to no. of objects in my .json file. So, I think the problem has been solved. If I am wrong can you please suggest json file structure as I am in control of the logger and also I am stuck while showing Department wise graphs in Visualize section, as Department is not getting displayed in field list.
Please help

[ruflin]

All good if you got the expect result