I'm trying to index json logs with a Fleet-managed agent using a custom log configuration. I successfully downloaded the agent to my computer and specified a path where the log files live.
However, when I generated a few log files, I noticed that the agent wouldn't detect log files that were json. I suspect I may need to edit the
agent.logging.ecs settings in the agent's yaml file, but I don't know how!
I've tried adding them via the 'Custom configurations' setting in the Edit Custom logs integration via Kibana/Fleet, but I think the changes are not getting applied how I imagined they would.
Here is my policy that I currently have (notice how the agent.logging.json is under the data_stream field instead of the agent field):
id: 151cb5e0-5f69-11eb-84d5-c3f10edecab1 revision: 2 outputs: default: type: elasticsearch hosts: - 'XXXX' agent: monitoring: enabled: false logs: false metrics: false inputs: - id: 4fd8dfb0-5f69-11eb-84d5-c3f10edecab1 name: python-log-integration revision: 1 type: logfile use_output: default meta: package: name: log version: 0.4.6 data_stream: namespace: testing streams: - id: logfile-log.log data_stream: dataset: tbd paths: - /Users/**/briqLogs/*.log agent.logging.json: true agent.logging.ecs: true fleet: kibana: protocol: https hosts: - XXXX
Here is an example of a json file that I want to index:
And here is the non-json file that made it into the data stream:
How do I index ECS formatted JSON files with a custom log integration?