I have a Nodejs app that ingests logs into Elasticsearch Service deployment 7.15.1 using Filebeat. More of less this is the example I've been doing adapted to my needs Ingest logs from a Node.js web application using Filebeat | Elasticsearch Service Documentation | Elastic and everything works fine.
Now I am trying to use Fleet and Elastic agent to send my custom json logs but here's the error I got:
[elastic_agent.filebeat][error] Harvester could not be started on new file: /Users/panagiotamitsopoulou/Documents/HYF/nodejs_ES_logs/logs/log.json, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /Users/panagiotamitsopoulou/Documents/HYF/nodejs_ES_logs/logs/log.json: open /Users/panagiotamitsopoulou/Documents/HYF/nodejs_ES_logs/logs/log.json: operation not permitted.
I am on Mac and here are the steps I've taken.
- On Kibana > Fleet I verified that my agent with Elastic Cloud agent policy is Healthy
- I enabled Collect agent logs under Elastic Cloud agent policy
- I created a new Fleet-managed agent and installed the agent to my machine with the
sudo ./elastic-agent install -f ....I copied from the UI.
- I did
chmod -R 777 logs
- I keep getting the permission error I wrote above
How can I solve above issue or better how can I troubleshoot if I am doing something wrong or if it is actually a bug?
A couple more questions since I would like to demonstrate above example to an audience if I manage to make it work and I would like to know how it actually works:
Collect agent logsunder Elastic Cloud agent policy is a prerequisite mandatory step I need to do to be able to send custom logs?
- Do I need to add anything else in my custom configuration?
- I found a similar post Adding config for custom log integration on an agent, where these two configuration options are mentioned
agent.logging.ecs. Do I need to update them? If yes where is the installed path of my agent to do so?
If needed I could share the github link of my code.