I have installed Elastic Agent on multiple servers to collect logs and send them to Elasticsearch via Fleet Server. I want to filter logs at the source so that only error-level logs are collected and sent, instead of collecting all logs and filtering them later in Elasticsearch.
I am using Fleet Server to manage Elastic Agents and have added the Elasticsearch Logs integration. My questions are:
Can I configure the log filtering centrally from Fleet Server so that all agents collect only error level logs?
Where should I apply this filtering? Should it be done in the Fleet integration settings, or do I need to manually configure each Elastic Agent?
What is the correct way to implement this filtering? Would adding a drop_event processor in the integration’s advanced settings work?
That integration doesn't seem to have the Advanced section, so you can't add processors. I hadn't noticed that this integration is different.
You would have to change the policy logging level in the setting, advanced setting, agent logging level. Others have had problems where this change doesn't take effect, so on each agent, go to the bottom of the logging page, change the level and click apply changes. I think that should be a bug, but it hasn't gained traction to be one.
If you have 1000's of agents, an ingest policy drop might be easier. I think you would create ingest pipeline logs-elastic_agent.filebeat@custom
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.