Hello,
I'm new to Elastic, so it's possible that I configured something wrong...
Also, please be precise on where/how to find things if you ask for more info.
Base installation
I installed a fresh Elastic recently:
- 8.6.1
- one node Elastic/kibana, on the same server than my dev Tomcat
- AWS EC2 (not k8s), Ubuntu
- bound on eth0 IP (not using localhost)
- Using only Fleet and Elastic-agent
- 1 agent policy: Fleet server, APM, System, System Audit and 3 "Custom Logs" integrations
The custom logs have:
- namespace = myproject
- dataset name = myservice (I have tomcat, access-logs, and a script)
- so I created 3 index template = logs-myservice-myproject* / datastream (overriding default pipeline and index lifecycle)
=> All this has worked, so the basics of installation should be OK.
Today I realized it had stopped sending logs (maybe after I changed something in Kibana, but not sure), while metrics and APM are still ok.
In the logs I found a lot of:
{"log.level":"error",
"message":"failed to publish events: temporary bulk send failure",
"component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},
"log":{"source":"log-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":176,"file.name":"pipeline/client_worker.go"},
"service.name":"filebeat",
"ecs.version":"1.6.0","ecs.version":"1.6.0"}
followed with these 2 "info" logs:
Connecting to backoff(elasticsearch(https://10.64.68.80:9200))
Attempting to connect to Elasticsearch version 8.6.1
After looking at any configuration issues (especially in pipelines regex, etc...), I saw some related github issues, so I decided to upgrade everything to 8.6.2
=> problem was solved when the agent was effectively updated/restarted (through kibana), but after some time it's there again...