[ES 8.6.1 & 8.6.2] Fleet's "Custom Logs" integration stops sending logs with "failed to publish events: temporary bulk send failure" message


I'm new to Elastic, so it's possible that I configured something wrong...
Also, please be precise on where/how to find things if you ask for more info.

Base installation

I installed a fresh Elastic recently:

  • 8.6.1
  • one node Elastic/kibana, on the same server than my dev Tomcat
  • AWS EC2 (not k8s), Ubuntu
  • bound on eth0 IP (not using localhost)
  • Using only Fleet and Elastic-agent
  • 1 agent policy: Fleet server, APM, System, System Audit and 3 "Custom Logs" integrations

The custom logs have:

  • namespace = myproject
  • dataset name = myservice (I have tomcat, access-logs, and a script)
  • so I created 3 index template = logs-myservice-myproject* / datastream (overriding default pipeline and index lifecycle)

=> All this has worked, so the basics of installation should be OK.

Today I realized it had stopped sending logs (maybe after I changed something in Kibana, but not sure), while metrics and APM are still ok.

In the logs I found a lot of:

"message":"failed to publish events: temporary bulk send failure",

followed with these 2 "info" logs:

Connecting to backoff(elasticsearch(
Attempting to connect to Elasticsearch version 8.6.1

After looking at any configuration issues (especially in pipelines regex, etc...), I saw some related github issues, so I decided to upgrade everything to 8.6.2
=> problem was solved when the agent was effectively updated/restarted (through kibana), but after some time it's there again...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.