ERROR Failed to publish events: temporary bulk send failure

Elastic Observability APM
, ,
1

I create the my apm server in ECS on AWS.
This APM Server is with 256 CPU and 1024 MiB per task with 5 tasks.

And eventualy I receive this errors:

2020-05-27T01:38:37.929Z ERROR elasticsearch/client.go:333 Failed to perform any bulk index operations: Post http://elastic.local:9200/_bulk: EOF

2020-05-27T01:38:39.183Z ERROR pipeline/output.go:121 Failed to publish events: Post http://elastic.local:9200/_bulk: EOF

2020-05-27T01:38:41.558Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure

or only this log:

ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure

Kibana version: 6.8.1

Elasticsearch version: 6.8.1

APM Server version: 6.8.1 (6 tasks in AWS ECS with loadbalancer)

APM Agent language and version:
nodejs -> 3.5.0
nodejs -> 3.6.1
elasticapm-php -> 7.0.0-rc2

Original install method (e.g. download page, yum, deb, from source, etc.) and version: The install with:

curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-6.8.1-x86_64.rpm && \
    rpm -vi apm-server-6.8.1-x86_64.rpm

Is there anything special in your setup?

In my file apm-server.yml has the customer parameters:
BULK_MAX_SIZE -> 5120
WORKERS -> 35
queue_mem_events -> workers * bulk_max_size (179200)

observation elk config:
My Elasticsearch is 9 nodes with 3 Master Ingestion and 6 Data Nodes.

2

Welcome @afonsoavr! I don't see anything obvious given the information you've provided. Is there anything logged by elasticsearch at this time? Have you tried enabling debug logging in apm-server?

Also, have any events been ingested? Does the same occur when the PHP agent is not in use?

3

Hi @gil,
Thank's for your answer.

My config for APM Server is "logging.level: warning" and "logging.to_stderr: true".
And trying differents configurations for combination BULK_MAX_SIZE and WORKERS.

And, I increase the configuration the 256 CPU for 512 CPU because in 256 CPU the logging increase errors in other moments.

About the configurations CPU, 256 ou 512 is a 0.25 Core and 0.5 Core.

In my last log is: ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure

In my case, dont has option the not run applications with PHP.
This error is normally associated with PHP agent?

4

I had asked about the PHP agent because the one you're using is not officially supported and wondered if it may submit data that is accepted by APM Server but not by Elasticsearch - that shouldn't happen but a lot work has been done to improve intake validation since 6.8.1. I would expect this information would be exposed by the Elasticsearch logs, are you able to access those?

You may try logging.level: debug if the Elasticsearch logs don't provide enough insight.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.