Failed to publish events: temporary bulk send failure for custom ES index

[robsv (Rob Svirskas)]

I am trying to set up a custom index name for Metricbeat to use in ES (so I can have more than one server send to the same ES). When I start metricbeat with systemctl, it gets in a loop and constantly throws a "Failed to publish events: temporary bulk send failure" error and retries. Here is the relevant portion of /etc/metricbeat/metricbeat.yml:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["c13u09:9200"]
index: "c11u07-metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

setup.template.name: "c11u07-metricbeat"
setup.template.fields: "fields.yml"
setup.template.pattern: "c11u07-metricbeat-*"
setup.template.enabled: true
setup.template.overwrite: true
setup.ilm.enabled: false

I am running metricbeat 7.7.0 and attempting to connect to ES 7.2.0. I do have this running on c13u09 (with the default index name) and that's working.
Any help greatly appreciated!

[Alex_Kristiansen]

@robsv,

Can you set logging.level to debug in the config and run this again? Can you send us a sample of those debug logs?

[robsv (Rob Svirskas)]

It looks like an index error of some sort. A snippet from the log follows.
Thanks,
Rob

Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.562-0400        INFO        template/load.go:101        template with name 'c11u07-metricbeat' loaded.
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.562-0400        INFO        [index-management]        idxmgmt/std.go:295        Loaded index template.
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.562-0400        INFO        [publisher_pipeline_output]        pipeline/output.go:111        Connection to backoff(elasticsearch(http://c13u09.int.janelia.org:9200)) established
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:217        PublishEvents: 41 events have been published to elasticsearch in 8.591569ms.
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=1, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=2, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=3, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=4, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=5, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=6, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=7, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=8, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400        DEBUG        [elasticsearch]        elasticsearch/client.go:390        Bulk item insert failed (i=9, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

[robsv (Rob Svirskas)]

@robsv

index: "c11u07-metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

Okay, it seems the problem is "beat.version" in
index: "c11u07-metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

I changed it to:
index: "c11u07-metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
and it's working.