I am trying to set up a custom index name for Metricbeat to use in ES (so I can have more than one server send to the same ES). When I start metricbeat with systemctl, it gets in a loop and constantly throws a "Failed to publish events: temporary bulk send failure" error and retries. Here is the relevant portion of /etc/metricbeat/metricbeat.yml:
I am running metricbeat 7.7.0 and attempting to connect to ES 7.2.0. I do have this running on c13u09 (with the default index name) and that's working.
Any help greatly appreciated!
It looks like an index error of some sort. A snippet from the log follows.
Thanks,
Rob
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.562-0400 INFO template/load.go:101 template with name 'c11u07-metricbeat' loaded.
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.562-0400 INFO [index-management] idxmgmt/std.go:295 Loaded index template.
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.562-0400 INFO [publisher_pipeline_output] pipeline/output.go:111 Connection to backoff(elasticsearch(http://c13u09.int.janelia.org:9200)) established
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:217 PublishEvents: 41 events have been published to elasticsearch in 8.591569ms.
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=1, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=2, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=3, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=4, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=5, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=6, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=7, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=8, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Jun 01 09:45:08 c11u07.int.janelia.org metricbeat[55247]: 2020-06-01T09:45:08.571-0400 DEBUG [elasticsearch] elasticsearch/client.go:390 Bulk item insert failed (i=9, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.