Metricbeat bulk item insert failed and elasticsearch has lots of adding template * for index patterns * logs

Elastic Stack Beats
1

metricbeat v7.0 logs:

2019-04-16T19:11:48.356+0800 DEBUG [elasticsearch] elasticsearch/client.go:338 PublishEvents: 51 events have been published to elasticsearch in 6.695631ms.
2019-04-16T19:11:48.356+0800 DEBUG [elasticsearch] elasticsearch/client.go:532 Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
2019-04-16T19:11:48.356+0800 DEBUG [elasticsearch] elasticsearch/client.go:532 Bulk item insert failed (i=1, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

elasticsearch logs:

{"type": "server", "timestamp": "2019-04-16T11:20:53,371+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "565c0c732374", "cluster.uuid": "KT9v6uRzRcyzSfcDeLdrhQ", "node.id": "EPCfP5ERRCSLQBE_HzutkA", "message": "adding template [metricbeat-devops] for index patterns [metricbeat-devops-]" }
{"type": "server", "timestamp": "2019-04-16T11:20:54,468+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "565c0c732374", "cluster.uuid": "KT9v6uRzRcyzSfcDeLdrhQ", "node.id": "EPCfP5ERRCSLQBE_HzutkA", "message": "adding template [metricbeat-devops] for index patterns [metricbeat-devops-]" }
{"type": "server", "timestamp": "2019-04-16T11:20:55,715+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "565c0c732374", "cluster.uuid": "KT9v6uRzRcyzSfcDeLdrhQ", "node.id": "EPCfP5ERRCSLQBE_HzutkA", "message": "adding template [metricbeat-devops] for index patterns [metricbeat-devops-]" }
{"type": "server", "timestamp": "2019-04-16T11:20:58,249+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "565c0c732374", "cluster.uuid": "KT9v6uRzRcyzSfcDeLdrhQ", "node.id": "EPCfP5ERRCSLQBE_HzutkA", "message": "adding template [metricbeat-devops] for index patterns [metricbeat-devops-]" }
{"type": "server", "timestamp": "2019-04-16T11:21:00,060+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "565c0c732374", "cluster.uuid": "KT9v6uRzRcyzSfcDeLdrhQ", "node.id": "EPCfP5ERRCSLQBE_HzutkA", "message": "adding template [metricbeat-devops] for index patterns [metricbeat-devops-]" }
{"type": "server", "timestamp": "2019-04-16T11:21:01,290+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "565c0c732374", "cluster.uuid": "KT9v6uRzRcyzSfcDeLdrhQ", "node.id": "EPCfP5ERRCSLQBE_HzutkA", "message": "adding template [metricbeat-devops] for index patterns [metricbeat-devops-]" }

metricbeat config:

metricbeat.config.modules:
  path: modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

name: metricbeat-128071-newbin 
queue.mem:
  events: 4096
  flush.min_events: 2048
  flush.timeout: 1s

output.elasticsearch:
  hosts: [ '172.27.128.71:9200']
  index: "metricbeat-devops-%{[beat.version]}-%{+yyyy.MM.dd}"
  worker: 2
  bulk_max_size: 150

processors:
- add_host_metadata: ~

setup.template:
  enabled: true
  overwrite: true
  name: metricbeat-devops
  pattern: metricbeat-devops-*
  settings:
    index.number_of_shards: 5
    index.number_of_replicas: 1
    index.refresh_interval: 60s
    index.merge.scheduler.max_thread_count: 1
    index.translog.durability: async
    index.translog.sync_interval: 30s
    index.translog.flush_threshold_size: 1gb

setup.kibana:
  host: "http://172.27.128.71:5601"

setup.dashboards:
  enabled: true
  index: metricbeat-devops-*

setup.ilm.enabled: false

logging:
  level: debug
  to_files: false

xpack.monitoring.enabled: true

Elasticsearch:

docker run --rm --name es -p 9200:9200 -p 9300:9300 -e discovery.type=single-node docker.elastic.co/elasticsearch/elasticsearch:7.0.0

Kibana:

docker run --rm --name kibana -p 5601:5601 -e ELASTICSEARCH_HOSTS=http://172.27.128.71:9200 docker.elastic.co/kibana/kibana:7.0.0

Elasticsearch doesn't have metricbeat-devops-* index and index pattern.

But, when I rename metricbeat-devops to metricbeat, metricbeat-devops-* to metricbeat-*, everything work well.

2

Can you try to replace %{[beat.version]} with %{[agent.version]} and try again?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.