Custom Metricbeat - Failed to publish events: temporary bulk send failure

Elastic Stack Beats
1

Hello,

I'm developing a beat based on metricbeat (https://github.com/CCSGroupInternational/vspherebeat)

The goal is to retrieve all performance metrics from Vcenter. We have four Vcenters , and we have one beat instance to each Vcenter (4) in the same machine running

[user@machine-name folder]$ ps -ef | grep vsp
fgrsant+ 16534     1 99 10:13 pts/3    00:04:00 ./vspherebeat
fgrsant+ 16560     1 99 10:13 pts/2    00:05:22 ./vspherebeat
fgrsant+ 16583     1 99 10:13 pts/0    00:05:11 ./vspherebeat
fgrsant+ 16605     1 99 10:13 pts/1    00:03:58 ./vspherebeat
fgrsant+ 16632 15449  0 10:16 pts/2    00:00:00 grep --color=auto vsp

We get a lot of Failed to publish events: temporary bulk send failure.

There is some of the log :

2019-06-19T09:43:15.279+0100	INFO	template/load.go:147	Elasticsearch template with name 'vspherebeat' loaded
2019-06-19T09:43:15.279+0100	INFO	[index-management]	idxmgmt/std.go:266	Loaded index template.
2019-06-19T09:43:15.279+0100	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(http://10.135.98.97:9200)) established
2019-06-19T09:43:15.829+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:15.926+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.110:9200))
2019-06-19T09:43:15.933+0100	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.1.1
2019-06-19T09:43:29.391+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:29.391+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.98:9200))
2019-06-19T09:43:29.397+0100	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.1.1
2019-06-19T09:43:29.432+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:29.432+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.97:9200))
2019-06-19T09:43:29.440+0100	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.1.1
2019-06-19T09:43:29.464+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:29.464+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.110:9200))
2019-06-19T09:43:29.481+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:29.481+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.110:9200))
2019-06-19T09:43:29.486+0100	INFO	template/load.go:84	Loading template for Elasticsearch version: 7.1.1
2019-06-19T09:43:29.486+0100	INFO	template/load.go:86	Existing template will be overwritten, as overwrite is enabled.
2019-06-19T09:43:29.494+0100	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.1.1
2019-06-19T09:43:29.495+0100	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.1.1
2019-06-19T09:43:29.498+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:29.498+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.97:9200))
2019-06-19T09:43:29.530+0100	INFO	elasticsearch/client.go:734	Attempting to connect to Elasticsearch version 7.1.1
2019-06-19T09:43:29.666+0100	INFO	template/load.go:147	Elasticsearch template with name 'vspherebeat' loaded
2019-06-19T09:43:29.666+0100	INFO	[index-management]	idxmgmt/std.go:266	Loaded index template.
2019-06-19T09:43:29.666+0100	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(http://10.135.98.98:9200)) established
2019-06-19T09:43:29.677+0100	INFO	template/load.go:84	Loading template for Elasticsearch version: 7.1.1
2019-06-19T09:43:29.677+0100	INFO	template/load.go:86	Existing template will be overwritten, as overwrite is enabled.
2019-06-19T09:43:29.939+0100	INFO	template/load.go:147	Elasticsearch template with name 'vspherebeat' loaded
2019-06-19T09:43:29.939+0100	INFO	[index-management]	idxmgmt/std.go:266	Loaded index template.
2019-06-19T09:43:29.939+0100	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(http://10.135.98.97:9200)) established
2019-06-19T09:43:29.950+0100	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-06-19T09:43:29.953+0100	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://10.135.98.110:9200))
2019-06-19T09:43:29.959+0100	INFO	template/load.go:84	Loading template for Elasticsearch version: 7.1.1
2019-06-19T09:43:29.959+0100	INFO	template/load.go:86	Existing template will be overwritten, as overwrite is enabled.

The four beat instances have the same configuration (Only differ in the name of index) :

vspherebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
 # reload.period: 86400s

setup.template.settings:
  index.number_of_shards: 50
  index.number_of_routing_shards: 50
  index.codec: best_compression
  #_source.enabled: false

setup.template.name: "vspherebeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: true
setup.template.enabled: true
setup.template.pattern: "vspherebeat-*"
setup.ilm.enabled: false

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["host1:9200", "host2:9200", "host3:9200"]

  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "user"
  password: "password"
  index: "vspherebeat-network-vcenter_name-%{+yyyy.MM.dd}"
  worker: 3

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: info
logging.selectors: ["*"]
logging.to_files: true
logging.files:
  path: ${path.config}/logs
  name: vspherebeat
  keepfiles: 10
  permissions: 0644

 queue.mem:
  events: 32768
  flush.min_events: 16384
  flush.timeout: 30s

Could someone help me despite this?

Thanks and Regards,
Fábio Santos

2

Hi @fdmsantos :slight_smile:

Forgive me for asking but why don't you use the supported vsphere module? https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-vsphere.html If you see that something is missing maybe you can open an enhancement request issue (because maybe more people is missing that feature) or open a PR yourself :wink:

Anyways, at first glance, I don't see anything particularly wrong but it seems an error with the Elasticsearch communication (network, auth...)

Please, can you try to setup console as output and see if you still have the same problem.

Can you also share the logs of Elasticsearch?

Thanks!

3

Hi, @Mario_Castro

Thanks for the answer.

I will follow your recommendations and I will back with the results (Probably only Monday).

About the supported vsphere module, this module retrieves very few information and the client wants all performance metrics. Cpu, Mem, Disk, rescpu, storage .... everything possible

Thanks and Regards,

Fábio Santos

4

Hi @Mario_Castro ,

Sorry for delay answer.

With the debugging, it seems network problems.

One Question, the metricbeat try send the events which failed again?

Many thanks.

Regards,
Fábio Santos

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.