Metricbeat performance

[fdmsantos (Fábio Santos)]

Hello,

I'm developing a beat based on metricbeat.
I use the report.Event to send an event.
There is any way to improve the performance of this method?
Exists any way to send an array of Events to the buffer instead of one to one?
if I change queue.mem configs will change anything in this method?

Thanks,
Fábio Santos

[ruflin]

The publisher in Beats will automatically batch multiple events together to then send a bulk request to Elasticsearch. Anything specific you discovered as a performance issue? Perhaps you can link to your code?

If you increase the queue size, more events will be queue in memory before sending to Elasticsearch. It does not directly affect the method.

[fdmsantos (Fábio Santos)]

Hi @ruflin,

Thanks for the answer.

This is the link to the code: https://github.com/CCSGroupInternational/vspherebeat

The goal is retrieve all performance metrics from vsphere.

For now, we have 9 metric sets and we're testing with two vspheres.

The virtualmachines, hosts and virtualswitches metrics sets are the metric sets which take a long time. We ran the beat without the report.Event method and the times are:

Virtual machines - Vsphere1 - 3m54s
Virtual machines - Vsphere2 - 7m15s
Hots - Vsphere1 - 2m4s
Hots - Vsphere2 - 5m14s
VirtualSwitches - Vsphere1 - 3.3s
VirtualSwitches - Vsphere2 - 9.4s

With the report.Event method:

Virtual machines - Vsphere1 - 1h33m34
Virtual machines - Vsphere2 - More than 2 hours (After 2h15m is still running)
Hots - Vsphere1 - 58m29
Hots - Vsphere2 - 2h0m27
VirtualSwitches - Vsphere1 - 11m22
VirtualSwitches - Vsphere2 - 38m5

The Internal queue configs used in the tests was:

queue.mem:
  events: 32768
  flush.min_events: 16384
  flush.timeout: 30s

Any ideas, improvements in code?

Thanks.

Regards,
Fábio Santos

[fdmsantos (Fábio Santos)]

Adding more info:

Number of events sent to elasticsearch:

Virtual machines - Vsphere1 - 2942280
Virtual machines - Vsphere2 - 3215376
Hots - Vsphere1 - 1777368
Hots - Vsphere2 - 2818549
VirtualSwitches - Vsphere1 - 1205280
VirtualSwitches - Vsphere2 - 2330640

[fdmsantos (Fábio Santos)]

Hi @ruflin,

After the increase the workers in the config file, the performance became very good. It's all fine now.

Thanks for help

Regards,
Fábio Santos