Heartbeat when reload.enabled: true

Elastic Stack Beats
1

Dear developers,

Elasticsearch 7.6.2

I have issue in the heartbeat when reload.enabled change to true.
The condition as follow:

  1. I have around 120 monitor yml file
  2. I created around 40 new monitor yml file
  3. Before the reload.enabled: true, the pipeline clients indicate the correct number
  4. Then, I thought if I make reload.enabled: true, I can add new monitor yml automatically read by heartbeat
  5. just copy the yml file to monitors.d folder
  6. The heartbeat log said there are new configuration as expected
  7. For several minutes, the pipeline client goes to more than 2000 and all metric of heartbeat going crazy
  8. I removed the new yml but the pipeline still high and higher
  9. restart the heartbeat and the pipeline client back to normal

When I revert to false, there is no issue anymore regarding the increasing the pipeline clients.

Regards,
Fadjar Tandabawana

2

Can you share your config? Also, can you clarify what you mean by more than 2000. Which metric is increasing?

Also, are you setting monitor.id for your monitors? Heartbeat uses that field to keep consistent track of monitors.

3

The config is very simple following the file example in the monitors.d folder, but for each target, I created seperate file.
The metric is the pipeline client, I saw it from the log, beside the pipeline text there something like:

pipepile: {clients: xxxx ....

So, the calculation going multiply related to the repetition of the pipeline clients in the same configuration.
BTW, that resolved by restarting the metricbeat, I just wondering the behavior is different from other beats.

Is the monitor.id can use normal text? or it should be number?

Regards,
Fadjar Tandabawana

4

monitor.id is any string you'd like it, something that's easy to understand by a human like my-service is generally preferable to a number.

Can you share your logs indicating the issue with the Elasticsearch client? I'm not clear on whether you're experience an issue with heartbeat communicating with ES, or with the data itself.

5

Dear @Andrew_Cholakian1,

After I put the ID, the event not occurred anymore..

Than you for support...

Regards,
Fadjar Tandabawana

6

Glad it worked out!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.