[heartbeat]add feature to enable publisher processor or other processors' debug log for just specific targets

Hey Elastic friends,

Obviously, there’s some basic and amazing debug features in Elastic heartbeat, however we met a user case these days that current debug features cannot handle it.
Given this example, since we have over 30k targets need to be monitored, in some case, we just want to observe the debug log of publisher processor for just some specific targets. That means we want to make sure the check for this target is really executed and the metrics have been sent out.

Compared to the current features to do that:

  1. starting heartbeat with -d "*", this prints all debug logs, that’s noisy and hard to find the log of our desired target, also uses too many disk. By the way, this still need a restart of heartbeat, this could be tough if it’s in use for production.

  2. starting heartbeat with -d "publisher" , this also prints too much useless logs and needs a restart of heartbeat.

Wondering if there’s an option that can be setup in target file scope, like the following:

- type: http
  schedule: 36 * * * * * *
  hosts:
   - http://127.0.0.1/readiness_api
  check.request:
    method: GET
  check.response:
    status:
    - 200
    - 201
  fields:
    metadata:
      debugOn: true

Heartbeat will record the publisher or all processors’ debug log for this target with debugOn: true option.
This needn’t restart heartbeat and can be specified to the desired targets as you wish.

Apologies for the delay @lowry , we somehow missed this.

Which debug data are you looking for? The debugf calls in the go code really can only go to the CLI, but if you let us know exactly what data you're looking for we can look into adding it as a first class feature.

@Andrew_Cholakian1 No worries.
Actually, I've made an implement in my local branch.
Wondering if I can file a PR then we could discuss there?
Thanks.

@Andrew_Cholakian1
Hi Andrew, filed a PR in beats repo:

Can you pls comment in there when you get a chance?
Thanks.