Any way to maintain a long list of hearbeat.monitors.urls?

Elastic Stack Beats
1

Hi,

My list in heartbeat.monitors.urls is long, is there any way to maintain it in a separate file.

I mean I want to it to be:

heartbeat.yml:

# Configure monitors inline
heartbeat.monitors:
- type: http

  # List or urls to query
  urls:
   path: **/path/to/my.yml**

my.list: (maybe .yml)

http://elasticsearch:9200
http://logstash:9600
http://....
.... long list.
....
2

Hi @zongzw, welcome to discuss :slight_smile:

I don't know of a way to obtain the list of urls from a separated file, but something you can do is to define the monitor in a separate file, like this:

- type: http
  urls:
    - http://elasticsearch:9200
    - http://logstash:9600
    - http://....
    .... long list.
    ....

Files like this one, with monitor definitions, can be included with heartbeat.config.monitors from the main configuration file:

heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s
3

Hi @jsoriano, thanks.

Yes, this is the best way for now to maintain the long list. :handshake:

4

You can create new file following the templates as much as you want.
I have several hundred files that created by bash script that only have single target, as follow

- type: icmp 
  name: cisco
  enabled: true
  schedule: '*/30 * * * * * *' 
  hosts: ["HOSTNAME01"]
  ipv4: true
  ipv6: false
  mode: any
  timeout: 16s
  wait: 1s

Each target have each file and I have several hundred files for heartbeat monitors, and no performance degradation.

This is can be done for http target instead of icmp.
If you need the bash script samples, please let me know

Regards,
Fadjar Tandabawana

1 Like
5

Thanks for kindly help. You are right. we may generate it from templates.
About the generation, the first idea comes to my mind is using scripts to convert json to yaml like https://www.json2yaml.com/ does. Is that same with your bash script samples?

6

This is my bash script samples:

    #!/bin/bash

    # Cisco IP List
    iplist=cisco_ip.txt

    # Environment

    base_file=base.txt
    DIRECTORY=`dirname $0`
    #echo $DIRECTORY

    # Cleanup configuration file
    echo "Cleanup conf file"
    rm -rf $DIRECTORY/*.yml


    # file naming
    first=10
    increment=1


    # Alive IP processing
    echo ""
    echo "Reading IP List"
    readarray -t iparray < $DIRECTORY/$iplist


    #length_oid=${#oid[@]}
    length_ip=${#iparray[@]}

    vals=($(seq -w -s ' '10 $increment $length_ip))

    for ((i=0; i<${length_ip}; i++)); do
      sed s/%host%/${iparray[$i]}/g $DIRECTORY/$base_file >> $DIRECTORY/cisco-${vals[$i]}-heartbeat-${iparray[$i]}.yml
    done

    # Rename anomaly file
    filemv=`basename $DIRECTORY/cisco-0*`
    filename=${filemv#"cisco-"}
    newfile=cisco-10$filename
    #echo $newfile
    mv $DIRECTORY/$filemv $DIRECTORY/$newfile


    echo ""
    # Remove input configuration in /etc/heartbeat/monitors.d
    rm -rf /etc/heartbeat/monitor.d/cisco*

    sleep 1

    echo "Build metricbeat Cisco"
    mv $DIRECTORY/*.yml /etc/heartbeat/monitors.d/

    echo "DONE"

And this is the template file (base.txt)

- type: icmp 
  name: cisco
  enabled: true
  schedule: '*/30 * * * * * *' 
  hosts: ["%host%"]
  ipv4: true
  ipv6: false
  mode: any
  timeout: 16s
  wait: 1s

The ciscto.txt is the list of the target hosts, one line per host

Regards,
Fadjar Tandabawana

7

That's cool. and That's what you meant "Each target have each file"
I got it. Appreciate!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.