Monitor beat's publisher request to Elastic Search

Elastic Stack Beats
1

Hi,

I am writing a new beat for a corporate service.

I have generated the index template, added it to ElasticSearch. I have my beat running publishing event, everything seem to be ok with no error in debug level logs.

However no index is created for my beat, and events do not show up in elasticsearch/Kibana
I don't know why.

Is there a way to monitor libbeat's publisher 's requests to ElasticSearch?

2

Monitoring network with Wireshark it seems like no requests are issued from my beat to Elastic Search.

3

Please show your Filebeat configuration, and make sure you format it as code with the </> toolbar button.

4

I am not using Filebeat. I am using my own beat. When I run Filebeat or TopbEat everything is fine. It must be an issue with my bit.

################### MyBeat Configuration Example #########################

############################# Input ############################################
input:
  # In seconds, defines how often to read server statistics
  period: 10

  # MyService  information
  myservicet:

    # Host on wich myservice server is listening
    host: "localhost:3000"
###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  ### Elasticsearch as output
  elasticsearch:

    hosts: ["localhost:9200"]

    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "admin"
    #password: "s3cr3t"

    # Dictionary of HTTP parameters to pass within the url with index operations.
    #parameters:
      #param1: value1
      #param2: value2

    # Number of workers per Elasticsearch host.
    #worker: 1

    # Optional index name. The default is "myservicebeat" and generates
    # [myservicebeat-]YYYY.MM.DD keys.
    #index: "myservicebeat"

    # A template is used to set the mapping in Elasticsearch
    # By default template loading is disabled and no template is loaded.
    # These settings can be adjusted to load your own template or overwrite existing ones
    #template:

      # Template name. By default the template name is myservicebeat.
      #name: "myservicebeat"

      # Path to template file
      #path: "myservicebeat.template.json"

      # Overwrite existing template
      #overwrite: false

    # Optional HTTP Path
    #path: "/elasticsearch"

    # Proxy server url
    #proxy_url: http://proxy:3128

    # The number of times a particular Elasticsearch index operation is attempted. If
    # the indexing operation doesn't succeed after this many retries, the events are
    # dropped. The default is 3.
    #max_retries: 3

    # The maximum number of events to bulk in a single Elasticsearch bulk API index request.
    # The default is 50.
    #bulk_max_size: 50

    # Configure http request timeout before failing an request to Elasticsearch.
    #timeout: 90

    # The number of seconds to wait for new events between two bulk API index requests.
    # If `bulk_max_size` is reached before this interval expires, addition bulk index
    # requests are made.
    #flush_interval: 1

    # Boolean that sets if the topology is kept in Elasticsearch. The default is
    # false. This option makes sense only for Packetbeat.
    #save_topology: false

    # The time to live in seconds for the topology information that is stored in
    # Elasticsearch. The default is 15 seconds.
    #topology_expire: 15



  ### Console output
  console:
    # Pretty print json event
    pretty: false


############################# Logging #########################################

logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  to_syslog: false

  # Write all logging output to files. Beats automatically rotate files if rotateeverybytes
  # limit is reached.
  to_files: true

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    path: ./logs

    # The name of the files where the logs are written to.
    name: myservicebeat_logs

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7


  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  level: debug
5

Can you start your beat with -e -d "*" and share the output? What libbeat version are your relying on?

6

It turns out I forgot to change the IP in the configuration file and I would have swear I had done it (which in fict I did).
But each time you make a

make update

you have to rewrite your config file of course.

I 'm feeling like these days when you just want to disappear.

7

Yes, it regenerates the config file every time.

For development I recommend to use a config file beatname.dev.yml which should be in .gitignore.