Metricbeat seems to be reading the same config value and getting different values

Elastic Stack Beats
1

Hi all!

Forgive me if this is simply my own ignorance about Beats architecture as it very well might be but I've come across something very curious and I'd love some input from the developer community.

Assume the following configuration present in /tmp/config.yml:

metricbeat:
  config:
    modules:
      path: /Users/mikeplace/devel/elastic/src/github.com/elastic/beats/metricbeat/modules.d/*.yml
      reload:
        enabled: false
output:
  elasticsearch:
    hosts:
    - localhost:9200
path:
  config: /Users/mikeplace/devel/elastic/src/github.com/elastic/beats/metricbeat
  data: /Users/mikeplace/devel/elastic/src/github.com/elastic/beats/metricbeat/data
  home: /Users/mikeplace/devel/elastic/src/github.com/elastic/beats/metricbeat
  logs: /Users/mikeplace/devel/elastic/src/github.com/elastic/beats/metricbeat/logs
processors:
- add_host_metadata: null
- add_cloud_metadata: null
setup:
  kibana: null
  template:
    settings:
      index:
        codec: best_compression
        number_of_shards: 1

metricbeat.modules:
- module: mysql
  metricsets:
    - "status"
  period: 10s
  hosts: ["wrong:test@tcp(127.0.0.1:3306)/"]

Note the very last line, which contains an intentionally incorrect DSN, specifically a username indicated as wrong.

Upon starting Metricbeat using the following command: ./metricbeat run -N --path.config /tmp -c config.yml some surprising behavior is observed on a MySQL server running locally. Below is a snippet of standard out on that service:

2018-12-11T20:39:44.842594Z 4692 [Note] Access denied for user 'root'@'172.25.0.1' (using password: YES)
2018-12-11T20:39:50.729821Z 4699 [Note] Access denied for user 'wrong'@'172.25.0.1' (using password: YES)
2018-12-11T20:39:54.853510Z 4704 [Note] Access denied for user 'root'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:00.737084Z 4710 [Note] Access denied for user 'wrong'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:04.856834Z 4715 [Note] Access denied for user 'root'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:10.744072Z 4721 [Note] Access denied for user 'wrong'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:14.843327Z 4726 [Note] Access denied for user 'root'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:20.729193Z 4733 [Note] Access denied for user 'wrong'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:24.850356Z 4737 [Note] Access denied for user 'root'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:30.736458Z 4744 [Note] Access denied for user 'wrong'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:34.860999Z 4749 [Note] Access denied for user 'root'@'172.25.0.1' (using password: YES)
2018-12-11T20:40:40.745147Z 4755 [Note] Access denied for user 'wrong'@'172.25.0.1' (using password: YES)

Note two things.

First, and most surprisingly, two usernames are being used to attempt to authenticate to the MySQL service. My expectation would have been that only one would be used.

Second -- it appears that there are two pollers operating every ten seconds. The first is probably the status module but what is the second?

Upon further investigation, something even more surprising was discovered which is that the Metricbeat module config parser seems to be returning different values depending on the caller.

I printed out some stack traces and some debugging statements and observed the following:

Note in the above that different values for the DSN that's being used are being generated based on what appears to be the caller.

So, my questions are as follows:

  1. Why are there two pollers trying to access the DB when just the status module should be attempting to access the DB?

  2. Why are the authentication attempts using different DSNs when only one is set in the configuration? Is the status module specifically generating one and then another is coming from...somewhere?

Again, I'm new to Beats and this might absolutely be my misunderstanding but I'd be quite pleased if somebody could set me straight. Thanks!

Thanks!

2

Can you remove the metricbeat.config.modules section and run the tests again. It may be pulling in configuration from that directory. It seems like Metricbeat is using a period of 10s for the “wrong” user and separate 10s polling rate for the “root” user, which would be normal if you had configured two separate instances of the module.

3

Yup! Good call. That looks like it!

Is it intended behavior that multiple configuration blocks merged together in this manner should result in multiple invocations of a poller? (Forgive me if that terminology isn't quite right.) :slight_smile:

4

Yes, you can configure multiple instances of any module and you can configure them in both the “modules.d” directory or in the main main configuration file. But most poeple chose just one way.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.