Metricbeat 6.4.1 - config test gives false ok


(Alex Riemenschneider) #1

Hi,

I'm trying to set up Metricbeat dashboards in our Kibana, but I'm getting authorization failures:

metricbeat setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [] for REST request [/_xpack/security/_authenticate], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}

My Metricbeat config is:

[root@xalnplvapxx metricbeat]# metricbeat export config
fields:
  env: production
logging:
  level: info
metricbeat:
  config:
    modules:
      path: /etc/metricbeat/modules.d/*.yml
      reload:
        enabled: false
output:
  logstash:
    hosts:
    - '[2a00:e400:xx::xx]:10104'
path:
  config: /etc/metricbeat
  data: /var/lib/metricbeat
  home: /usr/share/metricbeat
  logs: /var/log/metricbeat
setup:
  kibana:
    host: https://xalzplvapxx.xxx.net:5601
    password: xxxx
    setup:
      dashboards:
        index: os-xxx-metricbeat-*
    ssl:
      certificate_authorities:
      - /etc/kibana/certs/xxx.net-wildcard.ca.crt
    user: xxxx
  template:
    settings:
      _source:
        enabled: false
      index:
        codec: best_compression
        number_of_shards: 1
tags:
- ELK
- kibana

Checking with curl, I don't get authentication errors:

[root@xalnplvapxx metricbeat]# curl -k -u xxxx -p https://xalzplvapxx.xxx.net:5601/api/status
Enter host password for user 'xxxx':
{"name":"kibana.xxx.net","uuid":"21e470b6-439d-4a1d-a78b-4edb1657d500","version":{"number":"6.3.1","build_hash":"cb8398243cee7be0dddd118cd657d1f17be4e4be","build_number":17276,"build_snapshot":false},"status":{"overall":{"state":"green","title":"Green","nickname":"Looking good","icon":"success","since":"2018-10-16T10:00:03.451Z"},"statuses":[{"id":"plugin:kibana@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.451Z"},{"id":"plugin:elasticsearch@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.213Z"},{"id":"plugin:xpack_main@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:searchprofiler@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:tilemap@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:watcher@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:license_management@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.505Z"},{"id":"plugin:index_management@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:timelion@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.610Z"},{"id":"plugin:graph@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:monitoring@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.614Z"},{"id":"plugin:security@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:grokdebugger@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:dashboard_mode@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.644Z"},{"id":"plugin:logstash@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"},{"id":"plugin:console@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.653Z"},{"id":"plugin:console_extensions@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.662Z"},{"id":"plugin:metrics@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:03.664Z"},{"id":"plugin:reporting@6.3.1","state":"green","icon":"success","message":"Ready","since":"2018-10-16T10:00:04.267Z"}]},"metrics":{"last_updated":"2018-10-18T15:10:28.585Z","collection_interval_in_millis":5000,"uptime_in_millis":191433466,"process":{"mem":{"heap_max_in_bytes":138051584,"heap_used_in_bytes":119806920}},"os":{"cpu":{"load_average":{"1m":0.1767578125,"5m":0.08447265625,"15m":0.06103515625}},"cgroup":{"cpuacct":{"control_group":"/","usage_nanos":119891068365322},"cpu":{"control_group":"/","cfs_period_micros":100000,"cfs_quota_micros":-1,"stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}}}},"response_times":{"avg_in_millis":null,"max_in_millis":0},"requests":{"total":0,"disconnects":0,"status_codes":{}},"concurrent_connections":0}}

I'm using Metricbeat 6.4.2, although the Elastic Stack is still at version 6.3.1.
In the Elasticsearch Log I see a Active Directory authentication error, even if I use the elastic user in the metricbeat config:

[2018-10-18T16:47:46,702][WARN ][o.e.x.s.a.AuthenticationService] [xalzplvdbxx.xxx.de] Authentication to realm active_directory failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580', diagnosticMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580'))

This is puzzling to me, as the Native Realm has order 0 and the AD Realm order 5.
Apart from that, everything is working as expected. Can somebody point me in the right direction?


(Alex Riemenschneider) #2

I've found the error in my config. I have used

setup.kibana.user

instead of

setup.kibana.username

Unfortunately, a

metricbeat test config

told me that this config would be ok.


(Mario Castro) #3

Hi @ariemenschneider :slight_smile:

I'm glad that you found the solution. Do you mind to file us an issue reproducing the problem with test config command, please?

Best regards


(Alex Riemenschneider) #4

Hi Mario,

thanks for looking into this. To reproduce:

[root@xalnplvapxx metricbeat]# grep setup metricbeat.yml 
setup.template.settings:
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
#setup.dashboards.url:
setup.kibana:
setup.dashboards.index: "os-xxx-metricbeat-*"
setup.dashboards.always_kibana: true
setup.kibana.username: "ariemenschneider"
setup.kibana.password: "${ariemenschneider_password}"
setup.kibana.ssl.enabled: true
# `setup.kibana.host` options.
[root@xalnplvapxx metricbeat]# metricbeat test config path.config=/etc/metricbeat
Config OK                                                                                                                                                                                                                                    
[root@xalnplvapxx metricbeat]#

This is a working config and everything is fine...

[root@xalnplvapxx metricbeat]# grep setup metricbeat.yml                                                                                                                                                             
setup.template.settings:
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false
#setup.dashboards.url:
setup.kibana:
setup.dashboards.index: "os-xxx-metricbeat-*"
setup.dashboards.always_kibana: true
setup.kibana.user: "ariemenschneider"
setup.kibana.password: "${ariemenschneider_password}"
setup.kibana.ssl.enabled: true
# `setup.kibana.host` options.
[root@xalnplvapxx metricbeat]# metricbeat test config path.config=/etc/metricbeat                                                                                                                                                     
Config OK                                                                                                                                                                                                                                    
[root@xalnplvapxx metricbeat]#

This is a non-working config and should be caught by metricbeat test config. Note the setup.kibana.user: instead of setup.kibana.username:

[root@xalnplvapxx metricbeat]# metricbeat version
metricbeat version 6.4.1 (amd64), libbeat 6.4.1 [37b5f2d2a20f2734b2373a454b4b4cbb2627e841 built 2018-09-13 21:29:53 +0000 UTC]

System is RHEL7.5 and installation method was by RPM from the Elastic Repository.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.