HTTP Response 401 "Unauthorized" Using API_KEY (MetricBeat --> Kibana)

Lokesh_Sachdeva · May 24, 2022, 7:36am

Hi Elastic Team
We are trying to load data from Metricbeat to Kibana using API KEY. API Key is created and provided full superuser (elastic role) access.

API KEY is able to authenticate into Elasticsearch but while connecting to Kibana during metricbeat setup (.\metricbeat setup -e), it failed with an error : HTTP 401 Unauthorize.

https://localhost:5601/api/status fails: . Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.

Please note that curl on same URL is working fine with same Authorization Key.

Please find below the Logs and API KEY Access JSON document.

Setup From MetricBeat:

.\Metricbet setup -e
Running Using Metric beat Its failing while loading into Kibana with HTTP 401 Unauthorize Error

2022-05-21T18:14:52.729+0530    INFO    instance/beat.go:299    Setup Beat: winlogbeat; Version: 7.10.2
2022-05-21T18:14:52.748+0530    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'winlogbeat-7.10.2' as ILM is enabled.
2022-05-21T18:14:52.765+0530    INFO    eslegclient/connection.go:99    elasticsearch url: https://localhost:9200
2022-05-21T18:14:52.771+0530    INFO    [publisher]     pipeline/module.go:113  Beat name: MMD045048702357
2022-05-21T18:14:52.774+0530    INFO    beater/winlogbeat.go:69 State will be read from and persisted to C:\Projects\Data\ELK\Beats\WinLogBeat\data\.winlogbeat.yml
2022-05-21T18:14:53.098+0530    WARN    [cfgwarn]       registered_domain/registered_domain.go:60       BETA: The registered_domain processor is beta.
2022-05-21T18:14:53.245+0530    WARN    [cfgwarn]       registered_domain/registered_domain.go:60       BETA: The registered_domain processor is beta.
2022-05-21T18:14:53.266+0530    INFO    eslegclient/connection.go:99    elasticsearch url: https://localhost:9200
2022-05-21T18:14:53.364+0530    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 8.0.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2022-05-21T18:14:53.374+0530    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2022-05-21T18:14:53.385+0530    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2022-05-21T18:14:53.385+0530    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2022-05-21T18:14:53.388+0530    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{winlogbeat-7.10.2 {now/d}-000001}' as ILM is enabled.
2022-05-21T18:14:53.404+0530    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'winlogbeat-7.10.2-*' as ILM is enabled.
2022-05-21T18:14:53.410+0530    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {winlogbeat-7.10.2 {now/d}-000001} as ILM is enabled.
2022-05-21T18:14:53.414+0530    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {winlogbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-05-21T18:14:53.424+0530    INFO    template/load.go:183    Existing template will be overwritten, as overwrite is enabled.
2022-05-21T18:14:53.564+0530    INFO    template/load.go:117    Try loading template winlogbeat-7.10.2 to Elasticsearch
2022-05-21T18:14:53.908+0530    INFO    template/load.go:109    template with name 'winlogbeat-7.10.2' loaded.
2022-05-21T18:14:53.909+0530    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2022-05-21T18:14:53.918+0530    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2022-05-21T18:14:53.929+0530    INFO    kibana/client.go:119    Kibana url: https://localhost:5601
2022-05-21T18:14:53.992+0530    ERROR   instance/beat.go:956    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://localhost:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://localhost:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.

Connection To Kibana Role

Preformatted text`curl -v --cacert "C:\Projects\Certificates\Signer\ca.crt" --location --request GET 'https://localhost:5601/api/security/role' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='
Note: Unnecessary use of -X or --request, GET is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connected to localhost (::1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: C:\Projects\Certificates\Signer\ca.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2018 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=GB; ST=XXXXX; L=XXXXX; O=XXXXX; OU=GI; CN=Server_Certificate
*  start date: Apr 29 07:54:26 2022 GMT
*  expire date: Apr 28 07:54:26 2024 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: DC=com; DC=XXXXXX; DC=test01global; CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /api/security/role HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.78.0
> Accept: */*
> Content-Type: application/json;charset=UTF-8
> kbn-xsrf: true
> Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: a29bd265d8d8
< kbn-license-sig: a2b93ac49b41ba98f7fefe5376d8a057dea29560144ae157e0ff5e4c42f0d57b
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 17141
< vary: accept-encoding
< accept-ranges: bytes
< Date: Sat, 21 May 2022 12:39:20 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{ [15918 bytes data]
100 17141  100 17141    0     0   155k      0 --:--:-- --:--:-- --:--:--  160kcations":[]},{"name":"watcher_admin","metadata":{"_reserved":true},"transient_metadata":{"enabled":true},"elasticsearch":{"cluster":["manage_watcher"],"indices":[{"names":[".watches",".triggered_watches",".watcher-history-*"],"privileges":["read"],"allow_restricted_indices":true}],"run_as":[]},"kibana":[],"_transform_error":[],"_unrecognized_applications":[]},{"name":"watcher_user","metadata":{"_reserved":true},"transient_metadata":{"enabled":true},"elasticsearch":{"cluster":["monitor_watcher"],"indices":[{"names":[".watches"],"privileges":["read"],"allow_restricted_indices":true},{"names":[".watcher-history-*"],"privileges":["read"],"allow_restricted_indices":false}],"run_as":[]},"kibana":[],"_transform_error":[],"_unrecognized_applications":[]}]
* Connection #0 to host localhost left intact

Connection To Elasticsearch:

curl -v --cacert "C:\Projects\Data\ELK\Beats\WinLogBeat\certs\ca.crt" --location --request GET 'https://localhost:9200/_security/_authenticate' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1:9200...
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: C:\Projects\Data\ELK\Beats\WinLogBeat\certs\ca.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [32 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [877 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=es01
*  start date: May 19 10:43:14 2022 GMT
*  expire date: May 18 10:43:14 2025 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /_security/_authenticate HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.78.0
> Accept: */*
> Content-Type: application/json;charset=UTF-8
> kbn-xsrf: true
> Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [1168 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 266
<
{ [266 bytes data]
100   266  100   266    0     0   4019      0 --:--:-- --:--:-- --:--:--  4222{"username":"elastic","roles":[],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"_es_api_key","type":"_es_api_key"},"lookup_realm":{"name":"_es_api_key","type":"_es_api_key"},"authentication_type":"api_key"}
* Connection #0 to host localhost left intact

curl -v --cacert "C:\Projects\Certificates\Signer\ca.crt" --location --request GET 'https://localhost:5601/api/status' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='

* Connected to localhost (::1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: C:\Projects\Certificates\Signer\ca.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2018 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=XX; ST=XXXXXXXX; L=XXXXXX; O=XXXXXXXX; OU=GI; CN=ElasticCertificate_client
*  start date: Apr 29 07:54:26 2022 GMT
*  expire date: Apr 28 07:54:26 2024 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: DC=com; DC=XXXXXXX; DC=test01global; CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /api/status HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.78.0
> Accept: */*
> Content-Type: application/json;charset=UTF-8
> kbn-xsrf: true
> Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: a29bd265d8d8
< kbn-license-sig: a2b93ac49b41ba98f7fefe5376d8a057dea29560144ae157e0ff5e4c42f0d57b
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 12533
< vary: accept-encoding
< accept-ranges: bytes
< Date: Sat, 21 May 2022 12:50:16 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{ [12533 bytes data]
100 12533  100 12533    0     0   203k      0 --:--:-- --:--:-- --:--:--  214k1,"95":10.690559,"99":15.335423}},"uptime_in_millis":7601839.708579}],"response_times":{"avg_in_millis":341,"max_in_millis":460},"concurrent_connections":1,"requests":{"disconnects":0,"total":2,"statusCodes":{"200":2},"status_codes":{"200":2}}}}
* Connection #0 to host localhost left intact

Connection To Kibana API Status

> curl -v --cacert "C:\Projects\Certificates\Signer\ca.crt" --location --request GET 'https://localhost:5601/api/status' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='
> 
> * Connected to localhost (::1) port 5601 (#0)
> 
> * ALPN, offering h2
> 
> * ALPN, offering http/1.1
> 
> * successfully set certificate verify locations:
> 
> * CAfile: C:\Projects\Certificates\Signer\ca.crt
> 
> * CApath: none
> 
> } [5 bytes data]
> 
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> 
> } [512 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> 
> { [122 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> 
> { [21 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> 
> { [2018 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> 
> { [264 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> 
> { [52 bytes data]
> 
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> 
> } [1 bytes data]
> 
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> 
> } [52 bytes data]
> 
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> 
> * ALPN, server accepted to use http/1.1
> 
> * Server certificate:
> 
> * subject: C=XX; ST=XXXXXXX; L=XXXXX; O=XXXXXXXX; OU=GI; CN=ElasticCertificate_client
> 
> * start date: Apr 29 07:54:26 2022 GMT
> 
> * expire date: Apr 28 07:54:26 2024 GMT
> 
> * subjectAltName: host "localhost" matched cert's "localhost"
> 
> * issuer: DC=com; DC=XXXXXXX; DC=testglobal; CN=Elastic Certificate Tool Autogenerated CA
> 
> * SSL certificate verify ok.
> 
> } [5 bytes data]
> 
> > GET /api/status HTTP/1.1
> 
> > Host: localhost:5601
> 
> > User-Agent: curl/7.78.0
> 
> > Accept: */*
> 
> > Content-Type: application/json;charset=UTF-8
> 
> > kbn-xsrf: true
> 
> > Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
> 
> >
> 
> { [5 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> 
> { [265 bytes data]
> 
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> 
> { [265 bytes data]
> 
> * old SSL session ID is stale, removing
> 
> { [5 bytes data]
> 
> * Mark bundle as not supporting multiuse
> 
> < HTTP/1.1 200 OK
> 
> < x-content-type-options: nosniff
> 
> < referrer-policy: no-referrer-when-downgrade
> 
> < kbn-name: a29bd265d8d8
> 
> < kbn-license-sig: a2b93ac49b41ba98f7fefe5376d8a057dea29560144ae157e0ff5e4c42f0d57b
> 
> < content-type: application/json; charset=utf-8
> 
> < cache-control: private, no-cache, no-store, must-revalidate
> 
> < content-length: 12533
> 
> < vary: accept-encoding
> 
> < accept-ranges: bytes
> 
> < Date: Sat, 21 May 2022 12:50:16 GMT
> 
> < Connection: keep-alive
> 
> < Keep-Alive: timeout=120
> 
> <
> 
> { [12533 bytes data]
> 
> 100 12533 100 12533 0 0 203k 0 --:--:-- --:--:-- --:--:-- 214k1,"95":10.690559,"99":15.335423}},"uptime_in_millis":7601839.708579}],"response_times":{"avg_in_millis":341,"max_in_millis":460},"concurrent_connections":1,"requests":{"disconnects":0,"total":2,"statusCodes":{"200":2},"status_codes":{"200":2}}}}
> 
> * Connection #0 to host localhost left intact

API KEY JSON

---------------------------API KEY Role Addition------------------------
POST /_security/api_key
{
  "name": "ApiKey_Beats_User",
  "expiration": "365d",
  "role_descriptors": {
      "beats_cluster_access" : {
    "cluster" : [
      "all",
      "grant_api_key",
      "manage",
      "manage_api_key",
      "manage_autoscaling",
      "manage_ccr",
      "manage_data_frame_transforms",
      "delegate_pki",
      "manage_enrich",
      "manage_ilm",
      "manage_index_templates",
      "manage_ingest_pipelines",
      "manage_logstash_pipelines",
      "manage_ml",
      "manage_security"
    ],
    "indices" : [
      {
        "names" : [
          "*"
        ],
        "privileges" : [
          "all",
          "delete",
          "create_index",
          "create_doc",
          "index",
          "maintenance",
          "manage_follow_index",
          "manage_leader_index",
          "manage",
          "read",
          "manage_ilm",
          "auto_configure",
          "create",
          "monitor",
          "view_index_metadata",
          "delete_index",
          "read_cross_cluster",
          "write"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [
      {
        "application" : "kibana-.kibana",
        "privileges" : [
          "all"
        ],
        "resources" : [
          "*"
        ]
      }
    ],
    "run_as" : [
      "elastic"
    ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
  }
}

TonyWRobinson · June 3, 2022, 8:34pm

I'd like to here anyone's thoughts on this as well. I want to use api_key for authentication from metricbeat to the Elasticsearch cluster as well, but thus far the module configuration only seems to work with username and password. We definitely won't put the user name and password in plain text in a config, and I'd really like to avoid having to setup and manage local keystores on the various nodes to store credentials. I did find this issue on GitHub, so would be great if anyone has any updates:

github.com/elastic/beats

Metricbeat doesn't work properly with `api_key` in metricbeat elasticsearch module but work ok with `username`\`password`

opened 09:36PM - 04 Dec 21 UTC

gitpel

module Metricbeat Team:Integrations Feature: Stack Monitoring

Metricbeat doesn't work properly with `api_key` in metricbeat elasticsearch modu…le but work ok with `username\password` the same `api_key` works fine in the `output.elasticsearch.api_key` `/etc/metricbeat/modules.d/elasticsearch-xpack.yml` ```yaml --- - module: elasticsearch xpack.enabled: true period: 10s hosts: ["https://fqdn:8443"] ssl: certificate_authorities: - /etc/ssl/certs/ca-certificates.crt api_key: xxxxxxx:yyyyyyyy ``` ! note that the same `api_key` works fine in `/etc/metricbeat/metricbeat.yml` ```yaml --- output: elasticsearch: hosts: ['https://server-01:9200', 'https://server-02:9200', ...] ssl: certificate_authorities: ['/etc/ssl/certs/ca-certificates.crt'] api_key: xxxxxxx:yyyyyyyy metricbeat: config: modules: path: ${path.config}/modules.d/*.yml reload: enabled: false processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` error log: ```syslog .... Dec 4 21:12:00 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:00.767Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.node_stats: HTTP error 401 in : 401 Unauthorized Dec 4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.606Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.enrich: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized Dec 4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.610Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized Dec 4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.632Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized Dec 4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.660Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized .... ``` - Version: `metricbeat/stable,now 7.15.2 amd64 [installed]` - Operating System: `Debian GNU/Linux 11 (bullseye)` solution: use username\password instead of api_key in `/etc/metricbeat/modules.d/elasticsearch-xpack.yml`

stephenb · June 6, 2022, 4:27am

@Lokesh_Sachdeva

Looks like it is failing during setup.. perhaps the setup role is not defined correctly

Look like perhaps the kibana_admin role is missing? (it is not apologies)

TonyWRobinson · June 6, 2022, 12:55pm

Hi, @stephenb When we create an api key, we provided cluster and index privileges. Is there a part of that call where we also can add existing role names? Or do we just include the combination of privileges the kibana_admin role possesses?

I tried granting ALL cluster privileges to the .metricbeat-* and metricbeat-* indexes to the API key I created, and it still didn't work. When I switch to using the remote_monitoring_user, it works like a charm, which is what makes me think something is amiss.

To be clear: in my case, this is for the Elasticsearch MODULE. The Elasticsearch output section seems to take the API key without issue.

This is the API we are using to create the api key

stephenb · June 6, 2022, 3:12pm

@Lokesh_Sachdeva Please Confirm what version of Elasticsearch and Winlogbeat you are using

It look like perhaps you are using Elasticsearch 8.0.0 and Winlogbeat 7.10.2 is there a reason for that? or is it Metrcibeat (confused title says Metricbeat one thing details say Winlogbeat)

This is very confusing your command says metricbeat but the output is winlogbeat so I do not know what you are actually doing / running.

.\Metricbet setup -e
Running Using Metric beat Its failing while loading into Kibana with HTTP 401 Unauthorize Error

2022-05-21T18:14:52.729+0530    INFO    instance/beat.go:299    Setup Beat: winlogbeat; Version: 7.10.2
2022-05-21T18:14:52.748+0530    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'winlogbeat-7.10.2' as ILM is enabled.
2022-05-21T18:14:52.765+0530    INFO    eslegclient/connection.go:99    elasticsearch url: https://localhost:9200
2022-05-21T18:14:52.771+0530    INFO    [publisher]     pipeline/module.go:113  Beat name: MMD045048702357
2022-05-21T18:14:52.774+0530    INFO    beater/winlogbeat.go:69 State will be read from and persisted to C:\Projects\Data\ELK\Beats\WinLogBeat\data\.winlogbeat.yml
2022-05-21

There are pretty significant Changes between the 2 versions.

The privileges will need to fit 8.0 or whatever version of Elasticsearch you are running

Example I notice for setup role requires setup_admin

GET _security/role/ingest_admin

{
  "ingest_admin" : {
    "cluster" : [
      "manage_index_templates",
      "manage_pipeline" <!--- you have   "manage_ingest_pipelines"
    ],
    "indices" : [ ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : {
      "_reserved" : true
    },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

Which does not match up

I would go through the the 2 pages above set the cluster and index privileges... run the GET on each of the roles and set what is needed.

stephenb · June 6, 2022, 4:38pm

@Lokesh_Sachdeva

I just went through the Docs (this was 8.2.2)

And the following all worked, I created these while logged in as the elastic user.
This was for metricbeat but should follow same pattern for winlogbeat. ( I have a mac so I can not run winlogbeat)
I would also try to get your metricbeat and /or winlogbeat and Elasticsearch to the same version.

# SETUP ONLY 
POST /_security/api_key
{
  "name": "metricbeat-setup",
  "role_descriptors": {
    "metricbeat-setup": {
      "cluster": [
        "monitor",
        "manage_ilm",
        "manage_index_templates",
        "manage_pipeline"
      ],
      "indices": [
        {
          "names": [
            "metricbeat-*"
          ],
          "privileges": [
            "manage"
          ]
        }
      ],
      "applications": [
        {
          "application": "kibana-.kibana",
          "privileges": [
            "all"
          ],
          "resources": [
            "*"
          ]
        }
      ]
    }
  }
}


#PUBLISH ONLY 
POST /_security/api_key
{
  "name": "metricbeat-setup", 
  "role_descriptors": {
    "metricbeat_writer": { 
      "cluster": ["monitor", "read_ilm", "read_pipeline"],
      "index": [
        {
          "names": ["metricbeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}


# SETUP AND PUBLISH
POST /_security/api_key
{
  "name": "metricbeat-setup-publish",
  "role_descriptors": {
    "metricbeat-setup": {
      "cluster": [
        "monitor",
        "manage_ilm",
        "manage_index_templates",
        "manage_pipeline",
        "read_ilm",
        "read_pipeline"
      ],
      "indices": [
        {
          "names": [
            "metricbeat-*"
          ],
          "privileges": [
            "manage",
            "view_index_metadata",
            "create_doc"
          ]
        }
      ],
      "applications": [
        {
          "application": "kibana-.kibana",
          "privileges": [
            "all"
          ],
          "resources": [
            "*"
          ]
        }
      ]
    }
  }
}

stephenb · June 6, 2022, 4:41pm

@TonyWRobinson I think that is a separate issue from OP I would suggest opening a separate topic and be clear (like you are) that you need an API KEY to work with the metricbeat Elasticsearch module.

system · July 4, 2022, 4:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.