Hi Elastic Team
We are trying to load data from Metricbeat to Kibana using API KEY. API Key is created and provided full superuser (elastic role) access.
API KEY is able to authenticate into Elasticsearch but while connecting to Kibana during metricbeat setup (.\metricbeat setup -e), it failed with an error : HTTP 401 Unauthorize.
https://localhost:5601/api/status fails: . Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.
Please note that curl on same URL is working fine with same Authorization Key.
Please find below the Logs and API KEY Access JSON document.
Setup From MetricBeat:
.\Metricbet setup -e
Running Using Metric beat Its failing while loading into Kibana with HTTP 401 Unauthorize Error
2022-05-21T18:14:52.729+0530 INFO instance/beat.go:299 Setup Beat: winlogbeat; Version: 7.10.2
2022-05-21T18:14:52.748+0530 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'winlogbeat-7.10.2' as ILM is enabled.
2022-05-21T18:14:52.765+0530 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
2022-05-21T18:14:52.771+0530 INFO [publisher] pipeline/module.go:113 Beat name: MMD045048702357
2022-05-21T18:14:52.774+0530 INFO beater/winlogbeat.go:69 State will be read from and persisted to C:\Projects\Data\ELK\Beats\WinLogBeat\data\.winlogbeat.yml
2022-05-21T18:14:53.098+0530 WARN [cfgwarn] registered_domain/registered_domain.go:60 BETA: The registered_domain processor is beta.
2022-05-21T18:14:53.245+0530 WARN [cfgwarn] registered_domain/registered_domain.go:60 BETA: The registered_domain processor is beta.
2022-05-21T18:14:53.266+0530 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
2022-05-21T18:14:53.364+0530 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 8.0.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2022-05-21T18:14:53.374+0530 INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2022-05-21T18:14:53.385+0530 INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2022-05-21T18:14:53.385+0530 INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2022-05-21T18:14:53.388+0530 INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{winlogbeat-7.10.2 {now/d}-000001}' as ILM is enabled.
2022-05-21T18:14:53.404+0530 INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'winlogbeat-7.10.2-*' as ILM is enabled.
2022-05-21T18:14:53.410+0530 INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {winlogbeat-7.10.2 {now/d}-000001} as ILM is enabled.
2022-05-21T18:14:53.414+0530 INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {winlogbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-05-21T18:14:53.424+0530 INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2022-05-21T18:14:53.564+0530 INFO template/load.go:117 Try loading template winlogbeat-7.10.2 to Elasticsearch
2022-05-21T18:14:53.908+0530 INFO template/load.go:109 template with name 'winlogbeat-7.10.2' loaded.
2022-05-21T18:14:53.909+0530 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2022-05-21T18:14:53.918+0530 INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2022-05-21T18:14:53.929+0530 INFO kibana/client.go:119 Kibana url: https://localhost:5601
2022-05-21T18:14:53.992+0530 ERROR instance/beat.go:956 Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://localhost:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://localhost:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}.
Connection To Kibana Role
Preformatted text`curl -v --cacert "C:\Projects\Certificates\Signer\ca.crt" --location --request GET 'https://localhost:5601/api/security/role' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='
Note: Unnecessary use of -X or --request, GET is already inferred.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connected to localhost (::1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: C:\Projects\Certificates\Signer\ca.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2018 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=GB; ST=XXXXX; L=XXXXX; O=XXXXX; OU=GI; CN=Server_Certificate
* start date: Apr 29 07:54:26 2022 GMT
* expire date: Apr 28 07:54:26 2024 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* issuer: DC=com; DC=XXXXXX; DC=test01global; CN=Elastic Certificate Tool Autogenerated CA
* SSL certificate verify ok.
} [5 bytes data]
> GET /api/security/role HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.78.0
> Accept: */*
> Content-Type: application/json;charset=UTF-8
> kbn-xsrf: true
> Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: a29bd265d8d8
< kbn-license-sig: a2b93ac49b41ba98f7fefe5376d8a057dea29560144ae157e0ff5e4c42f0d57b
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 17141
< vary: accept-encoding
< accept-ranges: bytes
< Date: Sat, 21 May 2022 12:39:20 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{ [15918 bytes data]
100 17141 100 17141 0 0 155k 0 --:--:-- --:--:-- --:--:-- 160kcations":[]},{"name":"watcher_admin","metadata":{"_reserved":true},"transient_metadata":{"enabled":true},"elasticsearch":{"cluster":["manage_watcher"],"indices":[{"names":[".watches",".triggered_watches",".watcher-history-*"],"privileges":["read"],"allow_restricted_indices":true}],"run_as":[]},"kibana":[],"_transform_error":[],"_unrecognized_applications":[]},{"name":"watcher_user","metadata":{"_reserved":true},"transient_metadata":{"enabled":true},"elasticsearch":{"cluster":["monitor_watcher"],"indices":[{"names":[".watches"],"privileges":["read"],"allow_restricted_indices":true},{"names":[".watcher-history-*"],"privileges":["read"],"allow_restricted_indices":false}],"run_as":[]},"kibana":[],"_transform_error":[],"_unrecognized_applications":[]}]
* Connection #0 to host localhost left intact
Connection To Elasticsearch:
curl -v --cacert "C:\Projects\Data\ELK\Beats\WinLogBeat\certs\ca.crt" --location --request GET 'https://localhost:9200/_security/_authenticate' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying ::1:9200...
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: C:\Projects\Data\ELK\Beats\WinLogBeat\certs\ca.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [32 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [877 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=es01
* start date: May 19 10:43:14 2022 GMT
* expire date: May 18 10:43:14 2025 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* issuer: CN=Elastic Certificate Tool Autogenerated CA
* SSL certificate verify ok.
} [5 bytes data]
> GET /_security/_authenticate HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.78.0
> Accept: */*
> Content-Type: application/json;charset=UTF-8
> kbn-xsrf: true
> Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [1168 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 266
<
{ [266 bytes data]
100 266 100 266 0 0 4019 0 --:--:-- --:--:-- --:--:-- 4222{"username":"elastic","roles":[],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"_es_api_key","type":"_es_api_key"},"lookup_realm":{"name":"_es_api_key","type":"_es_api_key"},"authentication_type":"api_key"}
* Connection #0 to host localhost left intact
curl -v --cacert "C:\Projects\Certificates\Signer\ca.crt" --location --request GET 'https://localhost:5601/api/status' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='
* Connected to localhost (::1) port 5601 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: C:\Projects\Certificates\Signer\ca.crt
* CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2018 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=XX; ST=XXXXXXXX; L=XXXXXX; O=XXXXXXXX; OU=GI; CN=ElasticCertificate_client
* start date: Apr 29 07:54:26 2022 GMT
* expire date: Apr 28 07:54:26 2024 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* issuer: DC=com; DC=XXXXXXX; DC=test01global; CN=Elastic Certificate Tool Autogenerated CA
* SSL certificate verify ok.
} [5 bytes data]
> GET /api/status HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.78.0
> Accept: */*
> Content-Type: application/json;charset=UTF-8
> kbn-xsrf: true
> Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: a29bd265d8d8
< kbn-license-sig: a2b93ac49b41ba98f7fefe5376d8a057dea29560144ae157e0ff5e4c42f0d57b
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 12533
< vary: accept-encoding
< accept-ranges: bytes
< Date: Sat, 21 May 2022 12:50:16 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{ [12533 bytes data]
100 12533 100 12533 0 0 203k 0 --:--:-- --:--:-- --:--:-- 214k1,"95":10.690559,"99":15.335423}},"uptime_in_millis":7601839.708579}],"response_times":{"avg_in_millis":341,"max_in_millis":460},"concurrent_connections":1,"requests":{"disconnects":0,"total":2,"statusCodes":{"200":2},"status_codes":{"200":2}}}}
* Connection #0 to host localhost left intact
Connection To Kibana API Status
> curl -v --cacert "C:\Projects\Certificates\Signer\ca.crt" --location --request GET 'https://localhost:5601/api/status' --header 'Content-Type: application/json;charset=UTF-8' --header 'kbn-xsrf: true' --header 'Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ=='
>
> * Connected to localhost (::1) port 5601 (#0)
>
> * ALPN, offering h2
>
> * ALPN, offering http/1.1
>
> * successfully set certificate verify locations:
>
> * CAfile: C:\Projects\Certificates\Signer\ca.crt
>
> * CApath: none
>
> } [5 bytes data]
>
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
>
> } [512 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
>
> { [122 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
>
> { [21 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
>
> { [2018 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
>
> { [264 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, Finished (20):
>
> { [52 bytes data]
>
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
>
> } [1 bytes data]
>
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
>
> } [52 bytes data]
>
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
>
> * ALPN, server accepted to use http/1.1
>
> * Server certificate:
>
> * subject: C=XX; ST=XXXXXXX; L=XXXXX; O=XXXXXXXX; OU=GI; CN=ElasticCertificate_client
>
> * start date: Apr 29 07:54:26 2022 GMT
>
> * expire date: Apr 28 07:54:26 2024 GMT
>
> * subjectAltName: host "localhost" matched cert's "localhost"
>
> * issuer: DC=com; DC=XXXXXXX; DC=testglobal; CN=Elastic Certificate Tool Autogenerated CA
>
> * SSL certificate verify ok.
>
> } [5 bytes data]
>
> > GET /api/status HTTP/1.1
>
> > Host: localhost:5601
>
> > User-Agent: curl/7.78.0
>
> > Accept: */*
>
> > Content-Type: application/json;charset=UTF-8
>
> > kbn-xsrf: true
>
> > Authorization: ApiKey TU9CSjVvQUJ4YlRRdnExRmJYem86ZUFWS3BmNXFTM0NQc0lYeXIwVEgtQQ==
>
> >
>
> { [5 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>
> { [265 bytes data]
>
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
>
> { [265 bytes data]
>
> * old SSL session ID is stale, removing
>
> { [5 bytes data]
>
> * Mark bundle as not supporting multiuse
>
> < HTTP/1.1 200 OK
>
> < x-content-type-options: nosniff
>
> < referrer-policy: no-referrer-when-downgrade
>
> < kbn-name: a29bd265d8d8
>
> < kbn-license-sig: a2b93ac49b41ba98f7fefe5376d8a057dea29560144ae157e0ff5e4c42f0d57b
>
> < content-type: application/json; charset=utf-8
>
> < cache-control: private, no-cache, no-store, must-revalidate
>
> < content-length: 12533
>
> < vary: accept-encoding
>
> < accept-ranges: bytes
>
> < Date: Sat, 21 May 2022 12:50:16 GMT
>
> < Connection: keep-alive
>
> < Keep-Alive: timeout=120
>
> <
>
> { [12533 bytes data]
>
> 100 12533 100 12533 0 0 203k 0 --:--:-- --:--:-- --:--:-- 214k1,"95":10.690559,"99":15.335423}},"uptime_in_millis":7601839.708579}],"response_times":{"avg_in_millis":341,"max_in_millis":460},"concurrent_connections":1,"requests":{"disconnects":0,"total":2,"statusCodes":{"200":2},"status_codes":{"200":2}}}}
>
> * Connection #0 to host localhost left intact
API KEY JSON
---------------------------API KEY Role Addition------------------------
POST /_security/api_key
{
"name": "ApiKey_Beats_User",
"expiration": "365d",
"role_descriptors": {
"beats_cluster_access" : {
"cluster" : [
"all",
"grant_api_key",
"manage",
"manage_api_key",
"manage_autoscaling",
"manage_ccr",
"manage_data_frame_transforms",
"delegate_pki",
"manage_enrich",
"manage_ilm",
"manage_index_templates",
"manage_ingest_pipelines",
"manage_logstash_pipelines",
"manage_ml",
"manage_security"
],
"indices" : [
{
"names" : [
"*"
],
"privileges" : [
"all",
"delete",
"create_index",
"create_doc",
"index",
"maintenance",
"manage_follow_index",
"manage_leader_index",
"manage",
"read",
"manage_ilm",
"auto_configure",
"create",
"monitor",
"view_index_metadata",
"delete_index",
"read_cross_cluster",
"write"
],
"allow_restricted_indices" : false
}
],
"applications" : [
{
"application" : "kibana-.kibana",
"privileges" : [
"all"
],
"resources" : [
"*"
]
}
],
"run_as" : [
"elastic"
],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
}