Kibana Dashboard is not getting updated with latest data

Elastic Stack Kibana
1

In our Kibana, dashboards are not getting displayed the latest data from Nov 04, 2022 and we could see below logs snippets at kibana.log.

Could someone please assist what is missing here.

{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["error","elasticsearch","admin"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:9000/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 127.0.0.1:9000"}
{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["error","savedobjects-service"],"pid":21545,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["status","plugin:xpack_main@7.8.0","error"],"pid":21545,"state":"red","message":"Status changed from green to red - Unable to retrieve version information from Elasticsearch nodes.","prevState":"green","prevMsg":"Ready"}

2

Logs are stating that Kibana can't connect to your Elasticsearch node. Have you checked this?

3

Hi @jsanz Jorge,

Thank you for your quick reply.
Both Kibaba and Elastic search are configured on same server only.

In Elasticsearch log also found below log snippets. But I hope this is an warning messages and not an issue. Please correct me if I am wrong.

[2021-12-15T12:17:46,489][WARN ][o.e.d.c.s.Settings ] [test-node-1] [discovery.zen.minimum_master_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-12-15T12:18:03,709][WARN ][o.e.d.a.b.BulkRequestParser] [test-node-1] [types removal] Specifying types in bulk requests is deprecated.
[2022-01-03T12:10:06,315][WARN ][o.e.d.r.a.a.i.RestGetMappingAction] [test-node-1] [types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version.

[2022-07-04T11:58:03,708][WARN ][o.e.d.i.m.IdFieldMapper ] [test-node-1] Loading the fielddata on the _id field is deprecated and will be removed in future versions. If you require sorting or aggregating on this field you should also include the id in the body of your documents, and map this field as a keyword field that has [doc_values] enabled
[2022-09-03T15:16:56,837][WARN ][o.e.d.c.s.Settings ] [test-node-1] [discovery.zen.minimum_master_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

4

If you can query Elasticsearch to confirm it is running correctly (for example querying the _cat/health and _cat/nodes endpoints) then you need to carefully review your Kibana settings to confirm the connection with Elasticsearch is correctly configured.

5

Hi @jsanz ,

Seems you are correct and I could see below log snippets in kibana.log

{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["error","elasticsearch","admin"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:8800/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => read ECONNRESET"}
{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["error","elasticsearch","admin"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:8900/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 127.0.0.1:8900"}
{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["error","elasticsearch","admin"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:9000/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 127.0.0.1:9000"}
{"type":"log","@timestamp":"2022-11-07T12:28:24Z","tags":["error","savedobjects-service"],"pid":21545,"message":"Unable to retrieve version information from Elasticsearch nodes."}

{"type":"log","@timestamp":"2022-11-07T12:28:40Z","tags":["error","elasticsearch","monitoring"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:8800/_xpack => connect ECONNREFUSED 127.0.0.1:8800"}
{"type":"log","@timestamp":"2022-11-07T12:28:40Z","tags":["error","elasticsearch","monitoring"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:9000/_xpack => connect ECONNREFUSED 127.0.0.1:9000"}
{"type":"log","@timestamp":"2022-11-07T12:28:40Z","tags":["error","elasticsearch","monitoring"],"pid":21545,"message":"Request error, retrying\nGET http://localhost:8900/_xpack => connect ECONNREFUSED 127.0.0.1:8900"}
{"type":"log","@timestamp":"2022-11-07T12:28:42Z","tags":["error","plugins","taskManager","taskManager"],"pid":21545,"message":"Failed to poll for work: Error: No Living connections"}
{"type":"log","@timestamp":"2022-11-07T12:28:45Z","tags":["error","plugins","taskManager","taskManager"],"pid":21545,"message":"Failed to poll for work: Error: No Living connections"}
{"type":"log","@timestamp":"2022-11-07T12:28:48Z","tags":["error","plugins","taskManager","taskManager"],"pid":21545,"message":"Failed to poll for work: Error: No Living connections"}

6

Hi @jsanz ,

While trying to check health check command , getting below. Please assist me how to resolve it.

curl localhost:8800/_cat/health

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}[root@esesslx1045 pucaoacnt]#

7

You typically need to pass user/password because your Elasticsearch is secured. So something like this:

curl -a [USER]:[PASSWORD_HERE] http://localhost:8800/_cat/health

Also I see that in your Kibana logs it's attempting to connect to different hosts (localhost:8800, localhost:9000, localhost:8900). What is your kibana configuration?

8

Hello @jsanz ,

Below is the Kibana.yml and elasticsearch.yml configurations.

kibana.yml

server.host: "HOSTNAME"
server.basePath: "/kibana-prod"
server.rewriteBasePath: true
elasticsearch.hosts: ["http://localhost:8800", "http://localhost:8900", "http://localhost:9000"]
path.data: /PATH/kibana/1/data
logging.dest: /PATH/1/log/kibana.log
logging.rotate.enabled: true
logging.quiet: true

elasticsearch.username: "USERNAME"
elasticsearch.password: "PASSWORD"

elasticsearch.yml

cluster.name: CLUSTERNAME
node.name: prod-node-1
cluster.initial_master_nodes:
- prod-node-1
- prod-node-2
- prod-node-3
discovery.zen.minimum_master_nodes: 2
node.master: true
node.data: true
path.data: /PATH/elasticsearch/1/lib
path.logs: /PATH/elasticsearch/1/logs
http.port: 8800
bootstrap.memory_lock: true

xpack.security.enabled: true
xpack.security.authc.realms:
native.native1:
order: 0

And as per above conversation, seems Elasticsearch running fine on instance.
curl -X GET "http://localhost:8800" -u USERNAME:PASSWORD
{
"name" : "test-node-1",
"cluster_name" : "CLUSTERNAME",
"cluster_uuid" : "CLUSTERUID",
"version" : {
"number" : "7.8.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "HASHCODE",
"build_date" : "2020-06-14T19:35:50.234439Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Could you please assist me what I am missing.

Thanks in advance.

9

Do you have three different instances of Elasticsearch (prod-node-[1|2|3]) running all on the same host? Are they all running correctly or it is just the one on 8800?

10

Hello @jsanz

Yeah, 3 are running on same instance and those are running fine. I have checked the processes and looks good.

Those 3 instances are running on these 3 ports. Even we killed process for o e of the instance and restarted that.

11

So if you run _cat/nodes on any of them they all list all the three nodes and your Elasticsearch cluster is healthy right?

I'd try to connect Kibana to just one of them (I guess the master node) to see if it works and then add the other two.

I'm not an expert in Elasticsearch and Kibana operations, just throwing to you some questions to try to corner the problem.

12

Hi @jsanz

Thank you for your continuous support on this issue.

While I am trying to curl on any instance (3 ports), getting Unable to authenticate user issue. Please find below output.

]# curl -v http://localhost:8800/ -u USERNAME:PASSWORD

  • About to connect() to localhost port 8800 (#0)
  • Trying 127.0.0.1...
  • Connected to localhost (127.0.0.1) port 8800 (#0)
  • Server auth using Basic with user 'elastic'

GET / HTTP/1.1
Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXX
User-Agent: curl/7.29.0
Host: localhost:8800
Accept: /

< HTTP/1.1 401 Unauthorized

  • Authentication problem. Ignoring this.
    < WWW-Authenticate: Basic realm="security" charset="UTF-8"
    < content-type: application/json; charset=UTF-8
    < content-length: 387
    <
  • Connection #0 to host localhost left intact
    {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}
13

In a previous comment you showed that the node on 8880 was responding correctly.

I think you need to first ensure your nodes are all running and in the same cluster so you can query _cat/nodes and _cat/health to them and get valid responses and then try first to connect your Kibana instance just against the master node and only when that works, expand the hosts param to include the rest of the nodes of your cluster (if that is what you want).

14

Hi @jsanz

The info what I was provided earlier in UAT and latest log snippet from production.

I have checked both _cat/nodes and _cat/health and getting same for all elastic nodes.

$ curl -v http://localhost:8800/_cat/nodes -u USERNAME:PASSWORD

  • About to connect() to localhost port 8800 (#0)
  • Trying 127.0.0.1...
  • Connected to localhost (127.0.0.1) port 8800 (#0)
  • Server auth using Basic with user 'USERNAME'

GET /_cat/nodes HTTP/1.1
Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXX
User-Agent: curl/7.29.0
Host: localhost:8800
Accept: /

< HTTP/1.1 401 Unauthorized

  • Authentication problem. Ignoring this.
    < WWW-Authenticate: Basic realm="security" charset="UTF-8"
    < content-type: application/json; charset=UTF-8
    < content-length: 407
    <
  • Connection #0 to host localhost left intact
    {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_cat/nodes]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [USERNAME] for REST request [/_cat/nodes]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

$ curl -v http://localhost:8800/_cat/health -u USERNAME:PASSWORD

  • About to connect() to localhost port 8800 (#0)
  • Trying 127.0.0.1...
  • Connected to localhost (127.0.0.1) port 8800 (#0)
  • Server auth using Basic with user 'USERNAME'

GET /_cat/health HTTP/1.1
Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXX
User-Agent: curl/7.29.0
Host: localhost:8800
Accept: /

< HTTP/1.1 401 Unauthorized

  • Authentication problem. Ignoring this.
    < WWW-Authenticate: Basic realm="security" charset="UTF-8"
    < content-type: application/json; charset=UTF-8
    < content-length: 409
    <
  • Connection #0 to host localhost left intact
    {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [USERNAME] for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

I have checked the Elasticsearch process's on server and those are running fine.

15

Are you redacting the output of your nodes or USERNAME is the actual name of your user? :thinking:

16

Hello @jsanz

I am changing the actual username to USERNAME-:slight_smile:

17

haha ok

then please review the security settings because it seems your user is not correctly configured, if you are getting those authentication errors.

18

Hi @jsanz

Thank you for your guidance.

It was working fine until Nov 04, 2022 and getting Dashboards. Dashboards are stopped updating with required data after Nov 05, 2022.

As you suggested, I will review security settings on server.

19

Hello @jsanz

I got stuck on what to check. Do I need to change the elastic user password?

20

You shouldn't have to change your password.