Kibana throwing error "{"statusCode":503,"error":"Service Unavailable","message":"Service Unavailable"}"

[Ganesh2303 (Ganesh)]

Hi Team,
Im using kibana version 6.4.0 and when ever i try to save or create new index im getting 503 error.

All the elasticsearch service are up and running.

My setup is 3 Master and 3 Data and 1 coordinate Node.

My ES service status is Green.

Could any help me on this.

[matw (Matthias Wilhelm)]

Hi

Could you provide some details? How do you create this new indices in Kibana, in DevTools? Is there data in Elasticsearch and you've got an index pattern to query it e.g. in Discover?

Thx & Best,
Matthias

[Ganesh2303 (Ganesh)]

@matw Thanks for the quick response,

i used to create new index via Management window. When i try to save new search as well getting same error.

we have 2 kibana server and both working via coordinate nodes

[matw (Matthias Wilhelm)]

so you can query, but you cannot write into ES, or does both fail?

[Ganesh2303 (Ganesh)]

yes i can query the condition, but when i try to save the saved search or try to create index pattern im getting error in it.

[matw (Matthias Wilhelm)]

I see, are there any error messages in you Elasticsearch log? is there a proxy between Kibana and Elasticsearch

[Ganesh2303 (Ganesh)]

There is no proxy between ES and kibana.Could you be specific on which log do you need coordinate node es log or data node or master node

[matw (Matthias Wilhelm)]

I would search for error messages in all of them, could you record and share network activities of your browser when this error happens by a HAR file? thx!

[Ganesh2303 (Ganesh)]

Please find my network activity and message which get write on coordinate node log,

but both kibana and elasticsearch are running on both kibana server
message log:

[2021-02-10T10:02:27,899][DEBUG][o.e.a.a.c.n.i.TransportNodesInfoAction] [kibanapr23] failed to execute on node [ksKrcxEUQFuQSu6LntOmsw]
org.elasticsearch.transport.NodeNotConnectedException: [kibanapr23][10.xx.xxx.xx2:9300] Node not connected

Request Header:

Key	Value
Request	POST /api/saved_objects/search?overwrite=true HTTP/1.1
Accept	/
Content-Type	application/json
kbn-version	6.4.0
Referer	https://kibana02.net/app/kibana
Accept-Language	en-IE
Accept-Encoding	gzip, deflate
User-Agent	Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host	kibana02.net
Content-Length	336
Connection	Keep-Alive
Cache-Control	no-cache
Cookie	_ga=GA1.2.1210917231.1599139195; JSESSIONID=srv-FOu/9ZeeA1FxwGd1ldHy3g

Request Body:

{"attributes":{"title":"New Saved Search","description":"","hits":0,"columns":["_source"],"sort":["@timestamp","desc"],"version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{"index":"9288dde0-6043-11ea-9dba-3353da487509","highlightAll":true,"version":true,"query":{"query":"","language":"lucene"},"filter":}"}}}

Response Header:

Key	Value
Transfer-Encoding	chunked
Set-Cookie	JSESSIONID=srv-FOu/9ZeeA1FxwGd1ldHy3g
set-cookie	sid=Fe26.2ee37b0cc0e78679b91913667d9a79f96a7b96934a459d288caf7dc0afcf0373a4v4lYpdol5rXt_mFWN3ehAV_ZedWLX2VVfwR9786_jZg8bxFmucZTQViRQ-xCTZTbXqpzqg6zj8vOu8qr-P4xV0C3x8MpKUMOmRzfZEefuyuJ1kXRaMmXXpgo-8d1qE8cnVH2JEzQdNUHOPWMG3RTEPLyx0LW8P13-sUcJfkOO_w8915524fe39ac793e130758bad5aa799e0963939237e417ec16aa2b50d086598*NTjuzCgo31aRQXi465rr6h4KqB0gDP-VaKkwSLZIXLY; Secure; HttpOnly; Path=/
Response	HTTP/1.1 503 Service Unavailable
kbn-xpack-sig	527979ca47a97a6d9e917f5b850e5932
kbn-name	kibana
Date	Wed, 10 Feb 2021 08:07:31 GMT
content-type	application/json; charset=utf-8
cache-control	no-cache

Response body:

{"statusCode":503,"error":"Service Unavailable","message":"Service Unavailable"}

Cookies:

Direction	Key	Value	Expires	Domain	Path	Secure	HTTP only
Sent	JSESSIONID	srv-FOu/9ZeeA1FxwGd1ldHy3g	YCOUC
Sent	sid	Fe26.2ba9cafe8f7fdd6603b67a3c88778c4d984e27cd2299564ee9c9197fb13282590mbCfMt93XOUzU93ezbF6pQNO0A1Q2psO7OjQKgIRx8rgeXk9DOgVJqKa-k0PnfrXAumuNU6m3smhkPINf7PwlR5ra8FwKgtxUFx-2ma6Rx7GA_6JROq_eTqxbtQXlRwfaRJrFMshMYS-yY__iq0NROnX88zWGvsCHXVa7cDC6jQgc9fad4be901d7d65637f950c545217e70a534ae5d1021e7e0812d93521a6f474*Ox9K6gxj8J96nkTB0TRR1NurXrQCz2R8bjVY4G3E4cw
Received	sid	Fe26.2ee37b0cc0e78679b91913667d9a79f96a7b96934a459d288caf7dc0afcf0373a4v4lYpdol5rXt_mFWN3ehAV_ZedWLX2VVfwR9786_jZg8bxFmucZTQViRQ-xCTZTbXqpzqg6zj8vOu8qr-P4xV0C3x8MpKUMOmRzfZEefuyuJ1kXRaMmXXpgo-8d1qE8cnVH2JEzQdNUHOPWMG3RTEPLyx0LW8P13-sUcJfkOO_w8915524fe39ac793e130758bad5aa799e0963939237e417ec16aa2b50d086598*NTjuzCgo31aRQXi465rr6h4KqB0gDP-VaKkwSLZIXLY	At end of session		/	Yes	Yes
Received	JSESSIONID	srv-FOu/9ZeeA1FxwGd1ldHy3g	YCOUK	At end of session		/	No

Timing:

Name	Offset	Duration	Key
Wait‎‎	0 ms	126.23 s
Start‎‎	+ 126.23 s	47 ms
Start‎‎	+ 126.28 s	359 ms
Request‎‎	+ 126.64 s	30.03 s
Response‎‎	+ 156.67 s	< 1 ms
Gap‎‎	+ 156.67 s	< 1 ms
DOMContentLoaded (event)‎‎	+ 8.77 s	-
Load (event)‎‎	+ 14.51 s	-

[Ganesh2303 (Ganesh)]

@matw any help with that above information