Kibana throwing error "{"statusCode":503,"error":"Service Unavailable","message":"Service Unavailable"}"

Elastic Stack Kibana
#1

Hi Team,
Im using kibana version 6.4.0 and when ever i try to save or create new index im getting 503 error.

All the elasticsearch service are up and running.

My setup is 3 Master and 3 Data and 1 coordinate Node.

My ES service status is Green.

Could any help me on this.

#2

Hi

Could you provide some details? How do you create this new indices in Kibana, in DevTools? Is there data in Elasticsearch and you've got an index pattern to query it e.g. in Discover?

Thx & Best,
Matthias

#3

@matw Thanks for the quick response,

i used to create new index via Management window. When i try to save new search as well getting same error.

we have 2 kibana server and both working via coordinate nodes

#4

so you can query, but you cannot write into ES, or does both fail?

#5

yes i can query the condition, but when i try to save the saved search or try to create index pattern im getting error in it.

#6

I see, are there any error messages in you Elasticsearch log? is there a proxy between Kibana and Elasticsearch

#7

There is no proxy between ES and kibana.Could you be specific on which log do you need coordinate node es log or data node or master node

#8

I would search for error messages in all of them, could you record and share network activities of your browser when this error happens by a HAR file? thx!

#9

Please find my network activity and message which get write on coordinate node log,

but both kibana and elasticsearch are running on both kibana server
message log:

[2021-02-10T10:02:27,899][DEBUG][o.e.a.a.c.n.i.TransportNodesInfoAction] [kibanapr23] failed to execute on node [ksKrcxEUQFuQSu6LntOmsw]
org.elasticsearch.transport.NodeNotConnectedException: [kibanapr23][10.xx.xxx.xx2:9300] Node not connected

Request Header:

Key Value
Request POST /api/saved_objects/search?overwrite=true HTTP/1.1
Accept /
Content-Type application/json
kbn-version 6.4.0
Referer https://kibana02.net/app/kibana
Accept-Language en-IE
Accept-Encoding gzip, deflate
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host kibana02.net
Content-Length 336
Connection Keep-Alive
Cache-Control no-cache
Cookie _ga=GA1.2.1210917231.1599139195; JSESSIONID=srv-FOu/9ZeeA1FxwGd1ldHy3g

Request Body:

{"attributes":{"title":"New Saved Search","description":"","hits":0,"columns":["_source"],"sort":["@timestamp","desc"],"version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{"index":"9288dde0-6043-11ea-9dba-3353da487509","highlightAll":true,"version":true,"query":{"query":"","language":"lucene"},"filter":}"}}}

Response Header:

Key Value
Transfer-Encoding chunked
Set-Cookie JSESSIONID=srv-FOu/9ZeeA1FxwGd1ldHy3g
set-cookie sid=Fe26.2ee37b0cc0e78679b91913667d9a79f96a7b96934a459d288caf7dc0afcf0373a4v4lYpdol5rXt_mFWN3ehAV_ZedWLX2VVfwR9786_jZg8bxFmucZTQViRQ-xCTZTbXqpzqg6zj8vOu8qr-P4xV0C3x8MpKUMOmRzfZEefuyuJ1kXRaMmXXpgo-8d1qE8cnVH2JEzQdNUHOPWMG3RTEPLyx0LW8P13-sUcJfkOO_w8915524fe39ac793e130758bad5aa799e0963939237e417ec16aa2b50d086598*NTjuzCgo31aRQXi465rr6h4KqB0gDP-VaKkwSLZIXLY; Secure; HttpOnly; Path=/
Response HTTP/1.1 503 Service Unavailable
kbn-xpack-sig 527979ca47a97a6d9e917f5b850e5932
kbn-name kibana
Date Wed, 10 Feb 2021 08:07:31 GMT
content-type application/json; charset=utf-8
cache-control no-cache

Response body:

{"statusCode":503,"error":"Service Unavailable","message":"Service Unavailable"}

Cookies:

Direction Key Value Expires Domain Path Secure HTTP only
Sent JSESSIONID srv-FOu/9ZeeA1FxwGd1ldHy3g YCOUC
Sent sid Fe26.2ba9cafe8f7fdd6603b67a3c88778c4d984e27cd2299564ee9c9197fb13282590mbCfMt93XOUzU93ezbF6pQNO0A1Q2psO7OjQKgIRx8rgeXk9DOgVJqKa-k0PnfrXAumuNU6m3smhkPINf7PwlR5ra8FwKgtxUFx-2ma6Rx7GA_6JROq_eTqxbtQXlRwfaRJrFMshMYS-yY__iq0NROnX88zWGvsCHXVa7cDC6jQgc9fad4be901d7d65637f950c545217e70a534ae5d1021e7e0812d93521a6f474*Ox9K6gxj8J96nkTB0TRR1NurXrQCz2R8bjVY4G3E4cw
Received sid Fe26.2ee37b0cc0e78679b91913667d9a79f96a7b96934a459d288caf7dc0afcf0373a4v4lYpdol5rXt_mFWN3ehAV_ZedWLX2VVfwR9786_jZg8bxFmucZTQViRQ-xCTZTbXqpzqg6zj8vOu8qr-P4xV0C3x8MpKUMOmRzfZEefuyuJ1kXRaMmXXpgo-8d1qE8cnVH2JEzQdNUHOPWMG3RTEPLyx0LW8P13-sUcJfkOO_w8915524fe39ac793e130758bad5aa799e0963939237e417ec16aa2b50d086598*NTjuzCgo31aRQXi465rr6h4KqB0gDP-VaKkwSLZIXLY At end of session / Yes Yes
Received JSESSIONID srv-FOu/9ZeeA1FxwGd1ldHy3g YCOUK At end of session / No

Timing:

Name Offset Duration Key
Wait‎‎ 0 ms 126.23 s
Start‎‎ + 126.23 s 47 ms
Start‎‎ + 126.28 s 359 ms
Request‎‎ + 126.64 s 30.03 s
Response‎‎ + 156.67 s < 1 ms
Gap‎‎ + 156.67 s < 1 ms
DOMContentLoaded (event)‎‎ + 8.77 s -
Load (event)‎‎ + 14.51 s -
#10

@matw any help with that above information