Trouble with dashboard import in ECE cluster

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi,

I have an ECE deployment including one cluster that seems to be working fine. I have used APIs to interact with the kibana instance for example using curl to upload an index template.

I tried to import a saved dashboard using curl, which gave me an error :

504 Gateway Time-out

504 Gateway Time-out

The curl syntax has worked elsewhere but I tried doing the same from the Kibana UI in person and also got timeout behaviour.

I tried splitting the dashboard file into its component parts and uploading them one at a time, but even a small file gives the same timeout behaviour.

Could there be a reason the dashboard import is not working for me? How can I go about diagnosing what sounds like a load balancer / reverse proxy issue within ECE?

Any ideas?

2

The dashboard I want to import was a .json file so I had to rename it to .ndjson. Looking at a newly exported dashboard, the file content is different to the original file I wanted to import.

I've just realised I'm possibly importing a 6.x version dashboard into a 7.x cluster - if this is a problem should I expect a validation error or something slightly cryptic like a timeout?

3

That is a strange error ... do you get a body back with the 504 (Eg do curl -v)

The other thing you could try is to go to the allocator running the kibana instance, use docker ps to get the Kibana HTTPS port and curl -k directly into the port and see what you get back. That would isolate it away from being any of the ECE components vs Kibana itself

4

Alex,

thanks I was poking around on the allocator with docker looking for the server side logs of Kibana... I have tried the same curl command and this is what I found:

# curl -v -k -X POST -u elastic:mypwd https://localhost:18391/api/saved_objects/_import?overwrite=true -H 'kbn-xsrf: true' --form file=@file.ndjson
* About to connect() to localhost port 18391 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 18391 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=instance-0000000000.node.xxxmynode.cluster.local
*       start date: Nov 08 10:30:52 2019 GMT
*       expire date: Nov 05 10:30:52 2029 GMT
*       common name: instance-0000000000.node.xxxmynode.cluster.local
*       issuer: CN=elastic ce internal tls root
* Server auth using Basic with user 'elastic'
> POST /api/saved_objects/_import?overwrite=true HTTP/1.1
> Authorization: Basic xxxmytoken
> User-Agent: curl/7.29.0
> Host: localhost:18391
> Accept: */*
> kbn-xsrf: true
> Content-Length: 127121
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------74301f4a9db5
>
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

Basically there is a huge wait after the server sends 100 Continue. I have searched around this and there seems to be something about http/1.1 but I'm none the wiser.

Note I had to use the -k (insecure) option for curl with this local URL but the internet facing endpoint works without -k, giving the exact same behaviour

5

Try adding this to curl: -H 'Expect:' to disable the 100 Continue code

(If Kibana had the same problem then it's unlikely to help, and will just prove the issue is with Kibana serving the request itself)

If I had to guess based on the info so far, it would be that the formats are not compatible and that Kibana is not responding in the most useful way.

Once you can get a single API call direct to Kibana that fails with a timeout, it's probably worth taking this to the Kibana forum where they might have seen this before? (or direct you to open an issue)

6

Alex,

I had the same problem with the empty Expect header and -0, without the 100 Continue being sent. I'm not convinced that curl is sending the payload for Kibana to judge it, based on the curl -v output. I'll take it to the Kibana forum.

Thanks again

7

Forward link to the Kibana forum for completeness: Trouble importing dashboard

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.