Curator failed to connect to local Elasticsearch (on Ubuntu VM)

Elastic Stack Elasticsearch
1

I have Elasticsearch on Ubuntu VM.
I installed curator on the hosting windows server and succeeded to connect to elasticsearch, using the VM IP in the config file (and also succeeded to delete indices)
But when I installed Curator on the Ubuntu VM and try to connect using 127.0.0.1 or the VM IP, it fails.
I get "HTTP 503 error: ... A communication error occurred: "Operation timed out"... "

My curator config file is the initial ordinary config file from Configuration File | Curator Reference [5.8] | Elastic

My command line in the Terminal is:
$ sudo curator --config /curator/curator-config.yml /curator/actions.yml

I can connect to elasticsearch from filefox - when I type http://127.0.0.1:9200/ in the browser I do get elasticsearch JSON response

I can also connect to elastic from the terminal - when I type $ curl 127.0.0.1:9200 I do get elasticsearch JSON response

What might cause the connection failure?

2

Not sure how your VM is installed or configured, but it seems odd, indeed, that localhost is not working there. Have you tried pointing Curator to the VM IP from within the VM, instead of 127.0.0.1?

Also, getting a 503 here is a strange response. Curator only logs the responses it receives (it received a 503), so that is whatever Curator is connecting to saying, "we can't process something here, sorry we failed." In other words, if it were Curator's problem, it would be a 4xx error, meaning "Curator made a bad request." A 5xx error, like the 503 you received, implies that the server had a problem. Since you clearly can access the host and port from outside and inside using Curl, something else is going on here. I do not know how VMs networking is handled in Windows, but it certainly sounds strange.

If you set blacklist: [] and also set loglevel: debug in your configuration file, you will receive a lot more logging data. Perhaps something else will show up and help in debugging this.

3

Yes. It didn't work as well.

I got a very long log. Here is the interesting part:

INFO               curator.utils             get_client:1030 Testing client connectivity
2021-05-04 17:53:20,794 DEBUG     urllib3.connectionpool              _new_conn:227  Starting new HTTP connection (1): 15.90.78.11:8080
2021-05-04 17:53:20,841 DEBUG     urllib3.connectionpool          _make_request:452  http://15.90.78.11:8080 "GET http://127.0.0.1:9200/ HTTP/1.1" 301 None
2021-05-04 17:53:20,842 DEBUG     urllib3.connectionpool              _get_conn:272  Resetting dropped connection: 15.90.78.11
2021-05-04 17:54:36,557 DEBUG     urllib3.connectionpool          _make_request:452  http://15.90.78.11:8080 "GET http://15.17.232.53 HTTP/1.1" 503 751
2021-05-04 17:54:36,557 WARNING            elasticsearch       log_request_fail:285  GET http://127.0.0.1:9200/ [status:503 request:75.765s]
2021-05-04 17:54:36,558 DEBUG              elasticsearch       log_request_fail:302  > None
2021-05-04 17:54:36,558 DEBUG              elasticsearch       log_request_fail:307  < <HTML><HEAD>
<TITLE>Network Error</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Network Error (tcp_error)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
A communication error occurred: "Operation timed out"
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>

2021-05-04 17:54:36,558 WARNING            elasticsearch           _raise_error:320  Undecodable raw error response from server: Expecting value: line 1 column 1 (char 0)
2021-05-04 17:54:36,559 DEBUG     urllib3.connectionpool              _get_conn:272  Resetting dropped connection: 15.90.78.11
2021-05-04 17:54:36,575 DEBUG     urllib3.connectionpool          _make_request:452  http://15.90.78.11:8080 "GET http://127.0.0.1:9200/ HTTP/1.1" 301 None
2021-05-04 17:54:36,576 DEBUG     urllib3.connectionpool              _get_conn:272  Resetting dropped connection: 15.90.78.11

I don't understand why it is looking for 15.90.78.11. Is it a normal curator action or i should look for configuration of proxy of some type?

Thank you so much for your help

4

Do you have a system-level proxy configured? This may be an automatic behavior. Curator won't automatically query a proxy, so I presume this is enforced at the network level, either with the host OS, or the VM itself. It seems an extremely odd thing to do to make a localhost connection reach through a proxy. I do not know a way to work around this, as I do not use VMs in Windows.

5

Thanks to your help and some additional digging, I found the solution.
I will write it here, maybe it will help someone in the future:

My proxy configuration on the VM does point to 15.90.78.11
But the localhost and VM IP are in the NO_PROXY list:

NO_PROXY=15.17.232.53,localhost,127.0.0.1,127.0.0.0/8,::1
HTTPS_PROXY=http://15.90.78.11:8080/
HTTP_PROXY=http://15.90.78.11:8080/

So, I expected curator to ignore the proxy for 127.0.0.1 or the VM IP, but it didn't.
Since it is Ubuntu, I thought it might be related to case sensitivity, so I added another environment variable:

no_proxy=15.17.232.53,localhost,127.0.0.1,127.0.0.0/8,::1

(now I have both NO_PROXY and no_proxy)
And Curator is working as expected!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.