Find out which process sends HTTP requests

Elastic Stack Elasticsearch
1

Hi!

I've enabled security on my cluster and more or less everything went as expected, except for one node (out of 10) where I get a long warning message every 5 minutes:

[2019-09-11T11:50:02,010][WARN ][o.e.h.AbstractHttpServerTransport] [elastic.local.lan] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/10.50.90.5:60000}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

I can't find out what causes this.
The node has Filebeat, Metricbeat, Logstash installed but none of those have anything in their logs. Any ideas how could I identify the culprit?

Thank you!
YvorL

2

  1. Go to 10.50.90.5 and run

     sudo lsof -i -P -n | grep 60000

  2. The message you shared has a long hex string after that. This is the actual incoming HTTP request. Run

    echo <the_long_hex_string> | xxd -r -p

    and the result will be helpful, as the user agent or some other request header will probably tell you what is sending the request.

3

First of all, huge thanks! :slight_smile:

Unfortunately, the grep wouldn't work because the requests always used differents ports. However, decoding the message lead to this:

PUT /_all/_settings HTTP/1.1
Host: 10.50.90.5:9200
Authorization: Basic XXXXXXXXXXXXXXXXX
User-Agent: curl/7.47.0
Accept: /
Content-Length: 38
Content-Type: application/x-www-form-urlencoded
{
"index.number_of_replicas" : "0"
}

Is this something inside ES or this is a custom request?

4

This is something running curl trying to update the settings for all indices . Some cronjob someone is running ?

If you base64 decode XXXXXXXXXXXXXXXXX, maybe the username will give you a hint ?

5

Thanks again!

Have a nice day!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.