Elastic and kibana send resets on Ubuntu server

Elastic Stack Elasticsearch
1

I have elastic and kibana setup and running. I can curl to localhost:9200 as show below:

root@elk:~# curl http://127.0.0.1:9200
    {
      "name" : "elk",
      "cluster_name" : "elasticsearch",
      "cluster_uuid" : "EqHMzJiWT1-od9wkmcL8-w",
      "version" : {
    "number" : "7.3.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "de777fa",
    "build_date" : "2019-07-24T18:30:11.767338Z",
    "build_snapshot" : false,
    "lucene_version" : "8.1.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
      },
      "tagline" : "You Know, for Search"
    }




root@elk:~# wget localhost:5601
--2019-08-22 02:07:59--  http://localhost:5601/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5601... connected.
HTTP request sent, awaiting response... 302 Found
Location: /app/kibana [following]
--2019-08-22 02:07:59--  http://localhost:5601/app/kibana
Connecting to localhost (localhost)|127.0.0.1|:5601... connected.
HTTP request sent, awaiting response... 200 OK
Length: 72679 (71K) [text/html]
Saving to: ‘index.html.1’

index.html.1                          100%[=========================================================================>]  70.98K  --.-KB/s    in 0s

2019-08-22 02:08:00 (298 MB/s) - ‘index.html.1’ saved [72679/72679]

But when I try to connect externally, I get TCP resets:

oot@elk:~# tcpdump port 9200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
02:12:22.205554 IP 192.168.2.187.42340 > elk.9200: Flags [S], seq 1830289600, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:22.205610 IP elk.9200 > 192.168.2.187.42340: Flags [R.], seq 0, ack 1830289601, win 0, length 0
02:12:22.457587 IP 192.168.2.187.42341 > elk.9200: Flags [S], seq 4168675762, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:22.457606 IP elk.9200 > 192.168.2.187.42341: Flags [R.], seq 0, ack 4168675763, win 0, length 0
02:12:22.705841 IP 192.168.2.187.42340 > elk.9200: Flags [S], seq 1830289600, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:22.705883 IP elk.9200 > 192.168.2.187.42340: Flags [R.], seq 0, ack 1, win 0, length 0
02:12:22.958906 IP 192.168.2.187.42341 > elk.9200: Flags [S], seq 4168675762, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:22.958947 IP elk.9200 > 192.168.2.187.42341: Flags [R.], seq 0, ack 1, win 0, length 0
02:12:23.205999 IP 192.168.2.187.42340 > elk.9200: Flags [S], seq 1830289600, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:23.206040 IP elk.9200 > 192.168.2.187.42340: Flags [R.], seq 0, ack 1, win 0, length 0
02:12:23.459044 IP 192.168.2.187.42341 > elk.9200: Flags [S], seq 4168675762, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:23.459083 IP elk.9200 > 192.168.2.187.42341: Flags [R.], seq 0, ack 1, win 0, length 0
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel
root@elk:~# tcpdump port 5601
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
02:12:55.855770 IP 192.168.2.187.42349 > elk.5601: Flags [S], seq 190547619, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:55.855827 IP elk.5601 > 192.168.2.187.42349: Flags [R.], seq 0, ack 190547620, win 0, length 0
02:12:56.106356 IP 192.168.2.187.42350 > elk.5601: Flags [S], seq 2523034990, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:56.106394 IP elk.5601 > 192.168.2.187.42350: Flags [R.], seq 0, ack 2523034991, win 0, length 0
02:12:56.356717 IP 192.168.2.187.42349 > elk.5601: Flags [S], seq 190547619, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:56.356767 IP elk.5601 > 192.168.2.187.42349: Flags [R.], seq 0, ack 1, win 0, length 0
02:12:56.606771 IP 192.168.2.187.42350 > elk.5601: Flags [S], seq 2523034990, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:56.606801 IP elk.5601 > 192.168.2.187.42350: Flags [R.], seq 0, ack 1, win 0, length 0
02:12:56.857817 IP 192.168.2.187.42349 > elk.5601: Flags [S], seq 190547619, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:56.857871 IP elk.5601 > 192.168.2.187.42349: Flags [R.], seq 0, ack 1, win 0, length 0
02:12:57.107910 IP 192.168.2.187.42350 > elk.5601: Flags [S], seq 2523034990, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
02:12:57.107964 IP elk.5601 > 192.168.2.187.42350: Flags [R.], seq 0, ack 1, win 0, length

The firewall is off:

root@elk:~# ufw status verbose
Status: inactive

I can SSH to the server, so I know the network is fine.
Anyone have any ideas on this?

The OS is the latest Ubuntu server.

Thanks,

2

is elasticsearch configured to listen on anything else than localhost? Can you share the configuration file?

3

I have tried setting network.hos to the following.

 #network.host: 192.168.0.1
#
#network.host: 0.0.0.0
#network.host: 129.168.2.39
network.host: 127.0.0.1
# Set a custom port for HTTP:
#
#http.port: 9200
http.port: 9200

If I set it to anything other than 127.0.0.1, elastic search won't stay started and exits with either; status=1/failure or status=78/n/a

4

if you configure it on anything else than localhost, the bootstrap checks are actually run, please take a look at your logfiles and see what is preventing startup. The logs usually include link to description on how to fix the issue.

5

Which elasticsearch log should I be looking at?

Thanks

6

Check in the /var/log/elasticsearch directory

7

Yeah, I know the directory. But which file would actually give relevant information?

elasticsearch.log
gc.log
elasticsearch_audit.json
elasticsearch_deprecation.json
elasticsearch_deprecation.log
elasticsearch elasticsearch
elasticsearch_index_indexing_slowlog.json
elasticsearch_index_indexing_slowlog.log
elasticsearch_index_search_slowlog.json
elasticsearch_index_search_slowlog.log
elasticsearch.log
elasticsearch_server.json

Thanks,

8

the elasticsearch.log would be a first try.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.