Installation does not bind to 9200

[rhalbheer (Roger Halbheer)]

I am really sorry but I am completely lost. I spent more than a day now to get Elastiscsearch running on an RPI but failed miserably. Goal is to have the whole ELK stack there at the end.

• List item I installed a plain vanilla Rasbarian

• List item Connected it to my WLAN with a fixed IP

• List item Ran apt-get update and apt-get upgrade

• List item Ran apt-get install elasticsearch

so far everything works fine. But no matter what I do in the config file, the service runs but does not bind to port 9200, nor do I have any logs in /var/log/elasticserach.

Where can I look for the reason of this? I am completely lost and I am aware that this is most probably just a newbie error

Roger

[jasontedor]

I'm sorry for the trouble you're having. Let me see if we can help. Can you share the config file? How are you starting Elasticsearch? How do you know the service is running?

[warkolm (Mark Walkom)]

What happens when you start the service, can you show the output?

[rhalbheer (Roger Halbheer)]

No problem. I guess, the issue is with me, not ES.....
Let's start with the config file. I changed only this section in /etc/elasticsearch/elasticsearch.yml:

# Set both 'bind_host' and 'publish_host':
#
network.host: _global_

# Set a custom port for the node to node communication (9300 by default):
#
#transport.tcp.port: 9300

# Enable compression for all communication between nodes (disabled by default):
#
#transport.tcp.compress: true

# Set a custom port to listen for HTTP traffic:
#
http.port: 9200

# Set a custom allowed content length:
#
#http.max_content_length: 100mb

# Disable HTTP completely:
#
#http.enabled: false

On network.host, I kind of tried everything, including the internal IP address, localhost, global, site etc.

netstat -l -n shows this:

pi@ELK:/var/log/elasticsearch $ netstat -l -n
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
udp        0      0 0.0.0.0:5353            0.0.0.0:*                          
udp        0      0 0.0.0.0:5353            0.0.0.0:*                          
udp        0      0 0.0.0.0:35845           0.0.0.0:*                          
udp        0      0 0.0.0.0:68              0.0.0.0:*                          
udp6       0      0 :::5353                 :::*                               
udp6       0      0 :::5353                 :::*                               
udp6       0      0 :::43272                :::*                               
raw6       0      0 :::58                   :::*                    7          
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     5389     /run/systemd/fsck.progress
unix  2      [ ACC ]     STREAM     LISTENING     5394     /run/systemd/journal/stdout
unix  2      [ ACC ]     STREAM     LISTENING     13089    /run/user/1000/.lxterminal-socket-:0.0
unix  2      [ ACC ]     STREAM     LISTENING     10102    @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     12489    /tmp/ssh-G0q91MU1kbwx/agent.714
unix  2      [ ACC ]     SEQPACKET  LISTENING     7242     /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     9547     /var/run/avahi-daemon/socket
unix  2      [ ACC ]     STREAM     LISTENING     9550     /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     9553     /run/thd.socket
unix  2      [ ACC ]     STREAM     LISTENING     6754     /var/run/dhcpcd.sock
unix  2      [ ACC ]     STREAM     LISTENING     6756     /var/run/dhcpcd.unpriv.sock
unix  2      [ ACC ]     STREAM     LISTENING     10103    /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     15023    /tmp/.org.chromium.Chromium.ej2Aqt/SingletonSocket
unix  2      [ ACC ]     STREAM     LISTENING     12371    /tmp/ssh-ruIHOUGeSc92/agent.577
unix  2      [ ACC ]     STREAM     LISTENING     11159    /run/user/1000/menu-cached-:0
unix  2      [ ACC ]     STREAM     LISTENING     11423    /run/user/1000/pcmanfm-socket--0
unix  2      [ ACC ]     STREAM     LISTENING     7125     /run/user/1000/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     7130     /run/user/1000/gnupg/S.gpg-agent.extra
unix  2      [ ACC ]     STREAM     LISTENING     7133     /run/user/1000/bus
unix  2      [ ACC ]     STREAM     LISTENING     7135     /run/user/1000/gnupg/S.gpg-agent.ssh
unix  2      [ ACC ]     STREAM     LISTENING     7137     /run/user/1000/gnupg/S.gpg-agent.browser
unix  2      [ ACC ]     STREAM     LISTENING     7139     /run/user/1000/gnupg/S.gpg-agent
unix  2      [ ACC ]     STREAM     LISTENING     5371     /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     8190     /var/run/bluealsa/hci0

When I start the service, I kind of see nothing:

pi@ELK:/var/log/elasticsearch $ service elasticsearch start
pi@ELK:/var/log/elasticsearch $ 

and this is the status:

pi@ELK:/var/log/elasticsearch $ service elasticsearch status
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch; generated; vendor preset: enabled)
   Active: active (exited) since Thu 2017-09-21 08:16:05 CEST; 27s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 3850 ExecStop=/etc/init.d/elasticsearch stop (code=exited, status=0/S
  Process: 3921 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0

Sep 21 08:16:05 ELK systemd[1]: Starting LSB: Starts elasticsearch...
Sep 21 08:16:05 ELK systemd[1]: Started LSB: Starts elasticsearch.
lines 1-9/9 (END)

Does this help somehow?
Thank you
Roger

[warkolm (Mark Walkom)]

It's unusual that Elasticsearch would start and not put something in the logs.
Is there anything in the OS level logs, like /var/log/system.log or similar?

[rhalbheer (Roger Halbheer)]

I cleared the syslog and did

pi@ELK:/var/log $ service elasticsearch stop
pi@ELK:/var/log $ service elasticsearch start
pi@ELK:/var/log $ service elasticsearch status
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch; generated; vendor preset: enabled
   Active: active (exited) since Fri 2017-09-22 07:32:49 CEST; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6915 ExecStop=/etc/init.d/elasticsearch stop (code=exited, status=0/
  Process: 6985 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=

Sep 22 07:32:49 ELK systemd[1]: Starting LSB: Starts elasticsearch...
Sep 22 07:32:49 ELK systemd[1]: Started LSB: Starts elasticsearch.
lines 1-9/9 (END)

what I now have in syslog is

Sep 22 07:17:02 ELK CRON[6695]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 22 07:32:43 ELK systemd[1]: Stopping LSB: Starts elasticsearch...
Sep 22 07:32:43 ELK systemd[1]: Stopped LSB: Starts elasticsearch.
Sep 22 07:32:49 ELK systemd[1]: Starting LSB: Starts elasticsearch...
Sep 22 07:32:49 ELK systemd[1]: Started LSB: Starts elasticsearch.

which to me makes sense. /var/log/elasticsearch is empty (no files in there).
Looking at other logs, I see the installation in the auth.log:

Sep 20 16:23:07 ELK sudo:       pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/usr/bin/apt-get install elasticsearch
Sep 20 16:23:07 ELK sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 20 16:24:24 ELK groupadd[18792]: group added to /etc/group: name=elasticsearch, GID=116
Sep 20 16:24:24 ELK groupadd[18792]: group added to /etc/gshadow: name=elasticsearch
Sep 20 16:24:24 ELK groupadd[18792]: new group: name=elasticsearch, GID=116
Sep 20 16:24:25 ELK useradd[18799]: new user: name=elasticsearch, UID=111, GID=116, home=/var/lib/elasticsearch, shell=/bin/false
Sep 20 16:24:26 ELK usermod[18808]: change user 'elasticsearch' password
Sep 20 16:24:26 ELK chage[18814]: changed password expiry for elasticsearch
Sep 20 16:24:39 ELK sudo: pam_unix(sudo:session): session closed for user root
Sep 20 16:30:40 ELK sudo:       pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/usr/sbin/update-rc.d elasticsearch defaults 95 10
Sep 20 16:30:40 ELK sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 20 16:30:40 ELK sudo: pam_unix(sudo:session): session closed for user root
Sep 20 16:31:32 ELK sudo:       pi : TTY=pts/0 ; PWD=/etc/elasticsearch ; USER=root ; COMMAND=/bin/nano elasticsearch.yml
Sep 20 16:31:32 ELK sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 20 16:31:57 ELK sudo: pam_unix(sudo:session): session closed for user root

same in deamon.log and kern.log

Sep 22 07:32:43 ELK systemd[1]: Stopping LSB: Starts elasticsearch...
Sep 22 07:32:43 ELK systemd[1]: Stopped LSB: Starts elasticsearch.
Sep 22 07:32:49 ELK systemd[1]: Starting LSB: Starts elasticsearch...
Sep 22 07:32:49 ELK systemd[1]: Started LSB: Starts elasticsearch.

So, I am really lost.....

[jasontedor]

Have you set the JVM heap size appropriate for the Pi? The default is 2g and I suspect that's way too high for your Pi and you're running out of memory on startup? Maybe you could look at jvm.options and set the heap size appropriately? You should set it to half the available RAM on the Pi.

[rhalbheer (Roger Halbheer)]

No I did not. Good point. And we might be closer to my problem, I guess. I do not have any jvm.options file in /etc/elasticsearch

pi@ELK:/etc/elasticsearch $ ls
elasticsearch.yml  logging.yml

This is all I have. Should I already have one or do I have to create it? I do not have any jvm.options... I installed openjdk-8-jdk

[jasontedor]

I see, you’re running an old version of Elasticsearch. To set the heap size on the version you’re on you should set the ES_HEAP_SIZE environment variable. And of course, upgrade to the latest if you can.

[rhalbheer (Roger Halbheer)]

I just installed via apt-get install elasticsearch - do I not get the latest version then?
So will upgrade first

[jasontedor]

Those are the default packages unmaintained by Elastic. Instead, use our repository: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/deb.html