Fresh install, just dies. Great... now what?

Elastic Stack Logstash
1

So, I thought 'd give LS a try having heard so much about it.

I installed openjdk-7-jre onto a 12.04 ubuntu server then logatash 2.2.1 from the repositories (and then by hand when that didn't run). No errors are reported during install and

# java -version

returns

root@dns1:/# java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)`

and the command

# root@dns1:/opt/logstash# DEBUG=1 bin/logstash -e 'input { stdin { } } output { stdout {} }'

hangs, until ^C

Using DEBUG=1 I see

DEBUG: exec /opt/logstash/vendor/jruby/bin/jruby --1.9 -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J
-Djava.awt.headless=true -J-XX:CMSInitiatingOccupancyFraction=75 -J-XX:+UseCMSInitiatingOccupancyOnly -J
XX:+HeapDumpOnOutOfMemoryError -J-Xmx1g -J-XX:HeapDumpPath=/opt/logstash/heapdump.hprof     
opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -e input { stdin { } } output { stdout {} }`

with no warnings or errors. Given that LS is supposed to be great for newbies and I've wasted two hours investigatingthis with different variants of Java I'm a bit fed up... :wink:

Any ideas would be very welcome.

2

This is expected. Logstash is waiting to receive something on stdin. When you've started it, just type something into the terminal and press Enter. From where did you get the idea that you should run this command? If it's from the official documentation, perhaps there's room for improvement to correctly set users' expectations.

3

Hmmm I obviously haven't explained adequately - sorry....

Logstash isn't reading anything from stdin. If I type at it and send EOF nothing is processed at all. I've tried using echo to pipe data into it (to eliminate any tty driver issues) but that makes no difference. I was expecting the output as indicated in the Getting Started pages.

logstash --help

does nothing (no output)

Actually, running

logstash _anything_

to attempt to generate errors does nothing. Something is definitely wrong...

Regarding where I got that command from, I was following the command here
https://www.elastic.co/guide/en/logstash/current/first-event.html

and added the DEBUG=1 environment variable after examining the head of bin/logstash inan attempt to see something erroring

4

BTW, I've just reread my first post and the tone is a bit harsh - sorry, but it was 2am locally when I wrote it and was tired. No excuse but it still doesn't work... :wink:

5

Well, so I tried again. Fresh installation onto a different server

Same issue.

I'm beginning to think that Logstash simply doesn't work on Ubuntu 12.04.

Anyone care to comment?

9

Here's a pastebin of strace output

10

How about strace -f so that we can see what the Java process does and not just the launcher shell script?

11

Sorry. Like this ?

12

Hmm. The end looks like this:

[pid 23512] open("/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar", O_RDONLY) = 3
...
[pid 23512] lseek(3, 30487034, SEEK_SET) = 30487034
[pid 23512] read(3, "PK\3\4\24\0\10\10\10\0\267\0335H\0\0\0\0\0\0\0\0\0\0\0\0)\0\0\0", 30) = 30
[pid 23512] lseek(3, 30487105, SEEK_SET) = 30487105
[pid 23512] read(3, "\215QMO\0021\20}\5d\21P\24\277\277\"\301\30A\23\367\340\21cB\210&$\253\36$\\"..., 338) = 338
[pid 23512] lseek(3, 30486731, SEEK_SET) = 30486731
[pid 23512] read(3, "PK\3\4\24\0\10\10\10\0\265\0335H\0\0\0\0\0\0\0\0\0\0\0\0%\0\0\0", 30) = 30

If this is as far as it ever gets it looks like a JRuby problem. It doesn't look like it's even starting to load the Logstash Ruby source files.

13

OK. I let it run a bit longer - the trace never finishes. Logfile is pasted here

14

Soryr, I'm drawing a blank here. There are a lot of small reads of rt.jar but I'm sure if that's normal and, if not, why could be the cause of it. I'd try with a different JVM.

15

Sadly, I'm going to have to give up.

I've just spent a couple of hours uninstalling java (there were several variants installed) to a clean system and then installed Oracle's Java 7. I see exactly the same symptom.

Although I like the idea behind LS it's just not stable enough for my needs :frowning: - on Ubuntu 12.04 at least.

16

just checked... Oracle's Java 8 does the same too. So that's the same error under 6 different JVM's...

17

What filesystem are you running on?

18

Could you describe your environment a little more? Can you run 12.04.5?
This sounds like an issue with jruby firing. I googled around a bit for
jruby hanging on Ubuntu and it seems like if there are entropy issues it
can lead to jruby hanging with no obvious reason.

Anyone know if Logstash tests run on 12.04 boxes?

What filesystem are you running on?

19

Looks like from the trace you are running SELinux? Perhaps check its log as
well.

20

That's an interesting train of thought.

This is a VPS, running under OpenVZ so the filesystem is a proxy-filesystem known as simfs . SE-Linux will likely be running on the host, but it's not available to me.

In case it's relevant, the kernel is

root@me1:/opt/logstash# uname -a
Linux me1 2.6.32-042stab102.9 #1 SMP Fri Dec 19 20:34:40 MSK 2014 x86_64 x86_64 x86_64 GNU/Linux

Evidently, the environment is beyond my control.

I just installed from logstash_2.2.2-1_all.deb (I was previously installing from logstash_2.2.1-1_all.deb

21

Yeah I'm going to stick with entropy issues. The last real calls are to
open /dev/random and /dev/urandom and then logs just repeat themselves.
check this article out:

22

Check this Google search out: https://www.google.com/search?q=openvz+entropy

23

Also check out the Entropy section here:
https://github.com/jruby/jruby/wiki/Improving-startup-time and here:
https://github.com/jruby/jruby/issues/1896