Logstash don't parse logfile, but don't produce errors

Elastic Stack Logstash
1

Hi,

I have recently installed logstash + kibana + elasticsearch on my laptop.

My laptop hosts a Fedora Linux 21 x86_64
Logstash is version 1.5.4

After configuring logstash and starting it, it was not creating indices in elasticsearch with my initial configuration.
Logstash is not generating any error in its log files.

Searching on this list I found the following topics:

Unfortunatelly they didn't help me fix the issue, so far.

I have configured logstash as below to try to understand why it doesn't seems to read and parse the log files.

Input file:

input {
      file {
      path => ["/var/log/httpd/access_log"]
      exclude => ["*.gz", "*.bz2"]
      type => "apache_access"
      start_position => beginning
      }
}

Filter file:

filter {
  if [type] == "apache_access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      add_field => {"received_from" => "localhost" }
      }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  }
}

Output file:

output {
  stdout { codec => rubydebug }
}

When starting logstash with --debug I have the following trace.

[root@farsight ~]# /opt/logstash/bin/logstash --debug -f /etc/logstash/conf.d
Match data {:match=>{"message"=>"%{COMBINEDAPACHELOG}"}, :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"259", :method=>"register"}
Grok compile {:field=>"message", :patterns=>["%{COMBINEDAPACHELOG}"], :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"264", :method=>"register"}
regexp: /message {:pattern=>"%{COMBINEDAPACHELOG}", :level=>:debug, :file=>"logstash/filters/grok.rb", :line=>"266", :method=>"register"}
NotImplementedError: block device detection unsupported or native support failed to load
       blockdev? at org/jruby/RubyFileTest.java:67
         device? at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/helper.rb:67
  _sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/tail.rb:232
   sincedb_write at /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/tail.rb:205
        teardown at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.1/lib/logstash/inputs/file.rb:157
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:203
     synchronize at org/jruby/ext/thread/Mutex.java:149
     inputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:203
     start_input at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171

I have 2 questions.

  • Why this error is not reported in the logfiles of logstash as it seems quite serious?
  • Does anybody have a idea what can cause this problem?

Thank you.

2

I've worked around https://github.com/elastic/logstash/issues/3127 by creating the symlink as suggested:

ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so

This was on Ubuntu rather than Fedora though. Another option was to install the package that contains libcrypt.so but since that dragged in gcc I opted for this symlink hack.

Since this only appears to be a problem with Oracle's JRE another option would be to switch to OpenJDK.

3

Hi magnusbaeck,

I am using java-1.8.0-openjdk.x86_64.

/lib/x86_64-linux-gnu/libcrypt.so.1 doesn't exists on my system, but /usr/lib/libcrypt.so.1 do exists, so it should find it I suppose.

Thanks for your help.

4

To be complete,

I install the fedora repository of the elasticsearch components using this article:

https://www.ulyaoth.net/resources/tutorial-how-to-install-logstash-and-kibana-4-on-fedora-with-rsyslog.45/

5

/lib/x86_64-linux-gnu/libcrypt.so.1 doesn't exists on my system, but /usr/lib/libcrypt.so.1 do exists, so it should find it I suppose.

Do you also have /usr/lib/libcrypt.so? That's the file the JVM seems to be looking for.

6

No, here is the list of files beginning by libcrypt in /usr/lib:

[root@farsight ~]# ls -l /usr/lib/libcrypt*
-rwxr-xr-x. 1 root root   32492 27 fév  2015 /usr/lib/libcrypt-2.20.so
lrwxrwxrwx. 1 root root      19 13 aoû 14:19 /usr/lib/libcrypto.so.10 -> libcrypto.so.1.0.1k
-rwxr-xr-x. 1 root root 1935588 13 aoû 14:19 /usr/lib/libcrypto.so.1.0.1k
lrwxrwxrwx. 1 root root      22  8 sep 15:06 /usr/lib/libcryptsetup.so.4 -> libcryptsetup.so.4.7.0
-rwxr-xr-x. 1 root root  180540  8 sep 15:06 /usr/lib/libcryptsetup.so.4.7.0
lrwxrwxrwx. 1 root root      16 27 fév  2015 /usr/lib/libcrypt.so.1 -> libcrypt-2.20.so

So let's try adding a symlink from libcrypt-2.20.so to libcrypt.so

7

I just create the symlink and start logstash, but still same error :frowning:

I am wondering if the version of jruby bundled with logstash could be the source of the error. Is there a simple way to tell logstash to use the version installed from the official fedora package repository instead of the ones present in logstash subdir?

8

I have made some progress in defining where the problem is located.

It seems the problem is with the filter. If I remove it from the configuration, logstash stat without error, but it is also shuting down immediately without reading the content of the input file, even if there is not sincedb file.