Logstash 2.4 install not working

ddjembi · September 25, 2016, 9:15pm

Hi,
Just install Logstash on mac following [install instruction] (https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#installing-binary).

First example works fine (bin/logstash -e 'input { stdin { } } output { stdout {} }')
[second one] (https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html#advanced-pipeline) fails with the following error
Pipeline aborted due to error {:exception=>"ArgumentError", :backtrace=>["/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb:187:in register'", "org/jruby/RubyArray.java:1613:ineach'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb:185:in register'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:330:instart_inputs'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:329:instart_inputs'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:180:in start_workers'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:inrun'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}

Any idea why is not working as instructed ???

warkolm · September 25, 2016, 9:44pm

Can you show us the entire command you are calling, along with the shell and error?

magnusbaeck · September 26, 2016, 6:17am

By reading the stacktrace I'd say the problem is that the file path you've given in the file input is relative and not absolute.

The error message is horrible and a regression from Logstash 2.3. I've filed an issue to get this addressed.

ddjembi · September 26, 2016, 6:32pm

Logstash works fine with the following config in the inital-pipeline.conf file.
input {
generator {
lines => [
"line 1",
"line 2"
]
count => 3
}
}
output { stdout { } }

With the following command:
~/Documents/elk/logstash-2.4.0 $ bin/logstash -f initial-pipeline.conf

It gives the following results:
Settings: Default pipeline workers: 4
Pipeline main started
2016-09-26T18:18:34.420Z Denis-iMac27.local line 1
2016-09-26T18:18:34.431Z Denis-iMac27.local line 2
2016-09-26T18:18:34.431Z Denis-iMac27.local line 1
2016-09-26T18:18:34.432Z Denis-iMac27.local line 2
2016-09-26T18:18:34.432Z Denis-iMac27.local line 1
2016-09-26T18:18:34.433Z Denis-iMac27.local line 2
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

Trying the following config in first-pipeline.conf file
input {
file {
path => "NOTICES.TXT"
start_position => beginning
ignore_older => 0
}
}
output { stdout { } }

With the following command:
~/Documents/elk/logstash-2.4.0 $ bin/logstash -f first-pipeline.conf

It fails with the following shell errors:
Settings: Default pipeline workers: 4
Pipeline aborted due to error {:exception=>"ArgumentError", :backtrace=>["/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb:187:in register'", "org/jruby/RubyArray.java:1613:ineach'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb:185:in register'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:330:instart_inputs'", "org/jruby/RubyArray.java:1613:in each'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:329:instart_inputs'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:180:in start_workers'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:inrun'", "/Users/Denis/Documents/elk/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}.

Logstash also fails with all other config filters, inputs and output I've tried so far !!!

magnusbaeck · September 26, 2016, 6:59pm

path => "NOTICES.TXT"

As I said, this path must not be relative.

ddjembi · September 26, 2016, 7:21pm

Unbelievable !!! But you're right, everything works fine when using absolute path...
I cannot even begin to understand the poor quality of error message for such trivial error.

I'm just starting experiencing logstash and I've got some dozens of xml file to map into elastic. My hope was to use logstash and not a developing xml transformation myself to generate json files for import into elastic.

Is this kind of logstash error message a bug in 2.4 ... or should I expect any errors to be reported in such cryptic fashion ?

Thanks for your help anyway.

magnusbaeck · September 26, 2016, 7:39pm

Is this kind of logstash error message a bug in 2.4 ... or should I expect any errors to be reported in such cryptic fashion ?

As I said, this is a bug that was introduced in Logstash 2.4.

aghierna · September 29, 2016, 9:34am

Hi there!
It seems like I have a very similar problem. However, I have an absolute path and still get the same error.
I copy my conf file and the error:
input {
file {
path => "/Users/agarciah/Disco_D/Consultoria/Lineas/lineas.csv"
type => "lineas"
start_position => "beginning"
}
}

running: $ bin/logstash -f /users/agarciah/Downloads/logstashLineas.conf

gives:

Settings: Default pipeline workers: 4
Pipeline aborted due to error {:exception=>"ArgumentError", :backtrace=>["/usr/local/Cellar/logstash/2.4.0/libexec/vendor/bundle/jruby/1.9/gems/logstash-input-file-2.2.5/lib/logstash/inputs/file.rb:221:in register'", "/usr/local/Cellar/logstash/2.4.0/libexec/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:330:instart_inputs'", "org/jruby/RubyArray.java:1613:in each'", "/usr/local/Cellar/logstash/2.4.0/libexec/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:329:instart_inputs'", "/usr/local/Cellar/logstash/2.4.0/libexec/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:180:in start_workers'", "/usr/local/Cellar/logstash/2.4.0/libexec/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:inrun'", "/usr/local/Cellar/logstash/2.4.0/libexec/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}

Mac OSX - logstash 2.4 - elasticsearch 2.3.4

Thanks!

magnusbaeck · September 29, 2016, 3:49pm

Logstash seems to be complaining about your sincedb_path setting:

github.com

logstash-plugins/logstash-input-file/blob/v2.2.5/lib/logstash/inputs/file.rb#L221


      @logger.info("Renaming old ~/.sincedb to new one", :old => old_sincedb,
                   :new => @sincedb_path)
      File.rename(old_sincedb, @sincedb_path)
    end


    @logger.info("No sincedb_path set, generating one based on the file path",
                 :sincedb_path => @sincedb_path, :path => @path)
  end


  if File.directory?(@sincedb_path)
    raise ArgumentError.new("The \"sincedb_path\" argument must point to a file, received a directory: \"#{@sincedb_path}\"")
  end


  @tail_config[:sincedb_path] = @sincedb_path


  if @start_position == "beginning"
    @tail_config[:start_new_files_at] = :beginning
  end


  @codec = LogStash::Codecs::IdentityMapCodec.new(@codec)
end # def register

aghierna · September 29, 2016, 5:14pm

Thanks for your support. I changed the sincedb_path:
input {
file {
path => "/Users/agarciah/Disco_D/Consultoria/Lineas/lineas.csv"
type => "lineas"
start_position => "beginning"
sincedb_path => "/Users/agarciah/Disco_D/Consultoria/Lineas/sincedb.txt"
}
}
I can see sincedb.txt after running logstash but the same error message appears.