Logstash stops without opening file to read

Elastic Stack Logstash
1

Hi,

Here's my config file :

input{file {
path => "/data/ttr.csv"
sincedb_path => "/dev/null"
start_position => "beginning"
}}
output{
stdout{ }
}

When i start logstash with debug mode:

Reading config file {:file=>"logstash/agent.rb", :level=>:debug, :line=>"318", :method=>"local_config"}
Compiled pipeline code:
inputs =
filters =
outputs =
periodic_flushers =
shutdown_flushers = 

      input_file_1 = plugin("input", "file", LogStash::Util.hash_merge_many({ "path" => ("/data/ttr.csv") }, { "start_position" => ("beginning") }))
      inputs << @input_file_1
      output_stdout_2 = plugin("output", "stdout")
      outputs << @output_stdout_2

def filter_func(event)
events = [event]
logger.debug? && @logger.debug("filter received", :event => event.to_hash)
events
end
def output_func(event)
logger.debug? && @logger.debug("output received", :event => event.to_hash)
output_stdout_2.handle(event)

end {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"37", :method=>"initialize"}
Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"71", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"71", :method=>"lookup"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config
config LogStash::Inputs::File/@path = ["/data/ttr.csv"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
config LogStash::Inputs::File/@start_position = "beginning" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"71", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"line", :path=>"logstash/codecs/line"file=>"logstash/plugin.rb", :line=>"71", :method=>"lookup"}
config LogStash::Codecs::Line/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::Line charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
config LogStash::Outputs::Stdout/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"128", :method=>"config_init"}
Default settings used: Filter workers: 4
Registering file input {:path=>["/data/ttr.csv"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"128", :method=>"register"}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_3e41defd8eea49698d0fc0c9c77f1600", :path=>["/data/ttr.csv"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"171", :method=>"register"}
Pipeline started {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"89", :method=>"run"}
Logstash startup completed
_sincedb_open: /root/.sincedb_3e41defd8eea49698d0fc0c9c77f1600: No such file or directory - /root/.sincedb_3e41defd8eea49698d0fc0c9c77f1600 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"215", :method=>"_sincedb_open"}
_discover_file_glob: /data/ttr.csv: glob is: ["/data/ttr.csv"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
_discover_file: /data/ttr.csv: new: /data/ttr.csv (exclude is ) {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"141", :method=>"_discover_file"}
closing {:plugin=><LogStash::Inputs::File path=>["/data/ttr.csv"], start_position=>"beginning", codec=><LogStash::Codecs::Plain charset=>"UTF-8">, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">, :level=>:debug, :file=>"logstash/plugin.rb", :line=>"35", :method=>"do_close"}
closing {:plugin=><LogStash::Outputs::Stdout codec=><LogStash::Codecs::Line charset=>"UTF-8">, workers=>1>, :level=>:debug, :file=>"logstash/plugin.rb", :line=>"35", :method=>"do_close"}
Pipeline shutdown complete. {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"103", :method=>"run"}
Logstash shutdown completed

It stops before even reading the file.
I Don't understand why.

Any help please ?
Thanks.

2

Try changing the input to stdin {} and then cat /data/ttr.csv | logstash -f config.conf.

3

No sincedb_path set, generating one based on the file path

This isn't consistent with the configuration you showed us. Make sure you're running with the intended configuration.

4

Thanks for your reply, it works.

But I would like to do it with the file-plugin, the main purpose is to read different log files from a unique source, so the plugin would be useful fo types, charsets etc ..

I notice another issue, logstash stops every time I launch it after reading the file.
Logstash shutdown completed
This behaviour doesnt let me read new lines from files, how can I change this to keep logstash permanetly up and waiting for new events ?

5

Thanks for your reply.

May be I've tried with the wrong config file.

I just have tried again to run it with debug mode and i have this line for since_db.
_sincedb_open: reading from /data/sincedb {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"219", :method=>"_sincedb_open"}

6

But I would like to do it with the file-plugin, the main purpose is to read different log files from a unique source, so the plugin would be useful fo types, charsets etc ..

Of course, but during testing when you're probably reading the same file many times it's often more convenient to use the stdin input.

I notice another issue, logstash stops every time I launch it after reading the file.

Yes, that's expected. When the stdin input reaches the end of the input stream it'll signal to Logstash to shut down.

I just have tried again to run it with debug mode and i have this line for since_db.

_sincedb_open: reading from /data/sincedb {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"219", :method=>"_sincedb_open"}

Yes? This indicates that you have sincedb_path => "/data/sincedb" in your configuration.

7

@warkolm
@magnusbaeck
I tried this workaround https://github.com/elastic/logstash/issues/3127#issuecomment-101068714
And it seems to work.

8

Um, okay? What made you try that workaround, given that it's a workaround for a problem that you don't seem to have?