Logstash not reading the logs from file input

shashi0905 · December 9, 2016, 7:43am

I am trying to configure logstash to push data from log files to elasticsearch, but logstash is not reading the log files specified in the file input.

When run in debug mode, I see this message - "Plugin not defined in namespace, checking for plugin file".

Part of my configuration file :

input {
          beats {
            type => beats
           port => 5001
         }

         file{
           path => "/opt/application/proj/logs/prod-logs/proj.log.*"
           type => "proj-logs"
           start_position => "beginning"
        }
     }

Although beats is not configured yet, I want to test first with log files.

I have confirmed the file exists at the specified path, and there are no permission issues as well. I also tried modifying the log files to update its last modify time (to avoid any possible issue because of sincedb property), but it didn't work either.

There are no indices created in the elasticsearch, as no data being read by logstash.
What might be the possible issue, and what is the meaning of the message in the logs 'plugin not defined in namespace' ?

Some Lines from debug logs -

:message=>"Reading config file", :config_file=>"/images/ELKPKG/logstash-2.4.0/proj-funcANDapache-logstash.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"beats", :path=>"logstash/inputs/beats", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@path = [\"/opt/application/proj/logs/prod-logs/proj.log.*\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@type = \"proj-logs\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@start_position = \"beginning\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}

Logstash Version : 2.4.0
Elasticsearch Version : 2.4.0

Any help/support would be appreciated.

magnusbaeck · December 9, 2016, 8:02am

Look for log entries with "discover" in them. You could have a permission problem that results in Logstash not being able to find any files.

Perhaps Logstash thinks it has processed the files and is tailing them and waiting for more input. What's in the sincedb file?

shashi0905 · December 9, 2016, 10:51am

Thanks for your response.
Following are the entries with 'discover' and 'sincedb' in the debug logs

:message=>"Reading config file", :config_file=>"/images/ELKPKG/logstash-2.4.0/proj-funcANDapache-logstash.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@path = [\"/opt/application/proj/logs/prod-logs/proj.log.*\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@type = \"proj-logs\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@**sincedb**_write_interval = 15", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@delimiter = \"\\n\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@close_older = 3600", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"filter", :name=>"grok", :path=>"logstash/filters/grok", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
.................
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"filter", :name=>"drop", :path=>"logstash/filters/drop", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"config LogStash::Filters::Drop/@percentage = 100", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"starting agent", :level=>:info, :file=>"logstash/agent.rb", :line=>"213", :method=>"execute"}
:message=>"starting pipeline", :id=>"main", :level=>:info, :file=>"logstash/agent.rb", :line=>"487", :method=>"start_pipeline"}
:message=>"Registering file input", :path=>["/opt/application/proj/logs/prod-logs/proj.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"171", :method=>"register"}
:message=>"No **sincedb**_path set, generating one based on the file path", :**sincedb**_path=>"/home/osadmin/.**sincedb**_fe955e0b1809dbb46277ebb71ea4a22a", :path=>["/opt/application/proj/logs/prod-logs/proj.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"216", :method=>"register"}
:message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"_globbed_files: /opt/application/proj/logs/prod-logs/proj.log.*: glob is: [\"/opt/application/proj/logs/prod-logs/proj.log.2016-10-12\"]", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}
:message=>"config LogStash::Outputs::ElasticSearch/@hosts = [\"10.192.225.32:9200\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"_**discover**_file: /opt/application/proj/logs/prod-logs/proj.log.*: new: /opt/application/proj/logs/prod-logs/proj.log.2016-10-12 (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"310", :method=>"_**discover**_file"}
:message=>"_open_file: /opt/application/proj/logs/prod-logs/proj.log.2016-10-12: opening", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"86", :method=>"_open_file"}
:message=>"/opt/application/proj/logs/prod-logs/proj.log.2016-10-12: **sincedb** last value 17451552, cur size 17451552", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"123", :method=>"_add_to_**sincedb**"}

I don't think there is any permissions issue, as per the log entry file seems accessible to logstash. I'm not sure about the sincedb entry though, and the message 'plugin not defined in namespace' ?
Should I consider re-installing logstash instance?

magnusbaeck · December 9, 2016, 10:56am

I'm not sure about the sincedb entry though,

Not sure how to check it, or what are you unsure about?

and the message 'plugin not defined in namespace' ?

I don't think you should worry about that.

Should I consider re-installing logstash instance?

That probably won't help.

shashi0905 · December 9, 2016, 11:11am

Should I try deleting the sincedb files?

magnusbaeck · December 9, 2016, 11:55am

Yes, that could help. Shut down Logstash first, though.

shashi0905 · December 15, 2016, 11:15am

Hi Magnus,

I was able to find the issue, which was due to difference in the log files on qualification and production server. One of the field was missing in the production logs.

While debugging, I assumed that both the logs are same. But after checking all the configuration of logstash and elasticsearch, I turned to the logstash parsing section.... using the grok debugger against the log file being used. This is when I found out the issue... and moreover, the missing field was being used as a filter in the logstash parsing, resulting in, all the entries getting dropped.

Anyways, thanks for your help.

system · January 12, 2017, 11:16am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat not reading the input log file Beats filebeat	4	1019	July 1, 2021
Configuring logstash to listen to Filebeat Logstash	3	1608	March 15, 2017
Kibana not reading logstash output Logstash	2	845	June 9, 2017
Reading a log file into Logstash Logstash	9	25828	July 6, 2017
Debug input plugin Logstash	1	966	July 4, 2018

Logstash not reading the logs from file input

Related topics