Logstash not reading the logs from file input

Elastic Stack Logstash
1

I am trying to configure logstash to push data from log files to elasticsearch, but logstash is not reading the log files specified in the file input.

When run in debug mode, I see this message - "Plugin not defined in namespace, checking for plugin file".

Part of my configuration file :

input {
          beats {
            type => beats
           port => 5001
         }

         file{
           path => "/opt/application/proj/logs/prod-logs/proj.log.*"
           type => "proj-logs"
           start_position => "beginning"
        }
     }

Although beats is not configured yet, I want to test first with log files.

I have confirmed the file exists at the specified path, and there are no permission issues as well. I also tried modifying the log files to update its last modify time (to avoid any possible issue because of sincedb property), but it didn't work either.

There are no indices created in the elasticsearch, as no data being read by logstash.
What might be the possible issue, and what is the meaning of the message in the logs 'plugin not defined in namespace' ?

Some Lines from debug logs -

:message=>"Reading config file", :config_file=>"/images/ELKPKG/logstash-2.4.0/proj-funcANDapache-logstash.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"beats", :path=>"logstash/inputs/beats", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@path = [\"/opt/application/proj/logs/prod-logs/proj.log.*\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@type = \"proj-logs\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@start_position = \"beginning\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}

Logstash Version : 2.4.0
Elasticsearch Version : 2.4.0

Any help/support would be appreciated.

2

Look for log entries with "discover" in them. You could have a permission problem that results in Logstash not being able to find any files.

Perhaps Logstash thinks it has processed the files and is tailing them and waiting for more input. What's in the sincedb file?

3

Thanks for your response.
Following are the entries with 'discover' and 'sincedb' in the debug logs

:message=>"Reading config file", :config_file=>"/images/ELKPKG/logstash-2.4.0/proj-funcANDapache-logstash.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@path = [\"/opt/application/proj/logs/prod-logs/proj.log.*\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@type = \"proj-logs\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@**sincedb**_write_interval = 15", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@delimiter = \"\\n\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"config LogStash::Inputs::File/@close_older = 3600", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"filter", :name=>"grok", :path=>"logstash/filters/grok", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
.................
:message=>"Plugin not defined in namespace, checking for plugin file", :type=>"filter", :name=>"drop", :path=>"logstash/filters/drop", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"86", :method=>"lookup"}
:message=>"config LogStash::Filters::Drop/@percentage = 100", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"starting agent", :level=>:info, :file=>"logstash/agent.rb", :line=>"213", :method=>"execute"}
:message=>"starting pipeline", :id=>"main", :level=>:info, :file=>"logstash/agent.rb", :line=>"487", :method=>"start_pipeline"}
:message=>"Registering file input", :path=>["/opt/application/proj/logs/prod-logs/proj.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"171", :method=>"register"}
:message=>"No **sincedb**_path set, generating one based on the file path", :**sincedb**_path=>"/home/osadmin/.**sincedb**_fe955e0b1809dbb46277ebb71ea4a22a", :path=>["/opt/application/proj/logs/prod-logs/proj.log.*"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"216", :method=>"register"}
:message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"_globbed_files: /opt/application/proj/logs/prod-logs/proj.log.*: glob is: [\"/opt/application/proj/logs/prod-logs/proj.log.2016-10-12\"]", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"346", :method=>"_globbed_files"}
:message=>"config LogStash::Outputs::ElasticSearch/@hosts = [\"10.192.225.32:9200\"]", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"154", :method=>"config_init"}
:message=>"_**discover**_file: /opt/application/proj/logs/prod-logs/proj.log.*: new: /opt/application/proj/logs/prod-logs/proj.log.2016-10-12 (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"310", :method=>"_**discover**_file"}
:message=>"_open_file: /opt/application/proj/logs/prod-logs/proj.log.2016-10-12: opening", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"86", :method=>"_open_file"}
:message=>"/opt/application/proj/logs/prod-logs/proj.log.2016-10-12: **sincedb** last value 17451552, cur size 17451552", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"123", :method=>"_add_to_**sincedb**"}

I don't think there is any permissions issue, as per the log entry file seems accessible to logstash. I'm not sure about the sincedb entry though, and the message 'plugin not defined in namespace' ?
Should I consider re-installing logstash instance?

1 Like
4

I'm not sure about the sincedb entry though,

Not sure how to check it, or what are you unsure about?

and the message 'plugin not defined in namespace' ?

I don't think you should worry about that.

Should I consider re-installing logstash instance?

That probably won't help.

5

Should I try deleting the sincedb files?

1 Like
6

Yes, that could help. Shut down Logstash first, though.

7

Hi Magnus,

I was able to find the issue, which was due to difference in the log files on qualification and production server. One of the field was missing in the production logs.

While debugging, I assumed that both the logs are same. But after checking all the configuration of logstash and elasticsearch, I turned to the logstash parsing section.... using the grok debugger against the log file being used. This is when I found out the issue... and moreover, the missing field was being used as a filter in the logstash parsing, resulting in, all the entries getting dropped.

Anyways, thanks for your help.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.