Unable to start Logstash via systemd on RHEL - can't find Elapsed Plugin

Elastic Stack Logstash
1

Hi ELK-bods,

I'm going through my first install of ELK on Linux/RHEL and am not able to run Logstash via systemd as it seems it is not able to find/read the 'logstash-input-elapsed' plugin.

This plugin has been installed correctly and shows up in the plugin list:

[root@gbdocldibgdap02 6.1.0]# /usr/share/logstash/bin/logstash-plugin list elapsed
logstash-filter-elapsed
[root@gbdocldibgdap02 6.1.0]#

Additionally, when I run logstash directly from the command line, it works perfectly - including using the 'elapsed' plugin:

[2018-01-10T09:40:31,436][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.0"}
[2018-01-10T09:40:31,552][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-01-10T09:40:32,237][INFO ][logstash.filters.elapsed ] Elapsed, timeout: 60 seconds
[2018-01-10T09:40:32,239][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"uat-esb-audit-pipeline", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x348fd941@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2018-01-10T09:40:32,297][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"uat-esb-audit-pipeline"}
[2018-01-10T09:40:32,313][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["uat-esb-audit-pipeline"]}
[2018-01-10T09:40:32,644][INFO ][logstash.inputs.jdbc     ] (0.035720s) SELECT [rowID_PK] as primary_key
                                ,[createTimestamp]      as db_created
                                ,[eventTimestamp]       as event_created
                        FROM [ESB_PREPROD].[esb].[esb_audit]
                        WHERE [ESB_PREPROD].[ESB].[ESB_AUDIT].[rowID_PK] = 2130648
[2018-01-10T09:40:32,692][INFO ][logstash.filters.elapsed ] Elapsed, 'end event' received {:end_tag=>"END", :unique_id_field=>"unique_composite_id"}
[2018-01-10T09:40:32,832][INFO ][logstash.pipeline        ] Pipeline terminated {"pipeline.id"=>"uat-esb-audit-pipeline"}

However, when I try to start logstash via systemd:
systemctl start logstash

I get errors which seem to point to it not being able to find/read/access the 'elapsed' plugin:

[2018-01-10T09:41:50,655][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-01-10T09:41:50,657][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-01-10T09:41:50,909][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.0"}
[2018-01-10T09:41:50,969][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-01-10T09:41:51,646][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/filters/elapsed>, :path=>"logstash/filters/elapsed", :type=>"filter", :name=>"elapsed"}
[2018-01-10T09:41:51,656][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:uat-esb-audit-pipeline, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'elapsed'. Are you sure this is correct? Trying to load the elapsed filter plugin resulted in this error: no such file to load -- logstash/filters/elapsed", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:192:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:82:in `plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:114:in `plugin'", "(eval):64:in `<eval>'", "org/jruby/RubyKernel.java:994:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:86:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

It may be worth pointing out that the plugin was installed as an offline pack (my machines don't currently have access to the external sites) and I installed as 'root'.

I did notice that the plugin gemspec file was installed as 'root' whereas everything else within /usr/share/logstash/vendor/bundle/jruby/2.3.0/specifications was owned by 'logstash':

image
image.png742×369 8.87 KB

I had to chown/chmod this file to 'logstash' user and change the permissions:

[root@gbdocldibgdap02 6.1.0]# chown logstash:logstash /usr/share/logstash/vendor/bundle/jruby/2.3.0/specifications/logstash-filter-elapsed-4.0.3.gemspec
[root@gbdocldibgdap02 6.1.0]# chmod 664 /usr/share/logstash/vendor/bundle/jruby/2.3.0/specifications/logstash-filter-elapsed-4.0.3.gemspec

As without the chown/chmod starting via systemd didn't even get to the stage of running - instead I saw this line within /var/log/messages:

Jan 10 10:05:36 gbdocldibgdap02 logstash: Errno::EACCES: Permission denied - /usr/share/logstash/vendor/bundle/jruby/2.3.0/specifications/logstash-filter-elapsed-4.0.3.gemspec

Can someone advise:

  1. When installing plugins, is it a requirement to change the ownership/permissions of the installed plugin after installation? I would have thought the logstash-plugin executable would have handled this?

  2. What might be the problem which is stopping logstash from accessing the 'elapsed' plugin when its started via systemd.

All help appreciated as I'm a little stuck at this point :confused:

Cheers,
Steve

2

Hi Chaps,

Can anyone out there shed any light on this - I'm trying to set up a PoC and this is holding us up at the moment.

Cheers,
Steve

3

Hi stevedearl

have u tried

initctl start logstash
initctl status logstash
initctl stop logstash

I have n't gone through whole blog, but hope it works

4

Hi alpha_gamma,

Thanks for the comment but in my case our RHEL box is using systemd (not upstart) so I'm starting/stopping Logstash with:

systemctl start logstash
systemctl stop logstash

This is as per the Logstash documentation.

Cheers,
Steve

5

Have u tried Sudo -u with the commands,pls check

6

Hi alpha_gamma,

Thanks for the update. Can you provide any additional detail on what you mean?

I'm running the Logstash start/stop scripts as root already (should have mentioned that perhaps, sorry).

From the ELK v6.1 documentation around the RPM installation and configuration process I don't see anything on having to run systemctl with any sudo for any other account details.

If I run Logstash directly it finds and uses the Elapsed plugin without any problems at all:

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash

Additionally, it seems odd that I had to change the ownership of the installed plugin gemspec file.

Regards,
Steve

7

Hi All,

Just another call to see if anyone can help at all with this. Even something in terms of how to switch on more logging in Logstash (when running under systemd) or an answer to my question around plugin/gemspec file ownership would be helpful.

As I say, we're a bit stuck at the moment.

cheers,
Steve

8

Hi All,

I was working my way through the Logstash Ruby code which is called out within the Stacktrace I'm seeing:

[2018-01-18T08:32:34,703][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/filters/elapsed>, :path=>"logstash/filters/elapsed", :type=>"filter", :name=>"elapsed"}
[2018-01-18T08:32:34,707][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:uat-esb-audit-pipeline, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'elapsed'. Are you sure this is correct? Trying to load the elapsed filter plugin resulted in this error: no such file to load -- logstash/filters/elapsed", :backtrace=>
["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:192:in `lookup_pipeline_plugin'", 
"/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", 
"/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:82:in `plugin'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:114:in `plugin'", "(eval):64:in `<eval>'", "org/jruby/RubyKernel.java:994:in `eval'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:86:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", 
"/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in `block in execute'", 
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

and noticed that the section which is called out in /usr/share/logstash/logstash-core/lib/logstash/plugin.rb appears to be for backwards compatibility from 6.0 to 5.x:

  # This is keep for backward compatibility, the logic was moved into the registry class
  # but some plugins use this method to return a specific instance on lookup
  #
  # Should I remove this now and make sure the pipeline invoke the Registry or I should wait for 6.0
  # Its not really part of the public api but its used by the tests a lot to mock the plugins.
  def self.lookup(type, name)
    LogStash::PLUGIN_REGISTRY.lookup_pipeline_plugin(type, name)
  end

Could this be causing an issue perhaps (I appreciate I'm clutching at straws a bit here)

Can anyone confirm if they've been able to run the Elapsed plugin successfully using systemd on RHEL6. I'm using the ELK 6.1.0 binaries installed via RPM.

Cheers,
Steve

9

Hi All,

I've managed to resolve this issue by amending the user:group ownership of various files and dirs to 'logstash:logstash'. For info, I rah the following:

chown -R logstash:logstash /data/logstash
chown -R logstash:logstash /var/log/logstash
chown -R logstash:logstash /etc/logstash
chown -R logstash:logstash /usr/share/logstash

I suspect the above is a sledgehammer approach and I've probably changed the ownership of more files than I'd need to get around the problem - however, it has solved my issue.

Steve

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.