Logstash could not find any input plugin named 'journald', NameError

Elastic Stack Logstash
1

Hello Logstash Community,

I'm looking for some help resolving a NameError problem when trying to run the journald input plugin.

More specifically - I'm building a docker image for Logstash 5.5.1 and installing the journald plugin using:

logstash-plugin install logstash-input-journald

which completes successfully:

Validating logstash-input-journald
Installing logstash-input-journald
Installation successful

I also verify that the plugin is correctly installed by listing plugins on a running container:

logstash-plugin list --verbose | grep journald
logstash-input-journald (2.0.1)

The issue I'm running into is that when the plugin is started, I receive the following error:

[LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>
"Couldn't find any input plugin named 'journald'. Are you sure this is correct?
Trying to load the journald input plugin resulted in this error: Problems loading
the requested plugin named journald of type input. Error: NameError NameError"}

Here is the more verbose error:

[LogStash::Runner] ERROR logstash.plugins.registry - Problems loading a plugin
with {:type=>"input", :name=>"journald", :path=>"logstash/inputs/journald",
:error_message=>"NameError", :error_class=>NameError, :error_backtrace=>
["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in 
`namespace_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:in
`legacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:inbox_tray:
`lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:inbox_tray:
`lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:inbox_tray:
`lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:100:inbox_tray:
`plugin'", "(eval):12:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:72:in `initialize'", 
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:156:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", 
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'",
"/usr/share/logstash/logstash-core/lib/logstash/runner.rb:314:in `execute'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inbox_tray:
`run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:209:in `run'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inbox_tray:
`run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

For testing purposes I have simplified the journald plugin config as much as possible:

journald {
  lowercase => true
  path => "/var/log/journal"
  seekto => "head"
  thisboot => true
  type => "systemd"
}

My apologies for the wall of text. One final comment I'd like to make is that this was working fine using the elastic docker image docker.elastic.co/logstash/logstash:5.5.0 but that image is too bloated and so I'm trying to reduce the size by starting with alpine:3.6.

Does anyone know what's going on here and why it is that the plugin is throwing an error as if it is were not installed?

2

Bump.

3

Honestly, I have looked at the code and the version constraints and I can't see why the plugin registry does not find the gem.

Perhaps @Stuart_Warren (original author) can help.

4

Thanks for your response @guyboertje. While I have not found a solution to this yet, I have discovered others having the same issue on logstash's github:

Maybe there are some details there that will lead you to a solution.

5

I cannot reproduce.

% bin/logstash-plugin install logstash-input-journald
Validating logstash-input-journald
Installing logstash-input-journald
Installation successful
% bin/logstash -e 'input { journald { } }'
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /home/jls/build/logstash-5.5.0/logs which is now configured via log4j2.properties
[2017-09-05T14:01:21,698][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-09-05T14:01:21,719][INFO ][logstash.inputs.journald ] No sincedb_path set, generating one for the journal {:sincedb_path=>"/home/jls/build/logstash-5.5.0/.sincedb_journal"}
[2017-09-05T14:01:21,723][INFO ][logstash.pipeline        ] Pipeline main started
[2017-09-05T14:01:21,766][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
6

So this works with the Elastic logstash docker image, but not your custom alpine-based one? This issue is missing data -- are you sure your Dockerfile is correct?

The error you are getting happens like this:

  • pipeline asks for an input plugin named "journald"
  • but none is found.
  • so it raises NameError

If I had to guess, my guess is that you are installing the journald input in a different docker container than the one having this error.

7

Hello Jordan,

As I indicated in my post above, I've ran logstash-plugin list to verify that logstash-input-journald is installed in the logstash image I'm attempting to run. Please feel free to review my Dockerfile below:

FROM alpine:3.6

RUN apk add --no-cache openjdk8-jre tini su-exec

ENV LS_VERSION 5.5.2
ENV LOGSTASH_URL "https://artifacts.elastic.co/downloads/logstash"
ENV LOGSTASH_TARBALL "$LOGSTASH_URL/logstash-${LS_VERSION}.tar.gz"
ENV LOGSTASH_TARBALL_ASC "$LOGSTASH_URL/logstash-${LS_VERSION}.tar.gz.asc"
ENV LOGSTASH_TARBALL_SHA1 "2961489ccf8bef2bf9ae6c4eaaeeeb65b2ccd109"
ENV GPG_KEY "46095ACC8548582C1A2699A9D27D666CD88E42B4"

RUN apk add --no-cache libzmq bash
RUN apk add --no-cache -t .build-deps wget ca-certificates gnupg openssl \
  && cd /tmp \
  && wget --progress=bar:force -O logstash.tar.gz "$LOGSTASH_TARBALL"; \
  if [ "$LOGSTASH_TARBALL_SHA1" ]; then \
		echo "$LOGSTASH_TARBALL_SHA1 *logstash.tar.gz" | sha1sum -c -; \
	fi; \
	\
	if [ "$LOGSTASH_TARBALL_ASC" ]; then \
		wget --progress=bar:force -O logstash.tar.gz.asc "$LOGSTASH_TARBALL_ASC"; \
		export GNUPGHOME="$(mktemp -d)"; \
		gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY"; \
		gpg --batch --verify logstash.tar.gz.asc logstash.tar.gz; \
		rm -r "$GNUPGHOME" logstash.tar.gz.asc; \
	fi; \
  tar -xzf logstash.tar.gz \
  && mv logstash-$LS_VERSION /usr/share/logstash \
  && adduser -DH -s /sbin/nologin logstash \
  && rm -rf /tmp/* \
  && apk del --purge .build-deps

ENV PATH /usr/share/logstash/bin:/sbin:$PATH
ENV LS_SETTINGS_DIR /usr/share/logstash/config

RUN set -ex; \
  if [ -f "$LS_SETTINGS_DIR/log4j2.properties" ]; then \
    cp "$LS_SETTINGS_DIR/log4j2.properties" "$LS_SETTINGS_DIR/log4j2.properties.dist"; \
    truncate -s 0 "$LS_SETTINGS_DIR/log4j2.properties"; \
  fi

RUN /usr/share/logstash/bin/logstash-plugin uninstall logstash-input-http_poller \
  && /usr/share/logstash/bin/logstash-plugin uninstall logstash-output-http \
  && /usr/share/logstash/bin/logstash-plugin install --version 4.4.0 logstash-output-http \
  && /usr/share/logstash/bin/logstash-plugin install logstash-input-journald

VOLUME ["/etc/logstash/conf.d"]

COPY logstash-entrypoint.sh /

ENTRYPOINT ["/logstash-entrypoint.sh"]
CMD ["-e", ""]

where logstash-entrypoint.sh is as follows:

#!/bin/sh

set -e

# Add logstash as command if needed
if [ "${1:0:1}" = '-' ]; then
	set -- logstash "$@"
fi

# Run as user "logstash" if the command is "logstash"
if [ "$1" = 'logstash' ]; then
	chown -R logstash: /usr/share/logstash
	chown -R logstash: /etc/logstash/conf.d/
	# chown -R logstash: /opt/logstash/patterns

	set -- su-exec logstash tini -- "$@"
fi

exec "$@"

Please note I've set this Dockerfile up by combining concepts from older Elasticsearch dockerfiles and various github repos.

8

@jordansissel Could you try the same experiment with the Dockerfile I provided above? I think this is still an issue.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.