Unable to install log4j2 plugin into Logstash

[john.jensen (John)]

I'm trying to use log4j2 to send messages to our Logstash instance.
I'm running this command:
bin\logstash -r -p "C:\elasticsearch_course\logstash-input-log4j2" -f "C:\elasticsearch_course\logstash_data\test.conf"

The test.conf file:
input {
log4j2 {
port => 7000
mode => "server"
}

file {
path => "D:/logs/application.log"
sincedb_path => "nul"
start_position => "beginning"
}
}

filter {
grok {
match => {
"message" => "%{NOTSPACE:date} %{NOTSPACE:time} %{WORD:level} %{NUMBER:FIELD1} %{NOTSPACE:FIELD2} %{NOTSPACE:FIELD3} %{NOTSPACE:class}%{SPACE}%{NOTSPACE:FIELD4} %{WORD:method}"
}
remove_field => ["FIELD1","FIELD2","FIELD3","FIELD4"]
}
}

output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
index => "gopher-%{+YYYY.MM.dd}"
manage_template => true
template => "C:\elasticsearch_course\logstash_data\gopher_mapping.json"
template_name => "gopher_template"
}
}

I'm getting the following error:
Validating logstash-input-log4j2
Installing logstash-input-log4j2
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core":
In snapshot (Gemfile.lock):
logstash-core (= 6.6.0)

In Gemfile:
logstash-core-plugin-api (>= 0) java depends on
logstash-core (= 6.6.0) java

logstash-input-syslog (>= 0) java depends on
  logstash-filter-grok (>= 0) java depends on
    logstash-core (>= 5.6.0) java

logstash-input-log4j2 (>= 0) java depends on
  logstash-core (< 2.0.0, >= 1.4.0) java

logstash-core (>= 0) java

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash":
In Gemfile:
logstash-input-log4j2 (>= 0) java depends on
logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-input-log4j2 (>= 0) java', in any of the sources.

I'm using the latest versions of Logstash (6.6.0) and the plugin that I can find. I'm on a Windows machine and don't have much skill with Linux so I'm trying to translate from the references I've seen here and the log4j2 references.

How do I install the log4j2 plugin into Logstash on a Windows 10 computer?

[Badger]

@john.jensen

I'm trying to use log4j2 to send messages to our Logstash instance

Instead of using a plugin to connect them directly, write to a file and ingest the file using filebeat, or, my preference, configure log4j to write to an event log and ingest using winlogbeat.

[john.jensen (John)]

That is not acceptable due to project constraints. I need to do this without writing a log file.

[Badger]

OK, you still have other appenders that should be able to connect to logstash. I haven't tried it but how about a socket or http appender with a JsonLayout.

[john.jensen (John)]

The requirements document specifies this solution.

[Badger]

I feel your pain. Depending on which version you pulled, you have a plugin that was probably last updated 3 years ago, when it was running in logstash 2.1. You are trying to install that in logstash 6.6.

Specifically, the messages are telling you that you have version 6.6.0 of the package logstash-core, and logstash-input-log4j2 is compatible with versions 1.4.0 up to 2.0.0. Obviously 6.6.0 is not in that range.

So, you would need to build a new version of the plugin that

a) Is tagged as being compatible with 6.6.0 (probably a straightforward change)
b) Is actually compatible with 6.6.0.

I am pretty sure there have been changes in the last 4 major versions that will require changes in the plugin.

[john.jensen (John)]

So is there another solution I should be looking at? The main requirement was that we pick up the logging message without them being written to a file. If this isn't going to work I need to propose an alternate.

[Badger]

As I said, you may be able to use a socket or http appender with a JsonLayout in the log4j configuration pointed at a tcp or http input.