Custom filter plugin logging configuration through log4j

[srikanth_muddu (srikanth)]

I have installed new filter plugin with jar dependencies.I kept log4j2.properties for a java project and was able to create logging with a java project.But after installing plugin logs are not appending in the logstash-plain.log file.can you help, please

[yaauie (Ry Biesemeyer)]

• What is your logging configuration, and at what level is Logstash configured to log at?
• Does the plugin in question call out to the logger (some don't), and if so, does it log at or above the currently-configured log level?
• Are you running a pipeline that uses the plugin in question?
• Does the user who owns the Logstash process have permission to write to the directory it's configured to write to?

[srikanth_muddu (srikanth)]

@yaauie

• What is your logging configuration, and at what level is Logstash configured to log at?

"warn" is the loglevel i'm logging at

@yaauie

• Does the plugin in question call out to the logger (some don't), and if so, does it log at or above the currently-configured log level?

Can you please elaborate this question, I didn't get completely.

@yaauie

• Are you running a pipeline that uses the plugin in question?

yes, I am running pipeline.

@yaauie

• Does the user who owns the Logstash process have permission to write to the directory it's configured to write to?

yes user has read and write permission. you can see here
-rw-r--r-- 1 xxxx xxxx 47942 Mar 12 12:20 ../../../logs/logstash-plain.log
[/quote]

[yaauie (Ry Biesemeyer)]

Let me try again:

• what filter plugin did you install?
• do you have evidence that the plugin logs at warn or above? Not all plugins are equally good about emitting log messages, and even the ones that do tend to emit logs at debug or trace, which are too low to be included in your configured output.
• you say that you are running a pipeline, but is that pipeline configured to use the filter plugin that you are having trouble with? Are events flowing through your pipeline?

[srikanth_muddu (srikanth)]

@yaauie

Let me try again:

• what filter plugin did you install?

Instead of grok plugin to parse events i have implemented custom filter plugin which parses events without regex.

@yaauie

• do you have evidence that the plugin logs at warn or above? Not all plugins are equally good about emitting log messages, and even the ones that do tend to emit logs at debug or trace, which are too low to be included in your configured output.

@yaauie

• you say that you are running a pipeline, but is that pipeline configured to use the filter plugin that you are having trouble with? Are events flowing through your pipeline?

pipeline is configured with the plugin what i have installed and events are also parsing properly.But logs are not appending into log file.
"date" => "2018-02-22",
"useragent" => "-",
"httpmethod" => "GET",
"type" => "accesslog",
"time" => "11:47:14",
"username" => "-"

[yaauie (Ry Biesemeyer)]

Without seeing the plugin code and configuration, I'm having a hard time understanding what you expect to happen; is the plugin open-source, or would you be willing to share a tarball of it privately?

[yaauie (Ry Biesemeyer)]

@srikanth_muddu I received a DM from you with a log4j configuration, information about where you placed JARS, and text from a myplugin.rb, containing approximately 5 lines of code referencing an object that I presume is defined in one of those JARs.

It still is very unclear what you want to get out of this; the myplugin.rb doesn't include a valid Logstash filter definition (did you use bin/logstash-plugin generate --type filter --name "${FILTER_NAME}" to generate it? This is a super helpful tool to scaffold a new plugin for you).

An example of a filter plugin with JAR dependencies is logstash-filter-dissect; perhaps you can learn from it?