Is the multiline codec supposed to work with the JDBC input plugin?

Elastic Stack Logstash
#1

I been having a hard time processing log files stored in an Oracle database table. The logs are stored as CLOB, but converted to VARCHAR2 using a pipelined table function. This works great with the JDBC input plugin in Logstash.

The problem is when I try to use the multiline codec, since Logstash seems to be ignoring it. When running the equivalent configuration, but with the file input plugin, the multiline codec works as expected. I even created a minimal scenario with 3 rows on an Oracle table, with a VARCHAR2 column, and had the same result.

Is the multiline codec supposed to work with the JDBC input plugin? Is there additional configuration required? I see it getting called in the debug log, but it doesn't really do anything...

I am using Logstash 7.11.

#2

I do not believe the jdbc plugin uses a codec. There is a comment in the code that supports this view. As the documentation for the input says, "Each row in the resultset becomes a single event."

I believe there are other inputs or outputs that ignore the codec option.

#3

The jdbc plugin takes a codec in the configuration, you see it in the code that defaults to plain. I saw it in the debug as well.

Because it creates a single event per database returned row, it then looks like it really ignores the codec. Seems like something that should be configurable, or at the very least should have a warning or error in the log indicating so.

#4

Yes, it ignores the codec. The only reason it enables the codec option is that logstash requires it. Every input and output has to have a codec option. Using it is optional.

This is another example of this, with a workaround.

#5

I tried the workaround you mentioned in your link, using the TCP input/output plugins on two separate pipelines. The output of the pipeline comes out as JSON.

Can I change the format of the TCP output plugin somehow, so configuring the multiline codec is simpler/possible?
Or, do I need to include the JSON formatting added in the first pipeline as part of the multiline pattern?
Or, something else?

#6

Yes, the default codec for a tcp output is json. You could change that to line (or possibly plain), perhaps with a format option on the codec.