Is the multiline codec supposed to work with the JDBC input plugin?

I been having a hard time processing log files stored in an Oracle database table. The logs are stored as CLOB, but converted to VARCHAR2 using a pipelined table function. This works great with the JDBC input plugin in Logstash.

The problem is when I try to use the multiline codec, since Logstash seems to be ignoring it. When running the equivalent configuration, but with the file input plugin, the multiline codec works as expected. I even created a minimal scenario with 3 rows on an Oracle table, with a VARCHAR2 column, and had the same result.

Is the multiline codec supposed to work with the JDBC input plugin? Is there additional configuration required? I see it getting called in the debug log, but it doesn't really do anything...

I am using Logstash 7.11.

I do not believe the jdbc plugin uses a codec. There is a comment in the code that supports this view. As the documentation for the input says, "Each row in the resultset becomes a single event."

I believe there are other inputs or outputs that ignore the codec option.

The jdbc plugin takes a codec in the configuration, you see it in the code that defaults to plain. I saw it in the debug as well.

Because it creates a single event per database returned row, it then looks like it really ignores the codec. Seems like something that should be configurable, or at the very least should have a warning or error in the log indicating so.

Yes, it ignores the codec. The only reason it enables the codec option is that logstash requires it. Every input and output has to have a codec option. Using it is optional.

This is another example of this, with a workaround.

I tried the workaround you mentioned in your link, using the TCP input/output plugins on two separate pipelines. The output of the pipeline comes out as JSON.

Can I change the format of the TCP output plugin somehow, so configuring the multiline codec is simpler/possible?
Or, do I need to include the JSON formatting added in the first pipeline as part of the multiline pattern?
Or, something else?

Yes, the default codec for a tcp output is json. You could change that to line (or possibly plain), perhaps with a format option on the codec.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.