Multiline XML processing using logstash

Elastic Stack Logstash
1

I have multi line XML to process more than 1000 lines. I am generating single message out of it.
When using Logstash 2.4 multiline filter plugin was working perfectly. But as its been deprecated in 6.1.

Therefore modified config to use this as codec multiline for 6.1 ,
input {
stdin {
type => "xml file path"
codec => multiline {
pattern =>"^<response>"
negate => true
what => "previous"
}
}

this just process half of the files.

later added max_lines & max_bytes seeing reference document

stdin {
type => " xml path"
codec => multiline {
pattern =>"^<response>"
negate => true
what => "previous"
max_bytes => "180 mb"
max_lines => 1000
}
Above config does nothing and simply terminates the pipline saying...
[INFO ][logstash.pipeline ] Pipeline terminated

Any help?

2

Any help in this regard???

3

What comes prior to "Pipeline terminated"?

Seems easier to just use an exec input to cat the input file.

4

Log to refer:
[2018-01-05T14:09:26,315][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Users/saketkum/Desktop/logstash6/logstash-6.1.1/logstash-6.1.1/modules/netflow/configuration"}
[2018-01-05T14:09:26,498][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-01-05T14:09:27,018][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.1.1"}
[2018-01-05T14:09:27,365][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-01-05T14:09:34,368][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>500, "pipeline.batch.delay"=>10, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x5b60c5d7 run>"}
[2018-01-05T14:09:34,433][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
[2018-01-05T14:09:34,541][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-01-05T14:09:35,624][INFO ][logstash.pipeline ] Pipeline terminated {"pipeline.id"=>"main"}
[2018-01-05T14:37:40,821][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Users/saketkum/Desktop/logstash6/logstash-6.1.1/logstash-6.1.1/modules/fb_apache/configuration"}
[2018-01-05T14:37:40,836][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Users/saketkum/Desktop/logstash6/logstash-6.1.1/logstash-6.1.1/modules/netflow/configuration"}
[2018-01-05T14:37:41,061][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-01-05T14:37:41,545][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.1.1"}
[2018-01-05T14:37:42,224][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-01-05T14:37:49,735][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>500, "pipeline.batch.delay"=>10, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x22025f5 run>"}
[2018-01-05T14:37:49,827][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
[2018-01-05T14:37:49,953][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-01-05T14:37:51,022][INFO ][logstash.pipeline ] Pipeline terminated {"pipeline.id"=>"main"}

How to use exec input to cat the input file. I am using xpath to select some values from the tags of XML file.

5

How to use exec input to cat the input file.

You use the exec input to run "cat filename.xml".

6

Sorry for coming again. I am not understanding how to use cat here on Windows.

I am running logstash to process xml file once : logstash -f conf < xml file

Therefore stdin input i used. so now as multiline codec is deprecated from filter so added it to input.

Please clarify.

7

Oh, this is Windows. Well, use whatever equivalent command that's available to you.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.