XML filter issue

Elastic Stack Logstash
1

I am getting started with the XML filter plugin. Here is my config:

filter {
  xml {
    source => "message"
    store_xml => false
  }
}

Ideally I want to have Logstash just convert the XML to JSON (ie. convert all the tags to JSON keys and the data inside them to the values). Is there any way to do this?

Currently it just outputs the entire XML event to the message field but doesn't change it in any way.

2

You can use Xpath to extract the specific fields you want to extract from the xml message.

3

I'm not sure what the format of each message is though.

4

Are you saying each xml you intend to ingest could have very different structures? Do you have a sample file to share with us?

5

Correct. I can't share the files, but yes, the data structures would be different.

6

That's a problem then, they have to have a common structure that you could build XPaths off of. BTW, when I say common structure, I mean the same XML tag structure.
<tag><subtag><tertiarytag></tertiarytag></subtag></tag>

It can be pretty formatted in any kind of way

<tag>
<subtag><tertiarytag></tertiarytag></subtag>
</tag>

or

<tag>
<subtag>
<tertiarytag></tertiarytag>
</subtag>
</tag>

etc...

7

Okay, so is there a way to have an external program filter it? For example, I could use this: https://hackage.haskell.org/package/xml-to-json

to convert all the XML to JSON. The only issue is I see that exec is only for outputs, not filters.

9

Well you get structured data in the event if you set store_xml to be true. And logstash can output that as JSON.

10

That worked perfectly! I just didn't think store_xml would be relevant based on the name. Thank you so much!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.