Xml processing

Elastic Stack Logstash
1
<?xml version="1.0" encoding="UTF-8"?> 1794480 20180425195021 SES 10.174.79.158 2018-04-25 07:20:24.741642168 Wed, 25 Apr 2018 14:20:24 GMT 10.174.79.158 Apache-HttpClient/4.1.1 (java 1.5) VG1 10.174.131.158 2018-04-25 07:20:24 1794481 SES 10.174.79.158 2018-04-25 07:20:24.741642168 198.24.6.220 3758 200 Apache-HttpClient/4.1.1 (java 1.5) 2018-04-25 07:20:24.802608803 2018-04-25 07:20:24.861197008 VG1 10.174.131.158 2018-04-25 07:20:24 1 registeredDevices R2VuZXJpY19DUjEzMFBhcnQyXzIwMTgwNDI1XzE5NDkxNA== 1004 ownerIdtoIMSIandMSISDNmappingCfg.json is not configured. digits 1794482 20180425195021 SES 10.174.79.158 2018-04-25 07:20:24.741642168 Wed, 25 Apr 2018 14:20:24 GMT 10.174.79.158 Apache-HttpClient/4.1.1 (java 1.5) VG1 10.174.131.158 2018-04-25 07:20:24

Hello ALl,

I want to filter data between I have multiple occurences of in a given file

I have tried using xpath and mapping each and indiviudal file

mapping individual fields

2

Did you actually read what you posted? Unless you format the XML as preformatted text (use Markdown or the </> toolbar button) the tags will be stripped.

3

179448020180425195021SES10.174.79.158 2018-04-25 07:20:24.741642168Wed, 25 Apr 2018 14:20:24 GMT10.174.79.158Apache-HttpClient/4.1.1 (java 1.5)VG110.174.131.1582018-04-25 07:20:241794481SES10.174.79.158 2018-04-25 07:20:24.741642168198.24.6.2203758200Apache-HttpClient/4.1.1 (java 1.5)2018-04-25 07:20:24.8026088032018-04-25 07:20:24.861197008VG110.174.131.1582018-04-25 07:20:241registeredDevicesR2VuZXJpY19DUjEzMFBhcnQyXzIwMTgwNDI1XzE5NDkxNA==1004ownerIdtoIMSIandMSISDNmappingCfg.json is not configured.digits179448220180425195021SES10.174.79.158 2018-04-25 07:20:24.741642168Wed, 25 Apr 2018 14:20:24 GMT10.174.79.158Apache-HttpClient/4.1.1 (java 1.5)VG110.174.131.1582018-04-25 07:20:24

Thanks for getting back to me this the xml file I have, my bad I have posted wrong data earlier

if I want to extract data between trafficeventData tags is there a easier way to do

I am used to processing csv files, this is the first time i am trying to process xml file

I have a requirement to retain all the fields between these tags

4

Seriously, read what you're posting or use the preview pane. What you posted isn't shown XML and unless you fix this it's impossible to help.

5 
* <?xml version="1.0" encoding="UTF-8"?>
* <TrafficEventData>
*    <TE1>
*   <E60>1794480</E60>
*   <E21>20180425195021</E21>
*   <E61>SES</E61>
*   <E65>10.174.79.158 2018-04-25 07:20:24.741642168</E65>
*   <E91>Wed, 25 Apr 2018 14:20:24 GMT</E91>
*   <E66>10.174.79.158</E66>
*   <E69>Apache-HttpClient/4.1.1 (java 1.5)</E69>
*   <E72>VG1</E72>
*   <E73>10.174.131.158</E73>
*   <E74>2018-04-25 07:20:24</E74>
*    </TE1>
*    <TE0>
*   <E60>1794481</E60>
*   <E61>SES</E61>
*   <E65>10.174.79.158 2018-04-25 07:20:24.741642168</E65>
*   <E66>198.24.6.220</E66>
*   <E67>3758</E67>
*   <E24>200</E24>
*   <E69>Apache-HttpClient/4.1.1 (java 1.5)</E69>
*   <E70>2018-04-25 07:20:24.802608803</E70>
*   <E71>2018-04-25 07:20:24.861197008</E71>
*   <E72>VG1</E72>
*   <E73>10.174.131.158</E73>
*   <E74>2018-04-25 07:20:24</E74>
*   <S77>
*      <S121>
*         <E1000>1</E1000>
*         <E1001>registeredDevices</E1001>
*         <E1002>R2VuZXJpY19DUjEzMFBhcnQyXzIwMTgwNDI1XzE5NDkxNA==</E1002>
*         <E1007>1004</E1007>
*         <E206>ownerIdtoIMSIandMSISDNmappingCfg.json is not configured.</E206>
*         <E2409>digits</E2409>
*      </S121>
*   </S77>
*    </TE0>
*    <TE2>
*   <E60>1794482</E60>
*   <E21>20180425195021</E21>
*   <E61>SES</E61>
*   <E65>10.174.79.158 2018-04-25 07:20:24.741642168</E65>
*   <E91>Wed, 25 Apr 2018 14:20:24 GMT</E91>
*   <E66>10.174.79.158</E66>
*   <E69>Apache-HttpClient/4.1.1 (java 1.5)</E69>
*   <E72>VG1</E72>
*   <E73>10.174.131.158</E73>
*   <E74>2018-04-25 07:20:24</E74>
*    </TE2>
* </TrafficEventData>

sorry for the inconvenience i was not paying attention to the preview this the xml data

6

Can't you just use the xml filter with the default store_xml option? Then you'll basically convert the XML data to a JSON representation and thus retain all data.

7

thanks Magnus

8

Hello Magnus,

Thanks a to one small help, I want to extract a nodename(TNTMO071.) from the following input path

TNTMO071.2018-04-26_07%3A00.log

I have tried the following pattern and cannot see it in Kibana even though there is no error while running logstash

the following is the input and filter

file {
path =>"/opt/LogStashOutputFormatted/TMOES/*.log"
start_position => "beginning"
sincedb_path => "/dev/null"
type => "tmoes"
}
grok {
match => ["path", "/%{WORD:nodeName}.log$"]
}

9

So what do you see in Kibana? Copy/paste from the JSON tab so we can see an example document.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.