# Correct configuration file for XML input

**URL:** https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415
**Category:** Logstash
**Created:** [August 17, 2017, 12:55pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415 "2017-08-17T12:55:44Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![abhisek](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@abhisek](https://discuss.elastic.co/u/abhisek)
#### Post date: [August 17, 2017, 12:55pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/1 "2017-08-17T12:55:44Z")

</div>

My log look like this which is "|" separated.

The log consist of id, status, request, response fields.

The request and response are present in XML data which are "|" separated.

```
110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

```

And I have use configuration file like this :

```
input {
  file {
    path => "/opt/test5/practice_new/data.xml"
    sincedb_path => "/dev/null"

    multiline {
  pattern => "</soapenv:Envelope>"
  what => "previous"
  negate => "true"
}
  }
}
filter {
  grok {
    match => ["message", "%{DATA:method_id}|%{WORD:method_type}|%{GREEDYDATA:data}"]
  } 
}

output {
   elasticsearch {
     hosts => "http://localhost:XXXX"
     index => "req_res"
  }
stdout {}
}

```

But it is not working. Please let me know what changes should be made in configuration file.

My output should look like this:

```
ID- 110000

status- read

request - 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response - <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

```

Help me with the configuration file..

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 20, 2017, 8:39pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/2 "2017-08-20T20:39:45Z")

</div>

I can spot two bugs:

- If you want to match a literal "|" in a regular expression you need to escape it.
- The multiline configuration is most likely wrong. You'll want to adjust the expression so it matches the first line of the message, i.e. something like `^%{INT}\|%{WORD}\|<soapenv`.

While not directly related to your problem, the grok expression is inefficient. See my multiline suggestion above for an example of how to start the expression.

---

<div class="post-metadata">

### Author: ![abhisek](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@abhisek](https://discuss.elastic.co/u/abhisek)
#### Post date: [August 22, 2017, 6:57am UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/3 "2017-08-22T06:57:36Z")

</div>

Thanks magnus for your reply,

My issue was resolved with your multiline expression

> [@magnusbaeck](#):
>
> ^%{INT}|%{WORD}|\<soapenv

But there is a new issue that i'm stuck with, which is how will i parse for multiple xml log.

As you can see, my question has only one XML log data and which is working perfectly fine with your given pattern.

But when i add multiple log then it parse it as a 1 log data.

for ex :  
this is how my 1st log starts

> [@abhisek](#):
>
> 110000|read|\<soapenv:Envelope xmlns:

and my 2nd log is like this :

220000|write|\<soapenv:Envelope xmlns:....

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 24, 2017, 6:06am UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/4 "2017-08-24T06:06:16Z")

</div>

I'm not 100% sure what you mean so please show a complete example of a log you want to process.

---

<div class="post-metadata">

### Author: ![abhisek](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@abhisek](https://discuss.elastic.co/u/abhisek)
#### Post date: [August 24, 2017, 6:31am UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/5 "2017-08-24T06:31:51Z")

</div>

Thanks for reply

This is my first log:

```
110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

```

and the 2nd log :

```
213000|write|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

```

similarly there are "n" no. of logs

so using the above config file i'm able to map the 1st log but not the 2nd one.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 24, 2017, 2:36pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/6 "2017-08-24T14:36:16Z")

</div>

And in what way is it failing for the second set of log lines? What does your configuration look like?

---

<div class="post-metadata">

### Author: ![abhisek](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@abhisek](https://discuss.elastic.co/u/abhisek)
#### Post date: [August 28, 2017, 12:57pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/7 "2017-08-28T12:57:59Z")

</div>

my configuration look like this 🙂

```
input {
  file {
    path => "/opt/test5/practice_new/data.xml"
     start_position => "beginning"
        codec => multiline
  {
   pattern => "^%{INT}\|%{WORD}\|<soapenv:Envelope*>\|</soapenv:Envelope"
   negate => true
   what => "next"
  }
  }
}
filter {
  grok {
    match => ["message", "%{INT:method_id}\|%{WORD:method_type}\|%{GREEDYDATA:request}\|%{GREEDYDATA:response}"]
}
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "req_res"
  }
stdout {}
}

```

Currently my logs are being mapped like this:

```
ID- 110000

status- read

request - 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>| <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
213000|write|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response -<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
```

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 28, 2017, 1:59pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/8 "2017-08-28T13:59:23Z")

</div>

Use `what => "previous"` in your multiline codec.

---

<div class="post-metadata">

### Author: ![abhisek](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@abhisek](https://discuss.elastic.co/u/abhisek)
#### Post date: [August 30, 2017, 12:06pm UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/9 "2017-08-30T12:06:54Z")

</div>

Still getting the same issue.

---

<div class="post-metadata">

### Author: ![abhisek](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@abhisek](https://discuss.elastic.co/u/abhisek)
#### Post date: [August 31, 2017, 4:52am UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/10 "2017-08-31T04:52:14Z")

</div>

I tried a new config file

```
input {
  file {
    path => "/opt/test5/practice_new/data.xml"
    start_position => "beginning"
    codec => multiline {
            pattern => "^%{NUMBER:method_id}\|%{DATA:method_type}\|<soapenv:Envelope>"
            negate => true
            what => "previous"
        }
  }
}
filter {
  grok {
    match => ["(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request>[^|]*)\|(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)" ]
  }
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "req_res"
  }
stdout {}
}

```

Here i have used regular expression to match my logs.

but for this config it matches for only 1st log and does't not match form 2nd log onwards.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 28, 2017, 4:52am UTC](https://discuss.elastic.co/t/correct-configuration-file-for-xml-input/97415/11 "2017-09-28T04:52:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.