# Enrich application log massage

**URL:** <https://discuss.elastic.co/t/enrich-application-log-massage/199513>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 14, 2019, 6:54pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513 "2019-09-14T18:54:43Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sparmar](https://avatars.discourse-cdn.com/v4/letter/s/7993a0/32.png) [@sparmar](https://discuss.elastic.co/u/sparmar)\
**Post date:** [September 14, 2019, 6:54pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/1 "2019-09-14T18:54:44Z")

</div>

I have a java application that is writing out message as follows for example:  
00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received \<?xml version="1.0" encoding="UTF-8"?\> message 1 message unstructure message message message

I need to extract the tags and post the whole message as is , for example

Tag1 : message  
Tag2 : message  
Tag3 : message  
msg : Whole message as-is

Is this possible?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2019, 6:56pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/2 "2019-09-14T18:56:31Z")

</div>

This is typically something you do in Logstash or using an ingest node pipeline in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![sparmar](https://avatars.discourse-cdn.com/v4/letter/s/7993a0/32.png) [@sparmar](https://discuss.elastic.co/u/sparmar)\
**Post date:** [September 14, 2019, 7:12pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/3 "2019-09-14T19:12:58Z")

</div>

Many thanks for this really quick reply. Is there an example you can point out that I can use as a basis for do this?

---

<div class="post-metadata">

**Author:** ![endersonmaia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/endersonmaia/32/17308_2.png) [@endersonmaia](https://discuss.elastic.co/u/endersonmaia)\
**Post date:** [September 14, 2019, 9:54pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/4 "2019-09-14T21:54:07Z")

</div>

you could use the filebeat processor `copy_field`, that is merged but seems it's not documented

give it a try, and let us know it it's working

```
processors:
- copy_fields:
    fields:
        - from: message
        to: event.original
    fail_on_error: false
    ignore_missing: true

```

sources :

- [https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html)
- [https://github.com/elastic/beats/pull/11303](https://github.com/elastic/beats/pull/11303)
- [https://github.com/elastic/beats/issues/12633](https://github.com/elastic/beats/issues/12633)

---

<div class="post-metadata">

**Author:** ![sparmar](https://avatars.discourse-cdn.com/v4/letter/s/7993a0/32.png) [@sparmar](https://discuss.elastic.co/u/sparmar)\
**Post date:** [September 15, 2019, 12:36pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/5 "2019-09-15T12:36:49Z")

</div>

Many thanks for your help.

Here is my message that I send to ES

00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received \<?xml version="1.0" encoding="UTF-8"?\>  
\<sr:Response schemaVersion="3.0" xmlns:ds="[http://www.w3.org/2000/09/xmldsig#](http://www.w3.org/2000/09/xmldsig#)" xmlns:ns3="[http://www.companyurl/ResponseAndAlert](http://www.companyurl/ResponseAndAlert)" xmlns:ns4="[http://www.companyurl/S1SP](http://www.companyurl/S1SP)" xmlns:ns5="[http://www.companyurl/sm](http://www.companyurl/sm)" xmlns:sr="[http://www.companyurl/ServiceUserGateway](http://www.companyurl/ServiceUserGateway)"\>00:00:13,949 INFO [com.companyurl] (bq-thread-pool-threads - 881) COMP response by ServiceResponseController - received \<?xml version="1.0" encoding="UTF-8"?\>  
sr:Header  
sr:ResponseID11-22-33-44\</sr:ResponseID\>  
sr:ResponseCodeE62\</sr:ResponseCode\>  
sr:ResponseDateTime2019-08-31T09:57:04.98Z\</sr:ResponseDateTime\>  
\</sr:Header\>  
sr:Body  
sr:COMPAlertMessage  
sr:COMPAlertCodeN55\</sr:COMPAlertCode\>  
sr:COMPAlert  
sr:S1SPAlertDSP  
\<S1SPAlert schemaVersion="3.0" xmlns="[http://www.companyurl/ServiceUserGateway11-22-33-44S1CE20042019-08-31T09:57:03](http://www.companyurl/ServiceUserGateway11-22-33-44S1CE20042019-08-31T09:57:03)\<CanonicalizationMethod  
sr:ScheduleID38060\</sr:ScheduleID\>  
\</sr:S1SPAlert\>  
\</sr:COMPAlert\>  
\</sr:COMPAlertMessage\>  
\</sr:Body\>  
\</sr:Response\>

I need to, for example, lift sr:ScheduleID as a key (index I think it is called in ES language) and value 38060 to send to ES.

I cannot figure out how in the example to have provided, I can use - from: message to extract sr:ScheduleID and its value.

Hope to hear from you soon.

Many thanks in advance..

---

<div class="post-metadata">

**Author:** ![endersonmaia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/endersonmaia/32/17308_2.png) [@endersonmaia](https://discuss.elastic.co/u/endersonmaia)\
**Post date:** [September 15, 2019, 10:01pm UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/6 "2019-09-15T22:01:53Z")

</div>

my answer was focused on keep the "whole message as-is" part of the problem

to parse the sample message you sent, logstash is a more appropriate tool, like @Christian_Dahlqvist suggested

filebeat has the [dissect](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html) processor, that is limited for simpler string

for a XML message like yours, [logstash-filter-xml](https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html) would do a better job

so you wold use `filebeat` to collect the messages, send it to `logstash` via `filebeat-output`, and enrich the message with logstash's filters and finally forward it to ES via `logstash-output`

---

<div class="post-metadata">

**Author:** ![sparmar](https://avatars.discourse-cdn.com/v4/letter/s/7993a0/32.png) [@sparmar](https://discuss.elastic.co/u/sparmar)\
**Post date:** [September 16, 2019, 4:43am UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/7 "2019-09-16T04:43:54Z")

</div>

Oh wow!

This is really really helpful.

Many thanks... I am really grateful for you help.

Regards...

Suresh...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 14, 2019, 4:44am UTC](https://discuss.elastic.co/t/enrich-application-log-massage/199513/8 "2019-10-14T04:44:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.