Enrich application log massage

Elastic Stack Beats
1

I have a java application that is writing out message as follows for example:
00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received <?xml version="1.0" encoding="UTF-8"?> message 1 message unstructure message message message

I need to extract the tags and post the whole message as is , for example

Tag1 : message
Tag2 : message
Tag3 : message
msg : Whole message as-is

Is this possible?

2

This is typically something you do in Logstash or using an ingest node pipeline in Elasticsearch.

3

Many thanks for this really quick reply. Is there an example you can point out that I can use as a basis for do this?

4

you could use the filebeat processor copy_field, that is merged but seems it's not documented

give it a try, and let us know it it's working

processors:
- copy_fields:
    fields:
        - from: message
        to: event.original
    fail_on_error: false
    ignore_missing: true

sources :

5

Many thanks for your help.

Here is my message that I send to ES

00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received <?xml version="1.0" encoding="UTF-8"?>
<sr:Response schemaVersion="3.0" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:ns3="http://www.companyurl/ResponseAndAlert" xmlns:ns4="http://www.companyurl/S1SP" xmlns:ns5="http://www.companyurl/sm" xmlns:sr="http://www.companyurl/ServiceUserGateway">00:00:13,949 INFO [com.companyurl] (bq-thread-pool-threads - 881) COMP response by ServiceResponseController - received <?xml version="1.0" encoding="UTF-8"?>
sr:Header
sr:ResponseID11-22-33-44</sr:ResponseID>
sr:ResponseCodeE62</sr:ResponseCode>
sr:ResponseDateTime2019-08-31T09:57:04.98Z</sr:ResponseDateTime>
</sr:Header>
sr:Body
sr:COMPAlertMessage
sr:COMPAlertCodeN55</sr:COMPAlertCode>
sr:COMPAlert
sr:S1SPAlertDSP
<S1SPAlert schemaVersion="3.0" xmlns="http://www.companyurl/ServiceUserGateway11-22-33-44S1CE20042019-08-31T09:57:03<CanonicalizationMethod
sr:ScheduleID38060</sr:ScheduleID>
</sr:S1SPAlert>
</sr:COMPAlert>
</sr:COMPAlertMessage>
</sr:Body>
</sr:Response>

I need to, for example, lift sr:ScheduleID as a key (index I think it is called in ES language) and value 38060 to send to ES.

I cannot figure out how in the example to have provided, I can use - from: message to extract sr:ScheduleID and its value.

Hope to hear from you soon.

Many thanks in advance..

6

my answer was focused on keep the "whole message as-is" part of the problem

to parse the sample message you sent, logstash is a more appropriate tool, like @Christian_Dahlqvist suggested

filebeat has the dissect processor, that is limited for simpler string

for a XML message like yours, logstash-filter-xml would do a better job

so you wold use filebeat to collect the messages, send it to logstash via filebeat-output, and enrich the message with logstash's filters and finally forward it to ES via logstash-output

1 Like
7

Oh wow!

This is really really helpful.

Many thanks... I am really grateful for you help.

Regards...

Suresh...

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.