Enrich application log massage

I have a java application that is writing out message as follows for example:
00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received <?xml version="1.0" encoding="UTF-8"?> message 1 message unstructure message message message

I need to extract the tags and post the whole message as is , for example

Tag1 : message
Tag2 : message
Tag3 : message
msg : Whole message as-is

Is this possible?

This is typically something you do in Logstash or using an ingest node pipeline in Elasticsearch.

Many thanks for this really quick reply. Is there an example you can point out that I can use as a basis for do this?

you could use the filebeat processor copy_field, that is merged but seems it's not documented

give it a try, and let us know it it's working

- copy_fields:
        - from: message
        to: event.original
    fail_on_error: false
    ignore_missing: true

sources :

Many thanks for your help.

Here is my message that I send to ES

00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received <?xml version="1.0" encoding="UTF-8"?>
<sr:Response schemaVersion="3.0" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:ns3="http://www.companyurl/ResponseAndAlert" xmlns:ns4="http://www.companyurl/S1SP" xmlns:ns5="http://www.companyurl/sm" xmlns:sr="http://www.companyurl/ServiceUserGateway">00:00:13,949 INFO [com.companyurl] (bq-thread-pool-threads - 881) COMP response by ServiceResponseController - received <?xml version="1.0" encoding="UTF-8"?>
<S1SPAlert schemaVersion="3.0" xmlns="http://www.companyurl/ServiceUserGateway11-22-33-44S1CE20042019-08-31T09:57:03<CanonicalizationMethod

I need to, for example, lift sr:ScheduleID as a key (index I think it is called in ES language) and value 38060 to send to ES.

I cannot figure out how in the example to have provided, I can use - from: message to extract sr:ScheduleID and its value.

Hope to hear from you soon.

Many thanks in advance..

my answer was focused on keep the "whole message as-is" part of the problem

to parse the sample message you sent, logstash is a more appropriate tool, like @Christian_Dahlqvist suggested

filebeat has the dissect processor, that is limited for simpler string

for a XML message like yours, logstash-filter-xml would do a better job

so you wold use filebeat to collect the messages, send it to logstash via filebeat-output, and enrich the message with logstash's filters and finally forward it to ES via logstash-output

1 Like

Oh wow!

This is really really helpful.

Many thanks... I am really grateful for you help.



This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.