I have a java application that is writing out message as follows for example:
00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received <?xml version="1.0" encoding="UTF-8"?> message 1 message unstructure message message message
I need to extract the tags and post the whole message as is , for example
Tag1 : message
Tag2 : message
Tag3 : message
msg : Whole message as-is
Is this possible?
This is typically something you do in Logstash or using an ingest node pipeline in Elasticsearch.
Many thanks for this really quick reply. Is there an example you can point out that I can use as a basis for do this?
you could use the filebeat processor
copy_field, that is merged but seems it's not documented
give it a try, and let us know it it's working
- from: message
Many thanks for your help.
Here is my message that I send to ES
00:00:13,950 INFO [com.companyurl.application] (bq-thread-pool-threads - 881) ABC response by ApplicationManager - received <?xml version="1.0" encoding="UTF-8"?>
<sr:Response schemaVersion="3.0" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:ns3="http://www.companyurl/ResponseAndAlert" xmlns:ns4="http://www.companyurl/S1SP" xmlns:ns5="http://www.companyurl/sm" xmlns:sr="http://www.companyurl/ServiceUserGateway">00:00:13,949 INFO [com.companyurl] (bq-thread-pool-threads - 881) COMP response by ServiceResponseController - received <?xml version="1.0" encoding="UTF-8"?>
<S1SPAlert schemaVersion="3.0" xmlns="http://www.companyurl/ServiceUserGateway11-22-33-44S1CE20042019-08-31T09:57:03<CanonicalizationMethod
I need to, for example, lift sr:ScheduleID as a key (index I think it is called in ES language) and value 38060 to send to ES.
I cannot figure out how in the example to have provided, I can use - from: message to extract sr:ScheduleID and its value.
Hope to hear from you soon.
Many thanks in advance..
my answer was focused on keep the "whole message as-is" part of the problem
to parse the sample message you sent, logstash is a more appropriate tool, like @Christian_Dahlqvist suggested
filebeat has the dissect processor, that is limited for simpler string
for a XML message like yours, logstash-filter-xml would do a better job
so you wold use
filebeat to collect the messages, send it to
filebeat-output, and enrich the message with logstash's filters and finally forward it to ES via
This is really really helpful.
Many thanks... I am really grateful for you help.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.