Field Extraction/Parsing

[elborni96 (Mirko Bornivelli)]

Hi everyone, I'm a new user of the ELK stack.

I'm monitoring a file and I would like to extract fields (hereinafter a small extraction):

<Item> <title> Vulnerability in Cisco Products (March 6, 2019) </ title> <Link> https://www.certnazionale.it/news/2019/03/07/vulnerabilita-in-prodotti-cisco-6-marzo-2019/ </ link> <pubDate> Thu, 07 Mar 2019 10:01:10 +0000 </ pubDate> <dc: creator> <! [CDATA [National CERT]]> </ dc: creator> <Category> <! [CDATA [Vulnerability]]> </ category> <Category> <! [CDATA [cisco]]> </ category> <guid isPermaLink = "false"> https://www.certnazionale.it/?post_type=news&#038;p=1946 </ guid> <description> <! [CDATA [Cisco has released several security updates on March 6, 2019 that address multiple vulnerabilities in different products.]]> </ description> </ Item>

I would like to extract the folowing fields (with relative values):

title=Vulnerability in Cisco Products (March 6, 2019)

pudDate=Thu, 07 Mar 2019 10:01:10 +0000

description=Cisco has released several security updates on March 6, 2019 that address multiple vulnerabilities in different products.

Can you explain me how to extract fields at index time with Filebeat or Logstash?

Thanks

[pierhugues (Pier-Hugues Pellerin)]

Hello @elborni96, You cannot do that in Filebeat at the moment, you will need to use logstash and the logstash-filter-xml

[elborni96 (Mirko Bornivelli)]

Hi @pierhugues,

I need to understand how to parse / extract fields of events that come to me through the Filbeat, regardless of the type of file (XML, HTML, JSON, syslog ...).

Could you explain how to do it via LogStash?

Regards,
Mirko

[pierhugues (Pier-Hugues Pellerin)]

(XML, HTML, JSON, syslog ...).

The big picture would be something like this.

• Define different Filebeat inputs per document types and use fields to add a field to identify the type of document.
• Use the Logstash output in filebeat.
• Use the beats input in logstash
• Define a flow to send XML events to the logstash-xml-filter using conditionals.

After that you can what you want with the data of the event.

[elborni96 (Mirko Bornivelli)]

@pierhugues

I have defined Filebeat input that monitor my file (/home/root/test.log).

After i send monitored data to Logstash using the output function.

I have configured yet on Logstash the beats input, but i need to parse the content of the received events.

These events can be xml, syslog, json or other format; so i need to understand how to extract certain fields.

On Splunk Enterprise i can parse or extract fields using the following syntax:

EXTRACT-host_field = $regex_to_extract_host

How can do the same using parsing/extraction features of Logstash?

Can i create custom searchable fields valorized from the regex results?

Thanks,
Mirko