Create new Fields to Filter by

[Moritz_Kiesewetter]

Hello Community,
i'm still pretty new to Elastic-Stack.

I got a Cluster of 1 Master and 2 Nodes set up, and already got all my Beat-Agents deployed on the servers. Now i set up beat nodes on my Cluster aswell, to monitor any activity on it aswell (I use elk as a Security Setup). My Problem is that the fields which are filtered now are rudimentary.
The field "message" holds a few informations i actually want to filter by. How do i get my ES to put certain values in new fields? I found the fields.yml file, but i'm not sure how to use it, just write stuff somewhere? In which Syntax? Do i have to watch out for something?

Information in the message field looks like :

type=CRYPTO_KEY_USER msg=audit(1111111111.111:1111111): pid=111111 uid=0 auid=1111111111 ses=111111111 subj=system:

Let's say in this example i want the Information

type=CRYPTO_KEY_USER

seperated into an extra field, by which i can Filter in Kibana

Thanks in advance!

Mo

[steffens (Steffen Siering)]

Filebeat just collects log lines and sends each log line as an event. You need to parse the logs.

You can use filebeat modules, if you find one that matches the logs/service you want to monitor: https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-modules.html

Otherwise you need to parse the logs yourself. Using an Ingest Node pipeline (Elasticsearch feature), you can create a pipeline in Elasticsearch doing the parsing for you (e.g. dissect, grok, or key value parser). You can also try to apply the parsing in filebeat using the dissect processor.

[Moritz_Kiesewetter]

Sounds like a lot of work for me to do!
Well, thank you anyway. I'll be right on it