Parsing message field from CEF logs

Elastic Security SIEM
1

Hello all!

I am really new to this whole Elasticsearch field. I have come across problem. I have built my Elastic SIEM laboratory and I have logs from on of my other products - Secret Server. I`m receiving the logs just fine but I don't know how to parse message field.

0|Thycotic Software|Secret Server|11.1.000007|10004|SECRET - VIEW|2|msg=[[SecretServer]] Event: [Secret] Action: [View] By User: admin Item Name: JC-PS-Jump1 (Item Id: 49) Container Name: IPRO PS test (Container Id: 42)  suid=2 suser=admin cs4=admin cs4Label=suser Display Name src=10.1.1.149 rt=Feb 21 2022 13:49:45 fname=JC-PS-Jump1 fileType=Secret fileId=49 cs3Label=Folder cs3=IPRO PS test

There are a lot of fields that are separated with pipe that I would like to parse as well and write rules. For example the one field is src that is source IP :slight_smile:
Could you please help me with this problem? I guess I`m not the first one to ask this, but I could not find source with enough explanation for my noob skill.

2

I have been trying some things with GROK, but didn`t get far as there are some limitations there I can not overcome. For example I can not find how to combine multiple GROK patterns so I can get two words with space as separator.
Maybe there is a way I can just try using REGEX?

3

Hello @janis.cimins :slight_smile:

So I think there is a few things I could answer:

  1. Specifically for CEF, we already support parsing their data, if you are using filebeat we have a CEF module:
    CEF module | Filebeat Reference [8.0] | Elastic

  2. If you want to make custom ingest pipelines to add additional parsing on top, you could use the CEF processor in filebeat instead, and point it to your ingest pipeline you are building:
    Decode CEF | Filebeat Reference [8.0] | Elastic

  3. In terms of your GROK question, you can make your own custom patterns as well, if you want you can take a look at how we do it in our ingest pipelines, as they are all on github, one example would for example be Cisco, which would have plenty of examples:
    All packages:
    integrations/packages at main · elastic/integrations · GitHub
    Each package has a specific ingest folder, example for Cisco:
    integrations/default.yml at main · elastic/integrations · GitHub

The cisco example shows custom pattern definitions, that are then used with multiple patterns, at least 1 of the patterns have to match.

  1. More often than not, GROK might not even be required to parse data, for CEF example, it is usually in a Key Value pair like:
    KEY=VALUE
    We have something called a kv processor, that does all that magic for you, so you don't need to write a lot of GROK to produce the same results:
    KV processor | Elasticsearch Guide [master] | Elastic

Hope that answers some of your questions!

2 Likes
4

Hey!

Thanks!
I have CEF module enabled. It was parsing some fields and there is this message field left unparsed.
SO I`m stuck with some data OK and some left in this message field.

image
image1877×868 66 KB

About #2 I will check it out later I guess.

I have been playing with the GROK and I have some questions. For example this message:
0|Thycotic Software|Secret Server|11.1.000007|10004|SECRET - VIEW >> this can be divided into:

Thycotic Software
Secret Server
11.1.000007
10004
SECRET - VIEW

{
"the_rest": "|Secret Server|11.1.000007|10004|SECRET - VIEW|2|msg=[[SecretServer]] Event: [Secret] Action: [View] By User: admin Item Name: JC-PS-Jump1 (Item Id: 49) Container Name: IPRO PS test (Container Id: 42) suid=2 suser=admin cs4=admin cs4Label=suser Display Name src=10.1.1.149 rt=Feb 21 2022 13:49:45 fname=JC-PS-Jump1 fileType=Secret fileId=49 cs3Label=Folder cs3=IPRO PS test",
"OS_vendor": "Thycotic"
}

I have trouble splitting this like this. when I made some tries what I got was:
`|%{WORD:OS_vendor}\s+\w+%{GREEDYDATA:the_rest}

I see that for some reason I am getting the 2nd word there but it is not going under OS_vendor.

Maybe if I get this one I will then get all the fields out and get more useful data.

5

Hello,

So far I have no luck with this one - can any one help me with getting something more regards this one? As I said - I`m noob with elastic :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.