Filter cef

Elastic Stack Logstash
1

Hi,
in the logstash reference is no cef filter plugin but there is a codec cef which i use for my input.

input { stdin { codec => cef } }
filter {}
output { stdout { codec => rubydebug } }

This is working fine but i would need to target a specific field (cef_message). Does anybody know how i can use this as/like filter plugin like json in this example:

filter {
       json { source => "cef_message" }
}

I had already the idea to send it to another pipeline to use codec there in the input but then i loose the other fields.

I'm thankful for any help.

2

My experience receiving messages in which one field is cef encoded is that the cef codec is not useful on the initial input.

If you are going to end up in elasticsearch, then you can decide the final document id, parse out the cef, append the document id as another cef field, then route it to another pipeline that uses a cef codec. In the outputs of both pipelines set the document_id option and enabled upsert so that the documents are merged.

1 Like
3

Hmm, maybe there could be another approach to the problem but I wonder if that is even feasible.

Pipeline 1:

  • you read the file as json
  • in a filter section you extract the cef data and assign it to a field
  • encode the whole message as base64 and append it to the cef data at the end
  • prune the fields you do not need and allow only the merged cef + base64 go over
  • output that to the pipeline number 2.

Pipeline 2:

  • read from the pipeline 1 output
  • LIMITATION: the input is going to be: "your_data": "cef + b64 stuff". I do not know if you can control the pipeline communication to be a plain line?

Otherwise you would have to use something in the middle to accomplish that, like output this cef to a file, and another pipeline reads that file per your configured delimiter.

Or like Badger said, elasticsearch and merge as upsert.
Or another solution - memcached to store your b64 and recover in the 2nd line (with small ttl to decay). But how to output the text now from the 1st pipeline to another pipeline to enter as a format CEF.

Good luck.

1 Like
4

Is there a way to ingest that data as cef, not as a json? Because from my testing it looks that the pipeline to pipeline communication is only in JSON.

5

I was assuming that pipeline inputs and outputs would respect the codec if the user configured one, but that appears not to be the case. However, it works with a tcp input and output

input { generator { count => 1 lines => [ 'Foo' ] } }
filter {
    mutate { add_field => { cefField => "Jan 18 11:07:53 dsmhost CEF:0|Trend Micro|Deep Security Manager|<DSM version>|600|Administrator Signed In|4|suser=Master randomPiggyBackedData=a1s2d3f4g5" } }
}
output { tcp { host => "localhost" port => 12888 codec => plain { format => "%{cefField}" } } }

and

input { tcp { host => localhost port => 12888 codec => cef } }
output { stdout {} }

gets me

           "cefVersion" => "0",
        "deviceVersion" => "<DSM version>",
           "@timestamp" => 2019-05-16T14:27:44.576Z,
   "deviceEventClassId" => "600",
                 "port" => 48406,
       "sourceUserName" => "Master",
                 "name" => "Administrator Signed In",
"randomPiggyBackedData" => "a1s2d3f4g5",
               "syslog" => "Jan 18 11:07:53 dsmhost",
         "deviceVendor" => "Trend Micro",
                 "host" => "localhost",
             "@version" => "1",
        "deviceProduct" => "Deep Security Manager",
             "severity" => "4"

Note the randomPiggyBackedData, which could be all of the other fields encoded. As charlie suggested, you could then decode in the second pipeline so you do not have to join different events together.

2 Likes
6

Hey.
Yes, this is exactly what I was doing now.

I forgot to update the post.
That is the only method.

Thanks!
Hope that it helps!

7

Thank you guys for the solution, the working config does now the following steps:

  • base64encode the whole message
  • extract the cef part of the message
  • append the cef message with the base64 string as a cef conform field
  • output tcp the raw cef
  • input tcp the raw cef and apply codec cef
  • base64decode the original message
    done

Thx again :wink:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.