How to parse suricata rules with logstash

Elastic Stack Logstash
#1

Hi, guys,
I've been solving a "parsing" problem for a while now. First I tried it with GROK, then with KV filter. I have not been successful with either method.
So I am turning to you for help to see if anyone experienced could think of a solution.

I have this file on my disk and many other (similar)
My intention, is to reasonably parse the values that are contained in the rules

Example:

#alert tcp $EXTERNAL_NET any -> $HOME_NET 21 (msg:"ET FTP FTP CWD command attempt without login"; flow:established,to_server; flowbits:isnotset,ET.ftp.user.login; content:!"USER"; depth:4; content:"CWD"; nocase; reference:url,www.nsftools.com/tips/RawFTP.htm; reference:url,doc.emergingthreats.net/2010731; classtype:attempted-recon; sid:2010731; rev:4; metadata:created_at 2010_07_30, updated_at 2010_07_30;)

Will look like:

  • Action: alert
  • Port: tcp
  • Direction: $EXTERNAL_NET any -> $HOME_NET 21
  • Message: ET FTP FTP CWD command attempt without login
  • Flow: established,to_server
  • Flowbits: isnotset
  • Rule_Name: ET.ftp.user.login
  • ref_URL: Raw FTP Command List
  • classtype: attempted-recon
  • sid: 2010731
  • revision: 4
  • created: 2010_07_30
  • updated: 2010_07_30

FILE:
https://rules.emergingthreats.net/open/suricata/rules/emerging-ftp.rules

#2

This can be done :slight_smile: You just have to do it in a few steps to break everything apart.

First step I would use a grok pattern to match the initial stuff kind of like this:

\#%{NOTSPACE:action} %{NOTSPACE:port} %{DATA:Direction} \(%{GREEDYDATA:kv}\)

Then that will give you this:

{
  "action": [
    [
      "alert"
    ]
  ],
  "port": [
    [
      "tcp"
    ]
  ],
  "Direction": [
    [
      "$EXTERNAL_NET any -> $HOME_NET 21"
    ]
  ],
  "kv": [
    [
      "msg:"ET FTP FTP CWD command attempt without login"; flow:established,to_server; flowbits:isnotset,ET.ftp.user.login; content:!"USER"; depth:4; content:"CWD"; nocase; reference:url,www.nsftools.com/tips/RawFTP.htm; reference:url,doc.emergingthreats.net/2010731; classtype:attempted-recon; sid:2010731; rev:4; metadata:created_at 2010_07_30, updated_at 2010_07_30;"
    ]
  ]
}

Next use the kv filter on the kv field and define the semicolon and space as the splitter

kv {
   source => "kv"
   field_split_pattern => "\;\s"
   value_split => ":"
}

Then to separate the created and updated fields from the metadata field that will get created use another grok pattern to cut that up.

created_at %{NOTSPACE:created} updated_at %{NOTSPACE:created};

Then finally you can use a date filter to properly change the created and updated fields to real timestamps

#3

Thank you for your reply,
I'm finally trying to solve the problem another way. I mean, only halfway.

Now I have a problem with the GROK patter
Example:

alert tcp $EXTERNAL_NET any -> $SMTP_SERVERS 25 (msg:"EXPLOIT-KIT BottleEK landing page detected"; flow:to_server,established; file_data; content:"new RegExp|28 27 4D 53 49 45 5C 78 32 30 28 5C 78 35 63 64 2B 5C 78 35 63 2E 5C 78 35 63 64 2B 29 3B 27 29 3B|"; fast_pattern:only; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service smtp; reference:url,www.virustotal.com/file/588bb25acf86ac18323d800372bbdc0eb89ba3ce80ed3d891a9c41b8db93df26/analysis/; classtype:trojan-activity; sid:52587; rev:1;)

And my GROK from ELK debugger is currently working...

%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{WORD:dest_port}%{DATA}msg\:"%{DATA:msg}"%{DATA}sid\:%{NUMBER:SID}; rev\:%{NUMBER:revision};

Another problem is, that this GROK didn't want accept my logstash.

ERROR:

root@ubu-srv-101:/usr/share/logstash# ./bin/logstash -f /etc/logstash/conf.d/suricata.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-01-06 12:55:26.674 [main] runner - Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[WARN ] 2022-01-06 12:55:27.123 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-01-06 12:55:28.943 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[ERROR] 2022-01-06 12:55:29.726 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 11, column 151 (byte 313) after filter {\n  grok {\n    patterns_dir => [\"./patterns\"]\n    match => { \"message\" => \"^%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{WORD:dest_port}%{DATA}msg\\:\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[INFO ] 2022-01-06 12:55:29.846 [LogStash::Runner] runner - Logstash shut down.
#4

You have an error in your configuration, your pipeline is not starting.

"Expected one of [ \t\r\n], "#", "{", "}" at line 11, column 151

This message tells that something is wrong in your configuration near line 11, probably a filter curly or square bracket was not closed, please check your configuration or share your full configuration file.

#5

Of course I gladly can.

input {
  file {
    path => ["/mnt/SURICATA/active/*.rules"]
    mode => "read"
  }
}

filter {
  grok {
    patterns_dir => ["./patterns"]
    match => { "message" => "^%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{WORD:dest_port}%{DATA}msg\:"%{DATA:msg2}"%{DATA}sid\:%{NUMBER:SID}; rev\:%{NUMBER:revision}" }
       }
  mutate {
    remove_field => ["host"]
         }
}
 

output {
  stdout {
    codec => rubydebug }
  elasticsearch {
      hosts => ["https://XXX.XX.XX.XX:9200"]
      user => elastic
      password => "XXXX"
      ssl => true
      cacert => '/etc/logstash/elasticsearch-ca.pem'
    index => "suricata"
  }
}

I already checked this, but I didn't see anything wrong...

#6

Didn't see anything wrong also, but maybe escaping the colon, :, is interfering in something.

There is no need to escape the colon, so I would change all \: to : and try again.

#7

The colons don't need escaped but the quotation marks around %{DATA:msg2} do need to be escaped.

#8

Yeah thanks both of us @twilson and @leandrojmp . It's helped me.
Now it's work correctly.

With DATA:

alert ip $HOME_NET any -> X.X.X.X any (msg: "MISP e1328 [] Outgoing To IP: X.X.X.X";   classtype:trojan-activity; sid:19923821; rev:1; priority:3; reference:url,https://misp-url/events/view/1324;)

and GROK:

%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{DATA:dest_port} \(%{DATA} %{DATA:msg}\;%{SPACE}%{SPACE}%{SPACE} %{WORD}\:%{DATA:classtype}\;%{SPACE} %{WORD}\:%{NUMBER:sid}\;%{SPACE} %{WORD}\:%{NUMBER:rev}\;%{SPACE} %{WORD}\:%{NUMBER:priority}\;%{SPACE} %{WORD}\:%{WORD}\,%{DATA:url}\;\)

Now, need to edit just few other rule data types and it will be O.k.
Thanks !

#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.