# How to parse suricata rules with logstash

**URL:** <https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548>\
**Category:** Logstash\
**Created:** [December 21, 2021, 10:32am UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548 "2021-12-21T10:32:41Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![karpaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karpaz/32/95602_2.png) [@karpaz](https://discuss.elastic.co/u/karpaz)\
**Post date:** [December 21, 2021, 10:32am UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/1 "2021-12-21T10:32:41Z")

</div>

Hi, guys,  
I've been solving a "parsing" problem for a while now. First I tried it with GROK, then with KV filter. I have not been successful with either method.  
So I am turning to you for help to see if anyone experienced could think of a solution.

I have this file on my disk and many other (similar)  
My intention, is to reasonably parse the values that are contained in the rules

**Example** :

```auto
#alert tcp $EXTERNAL_NET any -> $HOME_NET 21 (msg:"ET FTP FTP CWD command attempt without login"; flow:established,to_server; flowbits:isnotset,ET.ftp.user.login; content:!"USER"; depth:4; content:"CWD"; nocase; reference:url,www.nsftools.com/tips/RawFTP.htm; reference:url,doc.emergingthreats.net/2010731; classtype:attempted-recon; sid:2010731; rev:4; metadata:created_at 2010_07_30, updated_at 2010_07_30;)

```

**Will look like** :

> - Action: alert
> - Port: tcp
> - Direction: $EXTERNAL\_NET any -\> $HOME\_NET 21
> - Message: ET FTP FTP CWD command attempt without login
> - Flow: established,to\_server
> - Flowbits: isnotset
> - Rule\_Name: ET.ftp.user.login
> - ref\_URL: [Raw FTP Command List](http://www.nsftools.com/tips/RawFTP.htm)
> - classtype: attempted-recon
> - sid: 2010731
> - revision: 4
> - created: 2010\_07\_30
> - updated: 2010\_07\_30

**FILE** :  
[https://rules.emergingthreats.net/open/suricata/rules/emerging-ftp.rules](https://rules.emergingthreats.net/open/suricata/rules/emerging-ftp.rules)

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [December 21, 2021, 2:38pm UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/2 "2021-12-21T14:38:52Z")

</div>

This can be done 🙂 You just have to do it in a few steps to break everything apart.

First step I would use a grok pattern to match the initial stuff kind of like this:

`\#%{NOTSPACE:action} %{NOTSPACE:port} %{DATA:Direction} \(%{GREEDYDATA:kv}\)`

Then that will give you this:

```auto
{
  "action": [
    [
      "alert"
    ]
  ],
  "port": [
    [
      "tcp"
    ]
  ],
  "Direction": [
    [
      "$EXTERNAL_NET any -> $HOME_NET 21"
    ]
  ],
  "kv": [
    [
      "msg:"ET FTP FTP CWD command attempt without login"; flow:established,to_server; flowbits:isnotset,ET.ftp.user.login; content:!"USER"; depth:4; content:"CWD"; nocase; reference:url,www.nsftools.com/tips/RawFTP.htm; reference:url,doc.emergingthreats.net/2010731; classtype:attempted-recon; sid:2010731; rev:4; metadata:created_at 2010_07_30, updated_at 2010_07_30;"
    ]
  ]
}

```

Next use the kv filter on the `kv` field and define the semicolon and space as the splitter

```auto
kv {
   source => "kv"
   field_split_pattern => "\;\s"
   value_split => ":"
}

```

Then to separate the `created` and `updated` fields from the metadata field that will get created use another grok pattern to cut that up.

`created_at %{NOTSPACE:created} updated_at %{NOTSPACE:created};`

Then finally you can use a [date filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html) to properly change the `created` and `updated` fields to real timestamps

---

<div class="post-metadata">

**Author:** ![karpaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karpaz/32/95602_2.png) [@karpaz](https://discuss.elastic.co/u/karpaz)\
**Post date:** [January 6, 2022, 12:56pm UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/3 "2022-01-06T12:56:42Z")

</div>

Thank you for your reply,  
I'm finally trying to solve the problem another way. I mean, only halfway.

Now I have a problem with the GROK patter  
**Example** :

```auto
alert tcp $EXTERNAL_NET any -> $SMTP_SERVERS 25 (msg:"EXPLOIT-KIT BottleEK landing page detected"; flow:to_server,established; file_data; content:"new RegExp|28 27 4D 53 49 45 5C 78 32 30 28 5C 78 35 63 64 2B 5C 78 35 63 2E 5C 78 35 63 64 2B 29 3B 27 29 3B|"; fast_pattern:only; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service smtp; reference:url,www.virustotal.com/file/588bb25acf86ac18323d800372bbdc0eb89ba3ce80ed3d891a9c41b8db93df26/analysis/; classtype:trojan-activity; sid:52587; rev:1;)

```

And my GROK from ELK debugger is currently working...

```auto
%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{WORD:dest_port}%{DATA}msg\:"%{DATA:msg}"%{DATA}sid\:%{NUMBER:SID}; rev\:%{NUMBER:revision};

```

Another problem is, that this GROK didn't want accept my logstash.

ERROR:

```auto
root@ubu-srv-101:/usr/share/logstash# ./bin/logstash -f /etc/logstash/conf.d/suricata.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO] 2022-01-06 12:55:26.674 [main] runner - Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[WARN] 2022-01-06 12:55:27.123 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO] 2022-01-06 12:55:28.943 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[ERROR] 2022-01-06 12:55:29.726 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"{\", \"}\" at line 11, column 151 (byte 313) after filter {\n grok {\n patterns_dir => [\"./patterns\"]\n match => { \"message\" => \"^%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{WORD:dest_port}%{DATA}msg\\:\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[INFO] 2022-01-06 12:55:29.846 [LogStash::Runner] runner - Logstash shut down.

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 6, 2022, 1:21pm UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/4 "2022-01-06T13:21:17Z")

</div>

You have an error in your configuration, your pipeline is not starting.

> "Expected one of [\t\r\n], "#", "{", "}" at line 11, column 151

This message tells that something is wrong in your configuration near line 11, probably a filter curly or square bracket was not closed, please check your configuration or share your full configuration file.

---

<div class="post-metadata">

**Author:** ![karpaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karpaz/32/95602_2.png) [@karpaz](https://discuss.elastic.co/u/karpaz)\
**Post date:** [January 6, 2022, 2:48pm UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/5 "2022-01-06T14:48:05Z")

</div>

Of course I gladly can.

```auto
input {
  file {
    path => ["/mnt/SURICATA/active/*.rules"]
    mode => "read"
  }
}

filter {
  grok {
    patterns_dir => ["./patterns"]
    match => { "message" => "^%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{WORD:dest_port}%{DATA}msg\:"%{DATA:msg2}"%{DATA}sid\:%{NUMBER:SID}; rev\:%{NUMBER:revision}" }
       }
  mutate {
    remove_field => ["host"]
         }
}
 

output {
  stdout {
    codec => rubydebug }
  elasticsearch {
      hosts => ["https://XXX.XX.XX.XX:9200"]
      user => elastic
      password => "XXXX"
      ssl => true
      cacert => '/etc/logstash/elasticsearch-ca.pem'
    index => "suricata"
  }
}

```

I already checked this, but I didn't see anything wrong...

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [January 6, 2022, 3:17pm UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/6 "2022-01-06T15:17:51Z")

</div>

Didn't see anything wrong also, but maybe escaping the colon, `:`, is interfering in something.

There is no need to escape the colon, so I would change all `\:` to `:` and try again.

---

<div class="post-metadata">

**Author:** ![twilson](https://avatars.discourse-cdn.com/v4/letter/t/2bfe46/32.png) [@twilson](https://discuss.elastic.co/u/twilson)\
**Post date:** [January 6, 2022, 3:39pm UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/7 "2022-01-06T15:39:57Z")

</div>

The colons don't need escaped but the quotation marks around %{DATA:msg2} do need to be escaped.

---

<div class="post-metadata">

**Author:** ![karpaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karpaz/32/95602_2.png) [@karpaz](https://discuss.elastic.co/u/karpaz)\
**Post date:** [January 11, 2022, 11:58am UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/8 "2022-01-11T11:58:54Z")

</div>

Yeah thanks both of us @twilson and @leandrojmp . It's helped me.  
Now it's work correctly.

With **DATA** :

```auto
alert ip $HOME_NET any -> X.X.X.X any (msg: "MISP e1328 [] Outgoing To IP: X.X.X.X"; classtype:trojan-activity; sid:19923821; rev:1; priority:3; reference:url,https://misp-url/events/view/1324;) 

```

and **GROK** :

```auto
%{DATA:action} %{DATA:type} %{DATA:src_dir} %{DATA:src_port} %{DATA:dir} %{DATA:dest_dir} %{DATA:dest_port} \(%{DATA} %{DATA:msg}\;%{SPACE}%{SPACE}%{SPACE} %{WORD}\:%{DATA:classtype}\;%{SPACE} %{WORD}\:%{NUMBER:sid}\;%{SPACE} %{WORD}\:%{NUMBER:rev}\;%{SPACE} %{WORD}\:%{NUMBER:priority}\;%{SPACE} %{WORD}\:%{WORD}\,%{DATA:url}\;\)

```

Now, need to edit just few other rule data types and it will be O.k.  
Thanks !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2022, 11:59am UTC](https://discuss.elastic.co/t/how-to-parse-suricata-rules-with-logstash/292548/9 "2022-02-08T11:59:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.