Suricata Redis>ELK Stack Mapping help please

Elastic Security
1

I am running Suricata on PFSense and my goal is to create a Network Security Monitoring Dashboard with world map. Likely in Grafana.

I use FreeNAS with ELK setup in a FreeBSD jail. My elasticsearch is 7.7.1 and did not install with the filebeat suricata module (i assume because open source license edition) so i cannot just run the filebeat suricata module setup.

I have Suricata sending the alert logs to Redis > Logstash and in to Elasticsearch. A daily index is created. index => "logstash-suricata-%{+YYYY.MM.dd}". That is all working fine!

However i am having trouble with elasticsearch mapping. It was suggested I map existing fields to ECS fields, which would then allow me to easily create Kibana dashboards. Could someone please guide me on how to map my existing setup to ECS fields?

Also, as i am creating new elasticsearch index's each day, how to I create mappings for each daily index?

Logstash Config 
input {
  redis {
    host => "192.168.188.***"
    data_type => "list"
    key => "suricata"
    codec => "json"
    threads => 12
    type => "SuricataIDPS"
  }
}
filter {
  if [type] == "SuricataIDPS" {
    date {
      match => [ "timestamp", "ISO8601" ]
    }
  }
  if [src_ip] {
    geoip {
      source => "src_ip"
      target => "src_geoip"
    }
  }
  if [dest_ip] {
    geoip {
      source => "dest_ip"
      target => "dest_geoip"
    }
  }
}
output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-suricata-%{+YYYY.MM.dd}"
    template => "/usr/local/etc/logstash/suricata_template.json"
    template_overwrite => true
  }
}

I don't know what this template does or if it is even working..

suricata_template.json 
{
  "template" : "logstash-suricata-*",
  "settings" : {
    "index.refresh_interval" : "5s",
    "number_of_replicas": 0,
    "number_of_shards": 1
},
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : false, "norms" : false},
      "dynamic_templates" : [ {
        "double_fields" : {
          "match" : "*",
          "match_mapping_type" : "double",
          "mapping" : { "type" : "double"}
        }
      }, {
        "long_fields" : {
          "match" : "*",
          "match_mapping_type" : "long",
          "mapping" : { "type" : "long", "doc_values" : true }
        }
      }, {
        "date_fields" : {
          "match" : "*",
          "match_mapping_type" : "date",
          "mapping" : { "type" : "date", "doc_values" : true }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date", "doc_values" : true },
        "@version": { "type": "text", "index": false },
        "dest_geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip", "doc_values" : true },
            "location" : { "type" : "geo_point", "doc_values" : true },
            "latitude" : { "type" : "double", "doc_values" : true },
            "longitude" : { "type" : "double", "doc_values" : true }
          }
        },
        "src_geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip", "doc_values" : true },
            "location" : { "type" : "geo_point", "doc_values" : true },
            "latitude" : { "type" : "double", "doc_values" : true },
            "longitude" : { "type" : "double", "doc_values" : true }
          }
        }
      }
    }
  }
}
2

Heya @bn1980 thanks for the post. There's a lot in there, so let me try to provide some answers.

I am running Suricata on PFSense and my goal is to create a Network Security Monitoring Dashboard with world map. Likely in Grafana.

Just curious, any reason you're not planning to use Kibana instead of Grafana?

My elasticsearch is 7.7.1 and did not install with the filebeat suricata module (i assume because open source license edition) so i cannot just run the filebeat suricata module setup.

The filebeat suricata module could be the fastest way to get you going. Is there a reason you can't install the free default distribution of Elasticsearch in your environment? If you do that, then you can run the module setup to create the mapping you need. In addition, the filebeat suricata module performs the ECS normalization that would allow you to take advantage of other free features included in Elastic's default distribution, such as the SIEM/security app in Kibana.

It was suggested I map existing fields to ECS fields, which would then allow me to easily create Kibana dashboards. Could someone please guide me on how to map my existing setup to ECS fields?

Suricata populates a lot of fields. If you can't use the Filebeat Suricata module, and you want to to use Logstash, you'll need to modify your Logstash configs to copy or rename the original Suricata fields to the corresponding ECS fields. Before attempting this yourself, you might consider checking with some other open source projects that use Suricata and ECS, such as http://rocknsm.io/

Also, as i am creating new elasticsearch index's each day, how to I create mappings for each daily index?

Elasticsearch index templates allow you to specify an index pattern with wildcard characters that can span a set of daily indices. So your index template might specify something like logstash-suricata-* which would match each day's index name. See Index templates | Elasticsearch Guide [7.9] | Elastic for more details.