DMARC in ElasticStack


(Walker) #1

I've been on a mission the last few weeks to get ElasticStack to ingest DMARC's aggregate reports that come from remote mail domains as compressed xml files. From the onset, one of the goals I had was to build something that was as "plug n play" as possible. I'm nearly complete, just a few more items to work out, hoping someone here can guide me through these.

Completed Work

  • Decompression of archives
  • Restructuring of XML documents
  • Ingest/Parse into ElasticStack

Remaining Tasks

  • Automatic Template Setup
  • Automatic Index Creation
  • Index Pattern Formatting

Automatic Template Setup
I have a custom template already configured and functional. Outside of using Kibana's dev console to copy/paste the template, what other methods do we have?

Automatic Index Creation
Is there a way, to have the index created automatically?

Index Pattern Formatting
There are IP fields using the datatype geo_point. I have these fields, along with a few others, formatted in Kibana as URLs that redirect to a WHOIS lookup service. Is there a way to do this kind of formatting automatically?


(Mark Walkom) #2

You can put it into a file and then refer to it in the Elasticsearch output definition - https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template

Elasticsearch does this by default, so if you ask for index-that-doesnt-exist-2018-02 then it will create that. What that means is that if Logstash sends data to an index that isn't in Elasticsearch, it get's created and the/a template gets applied.

Do you want a dynamic lookup based on that link, or to pull that data into Elasticsearch so it's displayed alongside?


(Walker) #3

hmm....is there a standard location for template.json's in the ElasticStack directory tree?

What I mean here is, even though Elasticsearch does this, you still have to setup an index pattern in Kibana. Is there a way to automate the Kibana part.

Not quite sure what you mean, lol. The IP field is of datatype geo_point and it utilized by the geoip filter to pull down geo information. The URL formatting opens another window to a website with the field's value inside the url ( http://website/{{value}}. Looking for a way to include the URL pattern into the field's formatting automatically.


(Mark Walkom) #4

I'd suggest something like /etc/logstash/templates, which doesn't exist by default so just create it.

Ahh gotcha. The index patterns are just documents in Elasticsearch, but there's no easy way to programatically create them and it's a bit of a pain at the moment as we build out that functionality. There are a few other topics with people doing this but it's a bit of a hurdle.

Gotcha. There's no way to do that automatically either, just like the patterns above it's in a state of development.


(Walker) #5

ok, last question regarding template file. In the pipeline, when specifying the template, is there an environment variable that can be referenced instead of a hardcoded path?


(Mark Walkom) #6

Yep, https://www.elastic.co/guide/en/logstash/current/environment-variables.html sounds like what you want.


(Walker) #7

blah...doesn't look like Windows configures any environment variables for Logstash's root directory, oh well. Thanks for all the help!


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.