DMARC in ElasticStack

I've been on a mission the last few weeks to get ElasticStack to ingest DMARC's aggregate reports that come from remote mail domains as compressed xml files. From the onset, one of the goals I had was to build something that was as "plug n play" as possible. I'm nearly complete, just a few more items to work out, hoping someone here can guide me through these.

Completed Work

  • Decompression of archives
  • Restructuring of XML documents
  • Ingest/Parse into ElasticStack

Remaining Tasks

  • Automatic Template Setup
  • Automatic Index Creation
  • Index Pattern Formatting

Automatic Template Setup
I have a custom template already configured and functional. Outside of using Kibana's dev console to copy/paste the template, what other methods do we have?

Automatic Index Creation
Is there a way, to have the index created automatically?

Index Pattern Formatting
There are IP fields using the datatype geo_point. I have these fields, along with a few others, formatted in Kibana as URLs that redirect to a WHOIS lookup service. Is there a way to do this kind of formatting automatically?

1 Like

You can put it into a file and then refer to it in the Elasticsearch output definition - Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

Elasticsearch does this by default, so if you ask for index-that-doesnt-exist-2018-02 then it will create that. What that means is that if Logstash sends data to an index that isn't in Elasticsearch, it get's created and the/a template gets applied.

Do you want a dynamic lookup based on that link, or to pull that data into Elasticsearch so it's displayed alongside? there a standard location for template.json's in the ElasticStack directory tree?

What I mean here is, even though Elasticsearch does this, you still have to setup an index pattern in Kibana. Is there a way to automate the Kibana part.

Not quite sure what you mean, lol. The IP field is of datatype geo_point and it utilized by the geoip filter to pull down geo information. The URL formatting opens another window to a website with the field's value inside the url ( http://website/{{value}}. Looking for a way to include the URL pattern into the field's formatting automatically.

I'd suggest something like /etc/logstash/templates, which doesn't exist by default so just create it.

Ahh gotcha. The index patterns are just documents in Elasticsearch, but there's no easy way to programatically create them and it's a bit of a pain at the moment as we build out that functionality. There are a few other topics with people doing this but it's a bit of a hurdle.

Gotcha. There's no way to do that automatically either, just like the patterns above it's in a state of development.

ok, last question regarding template file. In the pipeline, when specifying the template, is there an environment variable that can be referenced instead of a hardcoded path?

Yep, sounds like what you want.

1 Like

blah...doesn't look like Windows configures any environment variables for Logstash's root directory, oh well. Thanks for all the help!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.