# DMARC in ElasticStack **URL:** \ **Category:** Logstash\ **Created:** [February 18, 2018, 11:20pm UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375 "2018-02-18T23:20:00Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [February 18, 2018, 11:20pm UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/1 "2018-02-18T23:20:00Z")
I've been on a mission the last few weeks to get ElasticStack to ingest DMARC's aggregate reports that come from remote mail domains as compressed xml files. From the onset, one of the goals I had was to build something that was as "plug n play" as possible. I'm nearly complete, just a few more items to work out, hoping someone here can guide me through these. **Completed Work** - Decompression of archives - Restructuring of XML documents - Ingest/Parse into ElasticStack **Remaining Tasks** - Automatic Template Setup - Automatic Index Creation - Index Pattern Formatting **Automatic Template Setup** I have a custom template already configured and functional. Outside of using Kibana's dev console to copy/paste the template, what other methods do we have? **Automatic Index Creation** Is there a way, to have the index created automatically? **Index Pattern Formatting** There are IP fields using the datatype geo\_point. I have these fields, along with a few others, formatted in Kibana as URLs that redirect to a WHOIS lookup service. Is there a way to do this kind of formatting automatically? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 18, 2018, 11:29pm UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/2 "2018-02-18T23:29:41Z")
> [@wwalker](#): > > Automatic Template Setup You can put it into a file and then refer to it in the Elasticsearch output definition - [Elasticsearch output plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template) > [@wwalker](#): > > Automatic Index Creation Elasticsearch does this by default, so if you ask for `index-that-doesnt-exist-2018-02` then it will create that. What that means is that if Logstash sends data to an index that isn't in Elasticsearch, it get's created and the/a template gets applied. > [@wwalker](#): > > Index Pattern Formatting Do you want a dynamic lookup based on that link, or to pull that data into Elasticsearch so it's displayed alongside? ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [February 18, 2018, 11:37pm UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/3 "2018-02-18T23:37:32Z")
> [@warkolm](#): > > You can put it into a file and then refer to it in the Elasticsearch output definition - [Elasticsearch output plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template) hmm....is there a standard location for template.json's in the ElasticStack directory tree? > [@warkolm](#): > > Elasticsearch does this by default, so if you ask for `index-that-doesnt-exist-2018-02` then it will create that. What that means is that if Logstash sends data to an index that isn't in Elasticsearch, it get's created and the/a template gets applied. What I mean here is, even though Elasticsearch does this, you still have to setup an index pattern in Kibana. Is there a way to automate the Kibana part. > [@warkolm](#): > > Do you want a dynamic lookup based on that link, or to pull that data into Elasticsearch so it's displayed alongside? Not quite sure what you mean, lol. The IP field is of datatype geo\_point and it utilized by the geoip filter to pull down geo information. The URL formatting opens another window to a website with the field's value inside the url `( http://website/{{value}}`. Looking for a way to include the URL pattern into the field's formatting automatically. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 18, 2018, 11:44pm UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/4 "2018-02-18T23:44:25Z")
> [@wwalker](#): > > hmm....is there a standard location for template.json's in the ElasticStack directory tree? I'd suggest something like `/etc/logstash/templates`, which doesn't exist by default so just create it. > [@wwalker](#): > > What I mean here is, even though Elasticsearch does this, you still have to setup an index pattern in Kibana. Is there a way to automate the Kibana part. Ahh gotcha. The index patterns are just documents in Elasticsearch, but there's no easy way to programatically create them and it's a bit of a pain at the moment as we build out that functionality. There are a few other topics with people doing this but it's a bit of a hurdle. > [@wwalker](#): > > The URL formatting opens another window to a website with the field's value inside the url ( [http://website/{{value}](http://website/%7B%7Bvalue%7D)}. Looking for a way to include the URL pattern into the field's formatting automatically. Gotcha. There's no way to do that automatically either, just like the patterns above it's in a state of development. ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [February 19, 2018, 1:40am UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/5 "2018-02-19T01:40:09Z")
ok, last question regarding template file. In the pipeline, when specifying the template, is there an environment variable that can be referenced instead of a hardcoded path? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 19, 2018, 1:42am UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/6 "2018-02-19T01:42:56Z")
Yep, [https://www.elastic.co/guide/en/logstash/current/environment-variables.html](https://www.elastic.co/guide/en/logstash/current/environment-variables.html) sounds like what you want. ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [February 19, 2018, 1:48am UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/7 "2018-02-19T01:48:45Z")
blah...doesn't look like Windows configures any environment variables for Logstash's root directory, oh well. Thanks for all the help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 19, 2018, 1:49am UTC](https://discuss.elastic.co/t/dmarc-in-elasticstack/120375/8 "2018-03-19T01:49:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.