I've been on a mission the last few weeks to get ElasticStack to ingest DMARC's aggregate reports that come from remote mail domains as compressed xml files. From the onset, one of the goals I had was to build something that was as "plug n play" as possible. I'm nearly complete, just a few more items to work out, hoping someone here can guide me through these.
Completed Work
Decompression of archives
Restructuring of XML documents
Ingest/Parse into ElasticStack
Remaining Tasks
Automatic Template Setup
Automatic Index Creation
Index Pattern Formatting
Automatic Template Setup
I have a custom template already configured and functional. Outside of using Kibana's dev console to copy/paste the template, what other methods do we have?
Automatic Index Creation
Is there a way, to have the index created automatically?
Index Pattern Formatting
There are IP fields using the datatype geo_point. I have these fields, along with a few others, formatted in Kibana as URLs that redirect to a WHOIS lookup service. Is there a way to do this kind of formatting automatically?
Elasticsearch does this by default, so if you ask for index-that-doesnt-exist-2018-02 then it will create that. What that means is that if Logstash sends data to an index that isn't in Elasticsearch, it get's created and the/a template gets applied.
Do you want a dynamic lookup based on that link, or to pull that data into Elasticsearch so it's displayed alongside?
hmm....is there a standard location for template.json's in the ElasticStack directory tree?
What I mean here is, even though Elasticsearch does this, you still have to setup an index pattern in Kibana. Is there a way to automate the Kibana part.
Not quite sure what you mean, lol. The IP field is of datatype geo_point and it utilized by the geoip filter to pull down geo information. The URL formatting opens another window to a website with the field's value inside the url ( http://website/{{value}}. Looking for a way to include the URL pattern into the field's formatting automatically.
I'd suggest something like /etc/logstash/templates, which doesn't exist by default so just create it.
Ahh gotcha. The index patterns are just documents in Elasticsearch, but there's no easy way to programatically create them and it's a bit of a pain at the moment as we build out that functionality. There are a few other topics with people doing this but it's a bit of a hurdle.
Gotcha. There's no way to do that automatically either, just like the patterns above it's in a state of development.
ok, last question regarding template file. In the pipeline, when specifying the template, is there an environment variable that can be referenced instead of a hardcoded path?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.