Loading Index Template

[msauter]

Hello. Completely new to Elastic as well as this form. Initially working with auditbeat. Using 7.4. I am planning to send all of the captured audit information directly to Logstash first.

Since I’ll use Logstash to perform additional processing on the data collected by Auditbeat, I will not need to send any data directly to Elasticsearch, so I disabled that output
(output.elasticsearch) in the auditbeat config file (auditbeat.yml) by commenting it all out. At the same time, I configured the output.logstash section by uncommenting the lines output.logstash. I was assuming both should never coexist.

As I read the documentation in elastic.co related to "Loading the Elastic Template into Elasticsearch," I read, " By default, Auditbeat automatically loads the recommended template file, fields.yml , if the Elasticsearch output is enabled. If you want to use the default index template, no additional configuration is required. Otherwise, you can change the defaults in the auditbeat.yml config file..."

So I have enabled the Logstash output while disabling the elasticsearch output, so I am no longer going the 'default' route in terms of the index template which utilizes fields.yml. This is where my question (or two) come(s) in.

1.) Can I still use the same fields.yml template file even though I enabled Logstash output/disabled elasticsearch output?
2.) Regardless of the answer to #1, HOW do I load this index template into Elasticsearch when enabling Logstash output? Do I have to put any new/change any existing lines in the auditbeat.yml file like...
EX.)
setup.template.name: "my_new_index_template_name"
setup.template.fields: "path/to/my/new/fields.yml")
any other dictionary field/value pair settings???

• OR -
  Do nothing more and Elastic will still use the same fields.yml file/location that it would use if I only had just enabled output.elasticsearch (the default) instead?

Sorry this is so long, but just trying to be thorough.

Thank you.

Matt

[dedemorton]

You can use the default Beats template with Logstash. Either load the template before you disable the elasticsearch output, or override the output settings at the command line when you run the setup command. For example:

 ./auditbeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

Note that you can use -E <setting>=<value> to override any config settings.

See the docs about manual template loading for more info.

Next, set up Logstash to use the template. In the Logstash pipeline definition, set the index name based on metadata passed into Logstash by the beats input. For example:

input {
  beats {
    port => 5044
  }
}

. . . 

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }
}

Or use %{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd} if you're creating daily indices (and not using index lifecycle management).

See the docs for the beats input plugin for more detail.

Logstash is very powerful but adds complexity to your config. Make sure you've checked the list of Beats processors to see if you can achieve your processing goals without Logstash before adding it to your processing pipeline.