Custom configuration for new Beat

[Michael_Albers]

I'm creating a new Beat from the Developer Guide (https://www.elastic.co/guide/en/beats/devguide/master/index.html). The Beat is functioning well, but I would like to do some final configuration. I've looked all over and have found nothing about how to do this. I'm mainly interested in how to change the index name template the Beat is using and how to have the Beat supply information for Elasticsearch to create an index alias for the indexes the Beat will be writing to.

The Beat is always creating indexes named <beatname>-7.0.0-alpha1-yyyy.MM.dd and there is no alias. What do I need to do to override the index name template and create an alias? Thanks.

[pierhugues (Pier-Hugues Pellerin)]

Hello @Michael_Albers,

You will want to override the elasticsearch index option and I believe you will need to use the cfg low level api to do that.

cfg.SetString("output.elasticsearch.index", "myindex")

You might want to check for it before to make sure you don't override the user setting.

cfg.String("output.elasticsearch.index")

We use a similar flow to support Elastic's cloud cloud.id you might want to check it out.

Concerning creating an alias over multiple indices we never have a need for that in any of our beats, we usually query over wilcards defined indices like filebeat-* which gives a similar feel on the UI on an alias on multiple indices.

I am curious of your use case here?

Thanks.

[Michael_Albers]

@pierhugues,

Thanks for the quick response. I added the following snippet to my New function, after the cfg.Unpack call:

cfg.SetString("output.elasticsearch.index", -1, "test_index")

(SetString wanted an integer as the second parameter and other places which called that function used -1.)

SetString returned no error, however, the default index naming scheme was used.

The use case is pretty simple. The Beat is going to be doing its thing and another service I'm running will then query Elastic to get an average or percentile of the data produced by the Beat. The service will then base some request parameters on the average/percentile. The data generated by the Beat will be retained for a substantial amount of time, probably 1 year, so there will be many indexes.

I can probably live without the alias. I'd like to change the index names to closer match some other index naming conventions that are used. It would also be nice to have a version number in the index name that I can control.

[Michael_Albers]

@pierhugues

I've been playing around with the index settings further and here's what I've found.

I added the following to _meta/beat.yml

output.elasticsearch:
  index: test_index

And the corresponding setup.template keys to _meta/beat.yml and ran make update. This changed the name of the template being used. However, the indexes still used the default naming. In playing around with the generated config file further, I found that the use of the output.elasticsearch key in the _meta/config.yml in libbeat (see lines 60-62 at https://github.com/elastic/beats/blob/master/libbeat/_meta/config.yml) is essentially erasing the value I had set for index.

When I manually merged the two output.elasticsearch entries into

output.elasticsearch:
  index: test_index
  hosts: ["localhost:9200"]

I was able to get the index names I specified.

That begs the larger question, is this a bug in the libbeat YAML parsing? If not, how is one supposed to override any of these pre-defined values through the prescribed make setup; make procedure for building a custom Beat?