For anyone struggling in changing the index name of your beats, I've got a solution finally

First of all

My elasticsearch version: 7.2.0
My kibana version: not important but it is 7.2.0
My packetbeat version: 8.0.0 (current master branch)

Since my goal is to adding new features to packetbeat, so I downloaded the stable version of es and kibana.

What I want

I've got two packetbeats capturing different network interfaces, say packetbeat1 capturing enp1s0f0 and packetbeat2 capturing enp1s0f1.
I want to use my customized index name to seperate them, say packetbeat1-%{[agent.version]}-%{+yyyy.MM.dd} for packetbeat1 and packetbeat2-%{[agent.version]}-%{+yyyy.MM.dd} for packetbeat2

What I did

I followed the official guide, made some changes to my packetbeat.yml file by adding these lines:

# packetbeat.yml for "packetbeat1"
output.elasticsearch.index: "packetbeat1-%{[agent.version]}-%{+yyyy.MM.dd}" "packetbeat1"
setup.template.pattern: "packetbeat1-*"

Then, I restarted my packetbeat.

What I expect

I expected my elasticsearch to be added a new index named packetbeat1.

What I got

The ouput of curl -X GET "localhost:9200/_cat/indices?v" was still (columns after index is ignored by me for simplicity):

health status index                              ...
yellow open   packetbeat-8.0.0-2019.07.12-000001 ...

After all these

I've searched the entire elastic discuss forum, checked almost every link from my top 3 Google search pages. Nothing helpful! So I turned to the hard-core method: read the f**king code.

For all the clues I got, only the ouput log from ./packetbeat test config -e inspired me:

[index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'packetbeat-8.0.0' as ILM is enabled.

What?! I told you to set the index name as packetbeat1 in the config file, and you still setting it as packetbeat-8.0.0? Let's reveal the code related:

# package
174	var alias string
175	mode := s.ilm.Mode()
176	if mode != ilm.ModeDisabled {
177		alias = s.ilm.Alias().Name
178		log.Infof("Set %v to '%s' as ILM is enabled.", cfg.PathOf("index"), alias)
179	}
180	if mode == ilm.ModeEnabled {
181		indexName = alias
182	}

There you are, the annoying flag ilm.ModeDisabled. From the code I've read, I must set setup.ilm.enabled as false to accomplish my goal. So I did it:

# packetbeat.yml for "packetbeat1"
output.elasticsearch.index: "packetbeat1-%{[agent.version]}-%{+yyyy.MM.dd}" "packetbeat1"
setup.template.pattern: "packetbeat1-*"
setup.ilm.enabled: false

And it worked like a champ!

Last thing

Is there anyone who can explain to me the reason for all these? Why the configuration of setup.ilm affects the behavior of setup.template? What's the consideration behind that?

Is the documentation misleading? If so, when will it be fixed?


Thanks for your feedback on the documentation. I have a couple of PRs that are already open to fix the documentation:

Please feel free to add review comments to the open PRs if you think the documentation needs more work.

In answer to your question, the ILM settings override setup.template because the ILM settings control the rollover alias that's used.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.