I finally managed to find the time to run a little verification experiment. I'm sorry for the delay.
It turned out that, on version
7.4.0, I had to do a little bit more to get beat output to land in different indexes. I used
filebeat for simplicity, but I think the configuration for output should be similar.
I made two configuration files,
filebeat1.yml looked like this:
- type: log
- add_host_metadata: ~
- add_cloud_metadata: ~
It turned out that because my backing Elasticsearch cluster had ILM (index lifecycle management) enabled, I had to set
false. For the second file, I changed
setup.template.pattern to reference
index2 rather than
index1, and pointed at a different file.
With all of this in place, I was able to run the following commands and have filebeat put the lines of my file in different indices:
./filebeat -c filebeat1.yml --path.data data1 run -e
./filebeat -c filebeat2.yml --path.data data2 run -e
I don't think you will need to worry about the
--path.data flag; I only used it because I was running two filebeat processes on the same machine.
To get a custom index name working with ILM enabled, I used different settings.
If you're on a recent version of the Elasticsearch stack, this ILM feature might be convenient.
I'm sorry for taking so long to get back to you, but I hope this information is helpful.