Create a new index

[mouna_boujrad]

hello, I installed packetbeat into two machines i want to have different indexes for each packetbeat agent, can I do that ?

to clarify more I installed a machine with elasticsearch kibana and packetbeat, I am getting all beats from that packetbeat, then I created new machine and installed only a packetbeat and point it to the elasticsearch on the first machine , when I run packetbeat setup and packetbeat -e it seems that it works fine but in the index management I just see one index called packetbeat how can I separete the index generated by the first packetbeat agent and the second one and instead of getting packetbeat-* get for example packetbeat1-* and packetbeat2-* just an example.

could anyone help me please

[William_Brafford]

Hi, Mouna!

I believe you need to configure the output for packetbeat on each machine — at least, that will be the simplest solution. Here is a guide to doing that:

https://www.elastic.co/guide/en/beats/packetbeat/current/elasticsearch-output.html#index-option-es

Note that you will need to adjust the setup.template.pattern and setup.dashboards.index settings as well. The setup.dashboards.index setting will make sure that your destination index has the correct mappings in Elasticsearch, and the setup.dashboards.index setting will make sure that Kibana's beats dashboards are using the correct underlying indices.

Can I ask, though, why you are planning on sending beats to different indexes? This may limit your ability to run certain aggregations and dashboards.

Finally, we have a section of our discussion forum that is specifically for discussing Packetbeat. You may be able to get better answers for your questions there, if they have to do with using Packetbeat specifically.

I hope this helps.

-William

[mouna_boujrad]

I have tow machines on each one I have a packetbeat installed ( the machines are from different networks ), I want to create two users, the first one can access to only indexe coming from its machine and the second one can access only to information from the second index that's why I want to separate the indexes mainely to create different views to each user, because I can't do it when having just one idexe that contain all the information, also about the setup.template.pattern and setup.dashboards.index I should define that in packetbeat.yml ? and I should create that pattern using the UI and elasticsearch index management is that correct sir ?

[William_Brafford]

I finally managed to find the time to run a little verification experiment. I'm sorry for the delay.

It turned out that, on version 7.4.0, I had to do a little bit more to get beat output to land in different indexes. I used filebeat for simplicity, but I think the configuration for output should be similar.

I made two configuration files, filebeat1.yml and filebeat2.yml. filebeat1.yml looked like this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/file/*1.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template:
  settings.index.number_of_shards: 1
  name: "index1"
  pattern: "index1-*"

setup.ilm:
  enabled: false

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "index1-%{[agent.version]}-%{+yyyy.MM.dd}"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

It turned out that because my backing Elasticsearch cluster had ILM (index lifecycle management) enabled, I had to set setup.ilm.enabled to false. For the second file, I changed output.elasticsearch.index, setup.template.name, and setup.template.pattern to reference index2 rather than index1, and pointed at a different file.

With all of this in place, I was able to run the following commands and have filebeat put the lines of my file in different indices:

./filebeat -c filebeat1.yml --path.data data1 run -e
./filebeat -c filebeat2.yml --path.data data2 run -e

I don't think you will need to worry about the --path.data flag; I only used it because I was running two filebeat processes on the same machine.

To get a custom index name working with ILM enabled, I used different settings.

setup.ilm:
  enabled: true
  rollover_alias: "index1-%{[agent.version]}"
  policy_name: "index1-%{[agent.version]}"

If you're on a recent version of the Elasticsearch stack, this ILM feature might be convenient.

I'm sorry for taking so long to get back to you, but I hope this information is helpful.

-William