Create index for filebeat with idicated host or log path

Elastic Stack Kibana
1

Hi,
I'm new in ELK and my main goal is to share logs for created users from different hosts.
So at this moment I have two hosts A and B where I installed and configured filebeat.
I need to create index for filebeat, but divided for each host.
I create role and spaces for different logs but can't use log from specific host
Trying many options but I can't find solution.
I would be grateful for the advice

2

Excessive splitting of logs to multiple indices can lead to problems later, too many shards causes more problems than a few large shards.

That said, I don't see enough here to understand the problem. Are your host logs getting to the indices? You can use Kibana monitoring to see index activity.

3

@rugenl Yes, but I need to share one source of logs to one user. I don't want give him access to all logs. I have one indices filebeat-* for all logs from two hosts and that is issue. Could I create index for one source of logs ? I could solve my problem

4

I'm not sure if I understand your question correctly. you have two sources of logs and you want to give access to one source of log to a specific user. for example logs from pc1 for userA and logs from pc2 to userB, am I correct?
If that's the case just change the default index name to include host.name in it and then create two users and give them specific access to each index.

5

Yes u understood very well. Ok, but when I create new index for example filebeat* and I type something more e.x filebeat*:host.name: pc1 . How create index with specific host ?

6

According to docs:

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, fields.log_type, to set the index:
output.elasticsearch:
hosts: ["http://localhost:9200"]
index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"

you also need to configure the setup.template.name and setup.template.pattern options (see Elasticsearch index template ).

And don't forget: Custom index settings are ignored when ILM is enabled.

1 Like
7

Here is my filebeat config to push elastic server logs to a different index, changed lines only.

setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-elastic-internal"
....output section
index: "filebeat-%{[agent.version]}-elastic-internal-%{+yyyy.MM.dd}"

This works with ILM.

8

So turnoff ILM or not ? I added what @rugenl wrote but nothing change.

output.elasticsearch
hosts: ["xxxxx"]
username: "xxx"
password: "xxx"
index: "filebeat-%{[host.name]}-elastic-internal-%{+yyyy.MM.dd}"
setup.ilm.rollover_alias: "filebeat-%{[host.name]}-elastic-internal"

9

ILM default actions vary with elasticsearch 7 and prior versions.

It may be the location of these lines, the setup.ilm..... doesn't go in the output section, find the current line for each.

It will work, but there are many different starting points and paths to get there.

10

Thank you for tips @rugenl and @borna_talebi . I made my custom index :smiley:

1 Like
11

Glad I could help.
Can you post your final config here? for future reference.

12

======================= Elasticsearch template setting =======================

setup.template.settings:
index.number_of_shards: 3
setup.template.name: "ferrari"
setup.template.pattern: "ferrari"
setup.ilm.enabled: false

---------------------------- Elasticsearch Output ----------------------------

output.elasticsearch:
hosts: ["xxxxxxxxxx:xxxx"]
username: "xxxx"
password: "xxxxxxxxxx"
index: "ferrari-%{+yyyy.MM.dd}"

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.