New Elasticsearch installation and initial index pattern - what is it related to?

[przemolb (P)]

Hello,

I am new in Elasticsearch world so please forgive my ignorance. I have just finished new ELK installation and after logging in to Kibana it is asking me for "index pattern". If I have 10 hosts shipping their logs to ELK and I will use filebeat on 5 of them and some other solution (i.e. syslog) on the other 5 what should I put in to the "index pattern" ? Is the "index pattern" somehow related to what is sending the logs or where it is sent from ? If the former should I use "filebeat-*" as the index pattern ? If the latter -> "hostname-*" as an index pattern ?

Thanks in advance
P.

[warkolm (Mark Walkom)]

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out!

@przemolb

should I use "filebeat-*" as the index pattern ?

It sounds like it, yes.

You can head to Console, in Dev Tools, and then run GET _cat/indices and then paste the response in here for us to check if you want to be 100% sure.

[przemolb (P)]

But is the index pattern related to what is sending or where it is sent from ?
Also what about the other 5 hosts (taken from the example above) which don't use filebeat ? Is the index pattern "filebeat-*" filter them out ?

[warkolm (Mark Walkom)]

@przemolb

But is the index pattern related to what is sending or where it is sent from ?

Yes.

@przemolb

Also what about the other 5 hosts (taken from the example above) which don't use filebeat ? Is the index pattern "filebeat-*" filter them out ?

I don't know how you have that setup, so I can't comment.

[przemolb (P)]

@warkolm

@przemolb

But is the index pattern related to what is sending or where it is sent from ?

Yes.

Yes to what ? I asked two question above.

@warkolm

@przemolb

Also what about the other 5 hosts (taken from the example above) which don't use filebeat ? Is the index pattern "filebeat-*" filter them out ?

I don't know how you have that setup, so I can't comment.

I have 10 hosts sending their logs to Elasticsearch. Five of them are using filebeat for that purpose, the rest 5 don't. Let's say I use index pattern "filebeat-*". Is it kind of filter to accept logs ONLY from the first 5 hosts and refuse logs from those without filebeat ?

[warkolm (Mark Walkom)]

@przemolb

Is it kind of filter to accept logs ONLY from the first 5 hosts and refuse logs from those without filebeat ?

No, it's simply where filebeat defines as the source for the files.

For the others, you need to define your own index.

[przemolb (P)]

Can I have the same index for both kind of servers: those with filebeat installed and those without it ?

[warkolm (Mark Walkom)]

Yes, but it's not recommend. We suggest having an index for each different data type. So put apache logs together, but system logs in another index and so on.

[przemolb (P)]

Thanks @warkolm. So keep related data together.
The data (/var/log/*) from above example comes from servers but in different way: with filebeat or without so it makes sense to keep them together anyway.
But thanks for the hint.