Creating different indexes for different modules from filebeat

Elastic Stack Beats
1

Hello, I'm very new to elk stack so please bear with me. I'm learning Elastic Stack from scratch and I have paid for and taken a few classes, but none of the classes I have gone through seem to go very in depth for the input configurations with beats. I have successfully configured cisco ios filebeats to ship to Elasticsearch, by following the built in instruction in kibana, however they always have the same index name. I would like to segregate the different cisco devices to have their own wildcard indexes. So instead of building a index of filebeat*, I would have 1 index be filebeat-cisco-ios* and another index be filebeat-cisco-asa*. If that is not possible, I would like be able to at least define different indexes by module.

I've done some research, and I'm not a coder by any means so a lot of this has been a crash course with me. From what I've read, I need to configure filebeat.inputs for this, which used to be called filebeat.prospectors. How would i do this. Does anyone have any examples by any chance.

Thanks in advance. If you need anything let me know and I will be sure to post it.

2

Hi @Lenny Welcome to the community and thanks for trying Elasticsearch

Before I attempt to answer the question I would like to ask a question.

What are you trying to accomplish, what is your actual goal for separating the indices?

To be more specific, what are the functional difference from your perspective of 1 index which is filtered by a field event.dataset vs 2 separate indices..

What if you never saw the actual indices would it make a difference? (elastic is slowly headed that way with data streams)

There are some reasons to do this, just curious what problem you are trying to solve....

the tl;dr is that you can ... absolutely can... but you are going to need properly configure a number of items, all aligned...

Usually I tell folks to get used to elastic (example templates, index alias, rollover and ILM etc) for a bit before starting to make fundamental changes.

Apologies ahead of time that it will not be as easy as you would like... absolutely doable ... but not just a 1 line change...

3

Look at data streams Data streams | Elasticsearch Guide [master] | Elastic

However, you need to avoid a lot of small indices. Break indices by retention (and access depending on license level) and manage size and retention with ILM. Data Streams looks like it gives you a single view of data while allowing some data to be deleted sooner than other.

4

An Actually Look at Elastic Agent Cisco Integration which is Beta Today but Soon to Go GA.
Elastic Agent Integrations use Data Streams under the covers..

5

Thanks for the reply!
So, our overall end goal is logging, health monitoring, and alerting. We essentially want logs for all servers, iis and equivelants, all sql, and all cisco/networking infrastructure.
My goal for seperating indexes was I was assuming, and this is due to my ignorance with elastic stack, that it would be easier for me to manage the front end, if i created finer granularity on the back end. Seeing the indices would not make a difference.
So at this point, and correct me if i'm wrong, maybe it would be better to just start pulling in all cisco equipment, which is what i'm working on first, with the defaults provided by filebeat?

6

Thanks! I didn't know about this best practice. and thanks for the link too.

7

I would start with the defaults... learn... iterate.

I would also perhaps looks at Elastic Agent... it is the future of the collection / ingest for Elasticsearch.

8

I will do that, thank you so much for you input.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.