Filebeat cannot create custom index - even when ilm is disabled

gisellecarballo · October 28, 2022, 1:40pm

Hi Team,
I would like to ask for help as I cannot create custom index even when setup.ilm.enabled: is set to false.

Below is my filebeat config. Thanks.

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1

output.elasticsearch:
hosts: ["172.30.169.50:9200"]
ssl.verification_mode: "none"
username: "elastic"
password: "m4gN3s1@.123"
indices:

index: "testing-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"
when.equals:
fields:
event.module: "cisco"
index: "testing-%{[agent.version]}-fortinet-%{+yyyy.MM.dd}"
when.equals:
fields:
event.module: "fortinet"

stephenb · October 29, 2022, 12:05am

Hi @gisellecarballo

When you use a filebeat modules. The index name is actually defined in the module. (Well technically the input settings). And input settings override output settings. That's why you're not seeing your output setting take effect.
So you have to overwrite it in the module.

Yes that should be documented better. It is in a docs but not very clear... See here

It looks like you're using the fortinet in Cisco modules.

So perhaps go into the module ymls

And set the index there example

- module: cisco
  asa:
    enabled: true
    var.paths: ["/var/log/cisco-asa.log"]
    var.input: "file"
    index: "testing-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"

Then you don't even need to set the index setting in the elasticsearch output.

Think that'll work for you. Let us know

gisellecarballo · October 31, 2022, 2:20pm

Hi @stephenb ,

Unfortunately, your suggestion does not work with my case. I added both the index on the cisco and fortinet module but no luck.
Is there any work around you know ?

stephenb · October 31, 2022, 3:03pm

Huh ... I will take a look later today..

When you say it does not work ... what is the result?

What is the name of the index that is being written too?

What version of Filebeat and Elastic.

Can you share your cicso.yml

The more you provide the more we can help.

gisellecarballo · October 31, 2022, 3:36pm

Hi @stephenb ,
I really appreciate you getting back to me.
Filebeat, kibana and elasticsearch versions are all 7.17.5
When I tried the below configurations, log does not even come thru on the default index, in short no syslogs seen at all.


###filebeat/modules.d/cisco.yml####
- module: cisco
  ios:
    enabled: true
    var.input: syslog
    index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5005
    var.syslog_protocol: udp

###filebeat/modules.d/fortinet.yml####
- module: fortinet
  firewall:
    enabled: true
    var.input: udp
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5009
    index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"


###filebeat/filebeat.yml####
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log

- type: syslog
  enabled: false
  paths:
    - /var/log/*.log

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1

output.elasticsearch:
hosts: ["172.30.169.50:9200"]
ssl.verification_mode: "none"
username: "elastic"
password: "m4gN3s1@.123"

stephenb · October 31, 2022, 3:54pm

Well that is telling...

I would take out ALL the custom index stuff and first see if you are getting logs to the default filebeat-7.17.5 indices...

Then Report Back..'

I always tell people get the defaults working first .. then customize

gisellecarballo · October 31, 2022, 4:10pm

Hi @stephenb ,

Able to get the default working.
Please see image below.

stephenb · October 31, 2022, 4:12pm

Ok I will look at it a bit later... it is something simple that we are missing.
You can take out the fields

setup.template.fields: "fields.yml"
unless you modified it.. (which I hope you did not)

Also when you run the above config... did you look at the filebeat logs... it should tell you why it can not index...

Also assuming this is cut-npaste

Should be also are you trying to do http or https? it is unclear to me?

output.elasticsearch:
  hosts: ["172.30.169.50:9200"]
  ssl.verification_mode: "none"
  username: "elastic"
  password: "m4gN3s1@.123"

stephenb · October 31, 2022, 4:29pm

@gisellecarballo

Ohh Darn ... I forgot the input. prefix

###filebeat/modules.d/cisco.yml####
- module: cisco
  ios:
    enabled: true
    var.input: syslog
    input.index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5005
    var.syslog_protocol: udp

gisellecarballo · October 31, 2022, 4:43pm

I did not modify fields.yml
Below is the filebeat logs which I think would be relevant on our issue.
Elasticsearch is only http.

stephenb · October 31, 2022, 5:02pm

Apologies.. Really hard to read / work with screenshots... text is much better in the future.

Well you have other issues... your cluster / number of shards is full

I suggestion you clean up your cluster / address this first...

That is probably why it can not create the new indices...

So you need to get your cluster healthy / cleaned up first or increase

If you increase the shard limit that is just a work around not a real fix... it will cause problems at some point.

Also I always ask to show the command you ran...

it looks like you ran the following for lots of details

filebeat -e -d "*"

you can just run

filebeat -e

gisellecarballo · October 31, 2022, 7:08pm

I have cleaned up my cluster and yet custom indices were not working. I wonder what im missing here.

stephenb · October 31, 2022, 8:05pm

Well now you need to run filebeat without the -d "*" and look at the logs... the logs are very descriptive...

Did you fix the

input.index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"

stephenb · October 31, 2022, 10:04pm

@gisellecarballo Ok I found some issues and have it figured out...

Here are 2 fully functional approaches

I think this will work for you ..... using very close to your original...
This is a fully functional filebeat.yml... I just put the module directly in the filebeat.yml to test.
You were very close in the beginning then I gave you bad advice... see the 2nd example.

filebeat.modules:
  - module: cisco
    asa:
      enabled: true
      var.paths: ["/Users/sbrown/workspace/sample-data/cisco/asa.log"]
      var.input: "file"

setup.kibana:

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.overwrite: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"

or define the index int in the module BUT the event.dataset is not available must be defined manually, this is where I gave you bad advice and it was not working.

filebeat.modules:
  - module: cisco
    asa:
      enabled: true
      var.paths: ["/Users/sbrown/workspace/sample-data/cisco/asa.log"]
      var.input: "file"
      input.index: "testing-%{[agent.version]}-cisco.asa-%{+yyyy.MM.dd}"

setup.kibana:

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.overwrite: true


output.elasticsearch:
  hosts: ["localhost:9200"]

Both of these produce...

GET _cat/indices/?v
health status index                               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   testing-7.17.3-cisco.asa-2022.10.31 iz5_7S5YQziApDl2TML1gQ   1   1         88            0      104kb          104kb

gisellecarballo · November 2, 2022, 12:46pm

Hi @stephenb ,

In my case, the second one works.
Putting index on each filebeat module.
Thank you for your effort on this.

system · November 30, 2022, 2:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat is unable to create custom index Beats filebeat	12	1116	February 8, 2023
Filebat - Create a custom index on elasticsearch Beats filebeat	14	4697	October 23, 2019
Issue when create custom index with filebeat in elasticsearch Beats filebeat	2	326	December 23, 2021
Filebeat 7.5.1 Missing a step for custom index names config? Beats filebeat	3	905	February 12, 2020
Exporting to constant index name with Filebeat Beats filebeat	4	363	March 19, 2021

Filebeat cannot create custom index - even when ilm is disabled

Related Topics