Filebeat cannot create custom index - even when ilm is disabled

[gisellecarballo]

Hi Team,
I would like to ask for help as I cannot create custom index even when setup.ilm.enabled: is set to false.

Below is my filebeat config. Thanks.

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1

output.elasticsearch:
hosts: ["172.30.169.50:9200"]
ssl.verification_mode: "none"
username: "elastic"
password: "m4gN3s1@.123"
indices:

• index: "testing-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"
  when.equals:
  fields:
  event.module: "cisco"
• index: "testing-%{[agent.version]}-fortinet-%{+yyyy.MM.dd}"
  when.equals:
  fields:
  event.module: "fortinet"

[stephenb (Stephen Brown)]

Hi @gisellecarballo

When you use a filebeat modules. The index name is actually defined in the module. (Well technically the input settings). And input settings override output settings. That's why you're not seeing your output setting take effect.
So you have to overwrite it in the module.

Yes that should be documented better. It is in a docs but not very clear... See here

It looks like you're using the fortinet in Cisco modules.

So perhaps go into the module ymls

And set the index there example

- module: cisco
  asa:
    enabled: true
    var.paths: ["/var/log/cisco-asa.log"]
    var.input: "file"
    index: "testing-%{[agent.version]}-cisco-%{+yyyy.MM.dd}"

Then you don't even need to set the index setting in the elasticsearch output.

Think that'll work for you. Let us know

[gisellecarballo]

Hi @stephenb ,

Unfortunately, your suggestion does not work with my case. I added both the index on the cisco and fortinet module but no luck.
Is there any work around you know ?

[stephenb (Stephen Brown)]

@gisellecarballo

Unfortunately, your suggestion does not work with my case. I added both the index on the cisco and fortinet module but no luck.
Is there any work around you know ?

Huh ... I will take a look later today..

When you say it does not work ... what is the result?

What is the name of the index that is being written too?

What version of Filebeat and Elastic.

Can you share your cicso.yml

The more you provide the more we can help.

[gisellecarballo]

Hi @stephenb ,
I really appreciate you getting back to me.
Filebeat, kibana and elasticsearch versions are all 7.17.5
When I tried the below configurations, log does not even come thru on the default index, in short no syslogs seen at all.

###filebeat/modules.d/cisco.yml####
- module: cisco
  ios:
    enabled: true
    var.input: syslog
    index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5005
    var.syslog_protocol: udp

###filebeat/modules.d/fortinet.yml####
- module: fortinet
  firewall:
    enabled: true
    var.input: udp
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5009
    index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"


###filebeat/filebeat.yml####
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log

- type: syslog
  enabled: false
  paths:
    - /var/log/*.log

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 1

output.elasticsearch:
hosts: ["172.30.169.50:9200"]
ssl.verification_mode: "none"
username: "elastic"
password: "m4gN3s1@.123"

[stephenb (Stephen Brown)]

@gisellecarballo

When I tried the below configurations, log does not even come thru on the default index, in short no syslogs seen at all.

Well that is telling...

I would take out ALL the custom index stuff and first see if you are getting logs to the default filebeat-7.17.5 indices...

Then Report Back..'

I always tell people get the defaults working first .. then customize

[gisellecarballo]

Hi @stephenb ,

Able to get the default working.
Please see image below.

[stephenb (Stephen Brown)]

Ok I will look at it a bit later... it is something simple that we are missing.
You can take out the fields

setup.template.fields: "fields.yml"
unless you modified it.. (which I hope you did not)

Also when you run the above config... did you look at the filebeat logs... it should tell you why it can not index...

Also assuming this is cut-npaste

Should be also are you trying to do http or https? it is unclear to me?

output.elasticsearch:
  hosts: ["172.30.169.50:9200"]
  ssl.verification_mode: "none"
  username: "elastic"
  password: "m4gN3s1@.123"

[stephenb (Stephen Brown)]

@gisellecarballo

Ohh Darn ... I forgot the input. prefix

###filebeat/modules.d/cisco.yml####
- module: cisco
  ios:
    enabled: true
    var.input: syslog
    input.index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5005
    var.syslog_protocol: udp

[gisellecarballo]

I did not modify fields.yml
Below is the filebeat logs which I think would be relevant on our issue.
Elasticsearch is only http.

[stephenb (Stephen Brown)]

Apologies.. Really hard to read / work with screenshots... text is much better in the future.

Well you have other issues... your cluster / number of shards is full

I suggestion you clean up your cluster / address this first...

That is probably why it can not create the new indices...

So you need to get your cluster healthy / cleaned up first or increase

Cluster-level shard allocation and routing settings | Elasticsearch Guide...

If you increase the shard limit that is just a work around not a real fix... it will cause problems at some point.

Also I always ask to show the command you ran...

it looks like you ran the following for lots of details

filebeat -e -d "*"

you can just run

filebeat -e

[gisellecarballo]

I have cleaned up my cluster and yet custom indices were not working. I wonder what im missing here.

[stephenb (Stephen Brown)]

Well now you need to run filebeat without the -d "*" and look at the logs... the logs are very descriptive...

Did you fix the

input.index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"

[stephenb (Stephen Brown)]

@gisellecarballo Ok I found some issues and have it figured out...

Here are 2 fully functional approaches

I think this will work for you ..... using very close to your original...
This is a fully functional filebeat.yml... I just put the module directly in the filebeat.yml to test.
You were very close in the beginning then I gave you bad advice... see the 2nd example.

filebeat.modules:
  - module: cisco
    asa:
      enabled: true
      var.paths: ["/Users/sbrown/workspace/sample-data/cisco/asa.log"]
      var.input: "file"

setup.kibana:

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.overwrite: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "testing-%{[agent.version]}-%{[event.dataset]}-%{+yyyy.MM.dd}"

or define the index int in the module BUT the event.dataset is not available must be defined manually, this is where I gave you bad advice and it was not working.

filebeat.modules:
  - module: cisco
    asa:
      enabled: true
      var.paths: ["/Users/sbrown/workspace/sample-data/cisco/asa.log"]
      var.input: "file"
      input.index: "testing-%{[agent.version]}-cisco.asa-%{+yyyy.MM.dd}"

setup.kibana:

setup.ilm.enabled: false
setup.template.enabled: true
setup.template.name: "testing-%{[agent.version]}"
setup.template.pattern: "testing-*"
setup.template.overwrite: true


output.elasticsearch:
  hosts: ["localhost:9200"]

Both of these produce...

GET _cat/indices/?v
health status index                               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   testing-7.17.3-cisco.asa-2022.10.31 iz5_7S5YQziApDl2TML1gQ   1   1         88            0      104kb          104kb

[gisellecarballo]

Hi @stephenb ,

In my case, the second one works.
Putting index on each filebeat module.
Thank you for your effort on this.