Trying to change filebeat index

[Russell_Fulton]

I am trying to change the index filebeat writes to from the default "filebeat..." to "suricata..."

I have the following in my filebeat.yml:

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false
setup:template:
    enabled: true
    name: "suricata"
    pattern: "suricata-*"

running by hand (as nothing gets written to the log : ( )

rful011@secmonprd08:~$ sudo /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat
Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified

I can't see what is wrong.

At the moment I am doing this on just one machine but ultimately I want this deployed on several. Am I correct in assuming that I need to do this just once?

[leandrojmp (Leandro Pereira)]

@Russell_Fulton

setup:template:
    enabled: true
    name: "suricata"
    pattern: "suricata-*"

I think this is a typo, should be setup.template not setup:template.

[Russell_Fulton]

doh! thanks!

filebeat now starts but does not create the "suricata" index. I also tried setting setup.ilm.check_exists: true

logs:

2022-10-28T09:34:34.561+1300    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.1
2022-10-28T09:34:34.561+1300    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.1' as ILM is enabled.
2022-10-28T09:34:34.561+1300    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: https://secesprd02.its.auckland.ac.nz:9200
2022-10-28T09:34:34.561+1300    INFO    [publisher]     pipeline/module.go:113  Beat name: secmonprd08.its.auckland.ac.nz
2022-10-28T09:34:34.563+1300    INFO    beater/filebeat.go:118  Enabled modules/filesets:  (), suricata (eve)
2022-10-28T09:34:34.565+1300    INFO    instance/beat.go:492    filebeat start running.
2022-10-28T09:34:34.565+1300    INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2022-10-28T09:34:34.565+1300    INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=34044
2022-10-28T09:34:34.983+1300    INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=57700
2022-10-28T09:34:34.984+1300    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 1
2022-10-28T09:34:34.984+1300    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 1
2022-10-28T09:34:34.984+1300    WARN    [cfgwarn]       registered_domain/registered_domain.go:61       BETA: The registered_domain processor is beta.
2022-10-28T09:34:34.984+1300    WARN    [cfgwarn]       log/input.go:89 DEPRECATED: Log input. Use Filestream input instead.
2022-10-28T09:34:34.985+1300    INFO    [input] log/input.go:171        Configured paths: [/data/sensors/eve.json]      {"input_id": "51dbcb5e-d306-43e3-9141-4238e8d9e083"}
2022-10-28T09:34:34.985+1300    INFO    [crawler]       beater/crawler.go:141   Starting input (ID: 3131651449671214263)
2022-10-28T09:34:34.985+1300    INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 1
2022-10-28T09:34:34.986+1300    INFO    [input.harvester]       log/harvester.go:309    Harvester started for file.     {"input_id": "51dbcb5e-d306-43e3-9141-4238e8d9e083", "source": "/data/sensors/eve.json", "state_id": "native::131465221-2050", "finished": false, "os_id": "131465221-2050", "old_source": "/data/sensors/eve.json", "old_finished": true, "old_os_id": "131465221-2050", "harvester_id": "74e6c9a1-c305-4f90-8095-0702bd654655"}
2022-10-28T09:34:35.988+1300    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://secesprd02.its.auckland.ac.nz:9200))
2022-10-28T09:34:35.988+1300    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2022-10-28T09:34:35.988+1300    INFO    [publisher]     pipeline/retry.go:223     done
2022-10-28T09:34:36.055+1300    INFO    [esclientleg]   eslegclient/connection.go:284   Attempting to connect to Elasticsearch version 7.17.1
2022-10-28T09:34:36.084+1300    INFO    [esclientleg]   eslegclient/connection.go:284   Attempting to connect to Elasticsearch version 7.17.1
2022-10-28T09:34:36.140+1300    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2022-10-28T09:34:36.150+1300    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy filebeat exists already.
2022-10-28T09:34:36.150+1300    INFO    [index-management]      idxmgmt/std.go:397      Set setup.template.name to '{filebeat-7.17.1 {now/d}-000001}' as ILM is enabled.
2022-10-28T09:34:36.150+1300    INFO    [index-management]      idxmgmt/std.go:402      Set setup.template.pattern to 'filebeat-7.17.1-*' as ILM is enabled.
2022-10-28T09:34:36.150+1300    INFO    [index-management]      idxmgmt/std.go:436      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.17.1 {now/d}-000001} as ILM is enabled.
2022-10-28T09:34:36.150+1300    INFO    [index-management]      idxmgmt/std.go:440      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2022-10-28T09:34:36.167+1300    INFO    template/load.go:110    Template "filebeat-7.17.1" already exists and will not be overwritten.
2022-10-28T09:34:36.167+1300    INFO    [index-management]      idxmgmt/std.go:297      Loaded index template.
2022-10-28T09:34:36.171+1300    INFO    [index-management.ilm]  ilm/std.go:126  Index Alias filebeat-7.17.1 exists already.
2022-10-28T09:34:36.172+1300    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://secesprd02.its.auckland.ac.nz:9200)) established
2022-10-28T09:35:04.574+1300    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"user.slice"},"cpuacct":{"id":"user.slice","total":{"ns":407526084257}},"memory":{"id":"user.slice","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":495489024}}}},"cpu":{"system":{"ticks":100,"time":{"ms":107}},"total":{"ticks":860,"time":{"ms":871},"value":860},"user":{"ticks":760,"time":{"ms":764}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":13},"info":{"ephemeral_id":"bcbf09d7-8277-4b80-9aba-2266834f2af4","uptime":{"ms":30093},"version":"7.17.1"},"memstats":{"

output setup:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts:  ["secesprd02.its.auckland.ac.nz"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  index: "suricata-%{[agent.version]}-%{+yyyy.MM.dd}"

[stephenb (Stephen Brown)]

@Russell_Fulton

setup:template

setup.template

Huh did not see the updates...

[stephenb (Stephen Brown)]

Apologies, not sure what's going on. I'm not getting the full thread until I reply... 7.17.

Try adding

setup.ilm.enabled: false

I think I went through a couple different methods of this and why the certain settings work here

Filebeat writing to its own index

As explained in the post I linked, You have a moduled enabled in your configuration whether it is important now or not... It overrides the output settings in some cases but let's put that aside there are other issues... Also from the docs here... which is your key issue... When index lifecycle management (ILM) is enabled, the default index is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}" , for example, "filebeat-7.12.1-2022-07-28-000001" . Custom index settings are ignored when…

so I think you need

setup.ilm.enabled: false
setup.template:
    enabled: true
    name: "suricata"
    pattern: "suricata-*"