Creating dynamic index name with Filebeat based on custom event field fails

Elastic Stack Beats
1

I'm trying to have filebeat create a dynamic index name based on a custom event field and it is not working. Can custom event fields be used in the index name? If I use a non custom event field everything works fine (e.g. host.name). This is the config statement in filebeat.ym under output.elasticsearch:
index: "sonic_syslog-ticket-%{[Custom][ticket_num]}"
If I use
index: "sonic_syslog-ticket-%{[host][name]}"
everything works fine.
I started filebeat in debug mode and was looking at the logs. Not positive if this is the error message related to my problem but I see the following:

{"log.level":"debug","@timestamp":"2023-12-07T12:02:52.352-0500","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":452},"message":"Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"Index 0 out of bounds for length 0"}","service.name":"filebeat","ecs.version":"1.6.0"}

2

I got this working after getting a better understanding of the concept of fields in filebeat and elasticsearch. In my case, the custom field I was referring to is created in the elasticsearch ingest pipeline. Those fields are not yet created/available to filebeat. What I did was use a primitive dissect processor within filebeat.yml to process the log message and extract the field I was interested in. That field is now available to be referenced in the filebeat.yml file. I then used that field name within the index name.... Note, I also still use the elasticsearch ingest pipeline as well to do the real detailed processing.. Here is a snippet of my filebeat.yml file:

Blockquote

filebeat.inputs:
- type: filestream
  id: syslog_filebeat-id
  enabled: true
  prospector.scanner.check_interval: 1s
  paths:
    - /srv/decompress_dir/*/log/syslog
    - /srv/decompress_dir/*/log/syslog.[0-9]
    - /srv/decompress_dir/*/log/syslog.[1-9][0-9]
    - /srv/decompress_dir/*/log/syslog.[1-9][0-9][0-9]

  processors:
    - dissect:
        tokenizer: "%{key1};;;%{key2},%{key3},%{key4},%{ticket_num}"
        field: "message"
        target_prefix: "dissect"

output.elasticsearch:
  hosts: ["1.1.1.1:9200"]
  index: "syslog-case-num-%{[dissect][ticket_num]}"
  pipeline: my_syslog_pipeline

# ================ Elasticsearch template setting ==================
setup.template.settings:
  index.number_of_shards: 1
setup.template.name: "my_syslog"
setup.template.pattern: "syslog-case-*"
setup.ilm.enabled: false

Blockquote

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.