Custom index not getting created in Kibana

[sunil_s]

Unable to create new index in ES.

Please find below Filebeat configuration

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - xxx.log

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata:
      host: ${NODE_NAME}

setup.ilm.overwrite: true
setup.ilm.enabled: false
setup.template.name: 'test2'
setup.template.pattern: 'test2-*'
setup.template.overwrite: true
setup.template.enabled: false
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["http://xxxx:9200"]
  index: "test2-%{[agent.version]}-%{+yyyy.MM.dd}"  # Set your desired index name and pattern here

[stephenb (Stephen Brown)]

What version 7.x or 8.x?

[sunil_s]

version 8.5.0

[stephenb (Stephen Brown)]

Follow these instructions here (there is a bug in the 8.5. docs)

Ok try this.... this is a complete working sample

This will setup the template and ILM policy with your custom name, where you can control the daily rollover.

Do not try to create a data stream with the -%{+yyyy.MM.dd} syntax ... just go to the new ILM policy and set the rollover daily and the backing index will roll over daily... example in this case

.ds-customname-8.5.3-2023.11.26-000001

The commented-out last line you can uncomment that when you run setup but you MUST comment it back out or it will reload the dashboards every time which takes a very long time and it will look like filebeat is not working ... also on only use that if you plan to reuse a modules dashboards

filebeat.inputs:
- type: filestream
  id: syslogs
  enabled: true
  paths:
    - /var/log/system.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "customname-%{[agent.version]}"

setup.template.name: "customname-%{[agent.version]}"
setup.template.pattern: "customname-%{[agent.version]}"
setup.ilm.enabled: true
setup.ilm.policy_name: "customname"
# setup.dashboards.index: "customname-*"

[sunil_s]

Thank you @stephenb for looking into the issue. I was experimenting few stuff and was able to generate index with below yaml.

Pasting it below if anyone finds it useful

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - <log-path>.log
  processors:
     - decode_json_fields:
         fields: ["message"]
         process_array: false
         target: "" 


setup.ilm.overwrite: true
setup.ilm.enabled: false
setup.template.name: 'test6'
setup.template.pattern: 'test6-*'
setup.template.overwrite: true
setup.template.enabled: false
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["http://<host>:9200"]
  index: "test6-%{[agent.version]}-%{+yyyy.MM.dd}"  

Just one query.
The log file had 10k entries, but decode json option, i believe is adding more index's and i have 40k hits. is this something that we can control.

[stephenb (Stephen Brown)]

Did you try my solution... It is more flexible than yours but whatever works for you... Adding the date on the index name may not work real great in the long run.

@sunil_s

Just one query.
The log file had 10k entries, but decode json option, i believe is adding more index's and i have 40k hits. is this something that we can control.

You will need to provide more details are you sure your did not load the data more than once?

What does the source look like?

Do you see duplicates?

What do you mean add more indices?