Index in filebeat.yml doesn't match index in elasticsearch dataview

Elastic Stack Elasticsearch
1

I added filebeat to a Linux Mint system in an attempt to upload logs from it to my elk stack. The index is set to this in filebeat.yml:

index: "vicki-desktop-filestream"

When I started filebeat, I guess it created the index on the elasticsearch server as "filebeat-8.12.2" because that is the index that I can add to the data view to see the log traffic from that system. I tried to add "vicki-desktop-filesteam", but that isn't in the dropdown and isn't accepted if I type it in. I even created an index called that on the elasticsearch server using curl, but even though it allows me to create it, it doesn't ingest any data. I did create a data view by selecting the index "filebeat-8.12.2' though, and I see the data. I would like to be able to use the hostname as the index name since I have multiple systems that I want to pull in logs from and they will likely all use the same filebeat version. I have read through the index documentation, but I must be missing something.

Here is the filebeat.yml file without the comments:filebeat.inputs:

- type: log
  id: vicki-desktop-filestream-id
  enabled: true
  paths:
    - "/var/log/syslog"
    - "/var/log/messages"
    - "/var/log/*.log"
  multiline:
    pattern: '^[A-Z]\S{2,8} {1,2}\d{1,2} \d{2}:\d{2}:\d{2}\b'
    match: after
    negate: true
    processors:
    - add_locale: ~
output.elasticsearch:
    hosts: ["http://ubuntu-elk.mydomain.com:9200"]
    index: "vicki-desktop-filestream"
    pipeline: "vicki-desktop-filesteam-pipeline"
setup:
    template.enabled: false
    ilm.enabled: false
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "ubuntu-elk.mydomain.com:5601"
output.elasticsearch:
  hosts: ["ubuntu-elk.mydomain.com:9200"]
  preset: balanced
  username: "elastic"
  password: "changeme"
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Can anyone clarify what I am missing here?

Vickistan

2

Not sure if this is the issue, but you have 2 elasticsearch outputs configured, this is not supported, you can have only one output.

If I'm not wrong if you have two, it will just ignore everything except the last one.

3

That was absolutely what was happening. Thanks. Now I need to understand why this doesn't work:

    output.elasticsearch:
    hosts: ["ubuntu-elk.mydomain.com:9200"]
    preset: balanced
    username: "elastic"
    password: "changeme"
    index: "vicki-desktop-filestream"
    pipeline: "vicki-desktop-filesteam-pipeline"

when I comment out the second output and restart. I copied the uncommented lines from the one that was working up to this one. I expected that restarting filebeat on this system would create the index vicki-desktop-filestream, but it didn't. I know how to use a curl to create an index on the elasticsearch server, but I am not sure what the json body should look like or whether this is something that I should do:

curl -X PUT "localhost:9200/vicki-desktop-filestream?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "host.name.keyword": { "type": "text" }
    }
  }
}
'

I really just want to rename the index to the hostname instead of filebeat-8.12..2

Can you point me to an example or documentation or tell me what to do to rename the index?

Vickistan

4

Perhaps look at the section of the document for this very use case.

output.elasticsearch.index: "vicki-desktop-filestream-%{[agent.version]}"
setup.template.name: "vicki-desktop-filestream-%{[agent.version]}"
setup.template.pattern: "vicki-desktop-filestream-%{[agent.version]}"

We recommend the agent version but if not try exactly what you want

output.elasticsearch.index: "vicki-desktop-filestream"
setup.template.name: "vicki-desktop-filestream"
setup.template.pattern: "vicki-desktop-filestream"

That will create a data stream named vicki-desktop-filestream with a backing index...

GET _cat/indices/v*?v
GET _data_stream/vicki-desktop-filestream

# GET _cat/indices/v*?v 200 OK
health status index                                          uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open   .ds-vicki-desktop-filestream-2024.04.17-000001 qswF7nxdS7OnUiwd7Bj4OQ   1   1      10617            0      1.2mb          1.2mb        1.2mb

# GET _data_stream/vicki-desktop-filestream 200 OK
{
  "data_streams": [
    {
      "name": "vicki-desktop-filestream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-vicki-desktop-filestream-2024.04.17-000001",
          "index_uuid": "qswF7nxdS7OnUiwd7Bj4OQ",
          "prefer_ilm": true,
          "ilm_policy": "filebeat",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "status": "YELLOW",
      "template": "vicki-desktop-filestream",
      "ilm_policy": "filebeat",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}