Custom name with Filebeat

[griffer98 (john)]

I am trying to send data from filebeat straight to elasticsearch. But no matter what I do I can't get the name to be what I want. I have tried everything the docs and all the forums are saying to do but nothing works. I am trying to make a datastream called for example, myindex, but it keeps creating a data stream called filebeat-8.11. I already have a template and index policy created in kibana. Here is my filebeat config for this part. Do I even need to have a template section if I have the template specified in kibana already. With logstash, all I had to do was specify an index name and that would get pick up by a template in kibana as long as it matched the pattern.

output.elasticsearch:
  hosts: [ "localhost" ]
  index: 'myindex'

setup:
  ilm.enabled: false
  template.name: "myindex"
  template.pattern: "myindex"
  template.enabled: false
  template.overwrite: true

[stephenb (Stephen Brown)]

Hi @griffer98

It is different from your example... as I just ran mine and it worked exactly, you have extra lines.

Assuming you are starting from the base filebeat.yml did you see the docs here for the next practice.

One Caution if you set the dashboard setting only have it enabled when you run `filebeat setuip1 then comment it out... otherwise it will try to load the dashboards everytime you start

My Entire Fielbeeat this is my entire filebeat not a snippet

filebeat.inputs:

- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
setup.template.name: "myindex-%{[agent.version]}"
setup.template.pattern: "myindex-%{[agent.version]}"

setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "myindex-%{[agent.version]}"
  preset: balanced

Then run setup it create template and data stream
.\filebeat setup -e

Then Run
`.\filebeat -e``

Then running ships the data to the correct backing index

GET myindex-8.12.1/_search
# Result

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-myindex-8.12.1-2024.02.27-000001",
        "_id": "gZSj7I0BBi-RUJEfXTUa",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-02-27T22:15:35.481Z",
          "input": {
            "type": "filestream"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "mac": [
              "16-29-36-97-5A-61",
              "2A-79-B1-6A-5B-54",
              "5C-52-30-9C-EF-E0",
              "5E-52-30-C9-1E-64",
              "7E-52-30-9C-EF-E0",
              "82-B2-58-49-30-00",
              "82-B2-58-49-30-01",
              "82-B2-58-49-30-04",
              "82-B2-58-49-30-05",
              "AC-DE-48-00-11-22"
            ],
            "hostname": "hyperion.local",
            "architecture": "x86_64",
            "os": {
              "family": "darwin",
              "name": "macOS",
              "kernel": "23.3.0",
              "build": "23D60",
              "type": "macos",
              "platform": "darwin",
              "version": "14.3.1"
            },
            "id": "9E46F076-B7F1-53AA-921B-C2F983746B79",
            "ip": [
              "fe80::aede:48ff:fe00:1122",
              "fe80::1044:6645:65ff:4a82",
              "10.0.5.189",
              "fe80::2879:b1ff:fe6a:5b54",
              "fe80::2879:b1ff:fe6a:5b54",
              "fe80::464f:d4f:a4e2:74b8",
              "fe80::5717:2cd1:1158:7ad",
              "fe80::3ad7:2f31:4b2:436a",
              "fe80::ce81:b1c:bd2c:69e",
              "192.168.64.1",
              "fe80::5c52:30ff:fec9:1e64",
              "fd4f:f065:6c04:18a9:1ca9:5641:25f7:9947"
            ],
            "name": "hyperion.local"
          },
          "agent": {
            "name": "hyperion.local",
            "type": "filebeat",
            "version": "8.12.1",
            "ephemeral_id": "afc195d0-8ba1-4fb7-9553-a8d326878818",
            "id": "8494fae1-ecc8-4b6b-be8a-6f8f770bc5c5"
          },
          "message": """Wed Feb 21 22:27:26 hyperion jamf[38716]: Checking for policies triggered by "recurring check-in" for user "sbrown"...""",
          "log": {
            "offset": 2294128,
            "file": {
              "path": "/var/log/jamf.log",
              "device_id": "16777221",
              "inode": "87944943"
            }
          }
        }
      },
....

[griffer98 (john)]

Yes, i tried this too and nothing happened. that is no different from what I have in my example. Do I have to have the agent version in there. Is that unavoidable? Even so, I have also tried it with that and still does not work.

[stephenb (Stephen Brown)]

@griffer98 See Above try it exactly ... the code works I just ran it.

You have extra lines like
template.enabled: false

The Agent version is best practice to aid in upgraded and future work.

A data view can be created that is

myindex-*

[stephenb (Stephen Brown)]

I did it without the agent version it worked exactly...
Although I would recommend to use the agent version but that is up to you...

GET myindex/_search

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-myindex-2024.02.27-000001",
        "_id": "n5Sv7I0BBi-RUJEfXNEt",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-02-27T22:28:51.337Z",
          "log": {
            "offset": 18630,
            "file": {
              "path": "/var/log/fsck_apfs.log",
              "device_id": "16777221",
              "inode": "87938985"
            }
          },
          "message": "/dev/rdisk1s1: fsck_apfs completed at Tue Jan 23 08:33:30 2024",
          "input": {
            "type": "filestream"
          },
          "agent": {
            "id": "e7ff7114-2f94-462b-8d00-e4231f69bfda",
            "name": "hyperion.local",
            "type": "filebeat",
            "version": "8.12.1",
            "ephemeral_id": "70c029a7-12fe-4d2f-8372-0aaccd039ce1"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "mac": [
              "16-29-36-97-5A-61",
              "2A-79-B1-6A-5B-54",
              "5C-52-30-9C-EF-E0",
              "5E-52-30-C9-1E-64",
              "7E-52-30-9C-EF-E0",
              "82-B2-58-49-30-00",
              "82-B2-58-49-30-01",
              "82-B2-58-49-30-04",
              "82-B2-58-49-30-05",
              "AC-DE-48-00-11-22"
            ],
            "hostname": "hyperion.local",
            "architecture": "x86_64",
            "os": {
              "version": "14.3.1",
              "family": "darwin",
              "name": "macOS",
              "kernel": "23.3.0",
              "build": "23D60",
              "type": "macos",
              "platform": "darwin"
            },
            "id": "9E46F076-B7F1-53AA-921B-C2F983746B79",
            "name": "hyperion.local",
            "ip": [
              "fe80::aede:48ff:fe00:1122",
              "fe80::1044:6645:65ff:4a82",
              "10.0.5.189",
              "fe80::2879:b1ff:fe6a:5b54",
              "fe80::2879:b1ff:fe6a:5b54",
              "fe80::464f:d4f:a4e2:74b8",
              "fe80::5717:2cd1:1158:7ad",
              "fe80::3ad7:2f31:4b2:436a",
              "fe80::ce81:b1c:bd2c:69e",
              "192.168.64.1",
              "fe80::5c52:30ff:fec9:1e64",
              "fd4f:f065:6c04:18a9:1ca9:5641:25f7:9947"
            ]
          }
        }
      },
      {