Unable to get Filebeat setup correctly

Hi everyone,
I'm trying to get Filebeat set up on a local Mac Mini so it will ingest log files of a process to Elasticsearch. The log files contain a custom JSON format. I got it sort of working but in Elasticsearch the timestamp of my log file was not being recognized as such so I wanted to create a custom index mapping where I explicitly declare the time field with it's format. However, I'm not able to get that working. I don't see any index being created in Elasticsearch with the setup that I have now.

First I created a custom index template in Elasticsearch as follows:

curl -X PUT "localhost:9200/_index_template/network-probe-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["network-probe-*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "context": {
          "type": "keyword"
        },
        "level": {
          "type": "keyword"
        },
        "logger": {
          "type": "keyword"
        },
        "mdc": {
          "properties": {
            "responseTime":  { "type": "integer" },
            "reachable": { "type": "boolean"},
            "server": { "type": "keyword" }
          }
        },
        "message": {
          "type": "text"
        },
        "thread": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss.SSS"
        }
      }
    }
  }
}
'

Next, I have a filebeat.yml file which is as follows:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/jonck/Documents/dev/network-probe/logs/pings.log
  json.keys_under_root: true
  json.add_error_key: true

setup.template.settings:
  index.number_of_shards: 1
setup.template.name: "network-probe-template"
setup.template.pattern: "network-probe-template-*"
setup.ilm.enabled: false # I set this to false according to what I read here: https://discuss.elastic.co/t/filebat-create-a-custom-index-on-elasticsearch/197741 

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "network-probe-%{[beat.version]}-%{+yyyy.MM.dd}"

I am on a Mac running Catalina 10.15.7, both Elasticsearch-oss and Filebeat-oss is installed using brew and running them using brew services. Both Elasticsearch and Filebeat are versions 7.9.

I am not getting any errors in my filebeat log file, but nevertheless I am not seeing an index being created in Elasticsearch and none of my logs are being shipped to Elasticsearch.

Any help pointing me in the right direction would be much appreciated!

Hi,
Before someone from elastic team comes to help you, let's check few things.
1- I think [beat.version] is changed to [agent.version] so I suggest you to change that.
2- Check your connection and config with filebeat test output -c CONFIG_PATH and filebeat test config -c CONFIG_PATH
3- Run ./filebeat setup -e to make sure changes are applied.

Hi Borna,
Thanks for your reply!

I have changed the configuration to have [agent.version] instead of [beat.version]. Unfortunately I still don't see any data being pushed to Elastic.

I have also run the commands you mentioned, here is the output:

> filebeat test output -c /usr/local/etc/filebeat/filebeat.yml
elasticsearch: http://localhost:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.9.2

The TLS disabled was expected since I'm running the OSS version.

> filebeat test config -c /usr/local/etc/filebeat/filebeat.yml 
Config OK

Thanks for your help, much appreciated!

I think the problem is your setup.template.pattern.
The index your filebeat is creating is something like : network-probe-7.9.2-* but your index template is expecting network-probe-template-*. so change it to network-probe-* and see if it works.

Ok, I made that change, the good news is there is progress!

I now get this error message:

2020-10-11T15:28:15.666+0200	INFO	template/load.go:89	Template network-probe-template already exists and will not be overwritten.

So I guess I'm not understanding how Filebeat uses templates. I was under the assumption that if I create a template Filebeat would use that template to create a new index. But from the error message it seems like Filebeat is trying to create a template itself. I don't understand that conceptually, what would Filebeat base the template on? I have not given it a definition anywhere.

Anyway, apparently I haven't configured it correctly, would you happen to know how I can configure Filebeat so it will use the template that I created earlier using my curl statement to Elasticsearch?

Thanks again :slight_smile:

Hi
2020-10-11T15:28:15.666+0200 INFO template/load.go:89 Template network-probe-template already exists and will not be overwritten.
It's because your filebeat is trying to import a template with mapping defined in fields.yml file (etc/filebeat/fields.yml). but you've already defined a template with that name in your elasticsearch.
So you could delete your template in elasticsearch and change the mapping inside the fields.yml according to your network-probe-template.

Read this document. it should answer most of your question. I'm new to elasticsearch so I'm afraid I might give you a wrong answer.

I'll take a look at overriding the fields.yml file.

Thanks again Borna for taking the time to help me out!

Your welcome,

Sorry I forgot to link the document. :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.