Unable to get Filebeat setup correctly

Elastic Stack Beats
1

Hi everyone,
I'm trying to get Filebeat set up on a local Mac Mini so it will ingest log files of a process to Elasticsearch. The log files contain a custom JSON format. I got it sort of working but in Elasticsearch the timestamp of my log file was not being recognized as such so I wanted to create a custom index mapping where I explicitly declare the time field with it's format. However, I'm not able to get that working. I don't see any index being created in Elasticsearch with the setup that I have now.

First I created a custom index template in Elasticsearch as follows:

curl -X PUT "localhost:9200/_index_template/network-probe-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["network-probe-*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "context": {
          "type": "keyword"
        },
        "level": {
          "type": "keyword"
        },
        "logger": {
          "type": "keyword"
        },
        "mdc": {
          "properties": {
            "responseTime":  { "type": "integer" },
            "reachable": { "type": "boolean"},
            "server": { "type": "keyword" }
          }
        },
        "message": {
          "type": "text"
        },
        "thread": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss.SSS"
        }
      }
    }
  }
}
'

Next, I have a filebeat.yml file which is as follows:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/jonck/Documents/dev/network-probe/logs/pings.log
  json.keys_under_root: true
  json.add_error_key: true

setup.template.settings:
  index.number_of_shards: 1
setup.template.name: "network-probe-template"
setup.template.pattern: "network-probe-template-*"
setup.ilm.enabled: false # I set this to false according to what I read here: https://discuss.elastic.co/t/filebat-create-a-custom-index-on-elasticsearch/197741 

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "network-probe-%{[beat.version]}-%{+yyyy.MM.dd}"

I am on a Mac running Catalina 10.15.7, both Elasticsearch-oss and Filebeat-oss is installed using brew and running them using brew services. Both Elasticsearch and Filebeat are versions 7.9.

I am not getting any errors in my filebeat log file, but nevertheless I am not seeing an index being created in Elasticsearch and none of my logs are being shipped to Elasticsearch.

Any help pointing me in the right direction would be much appreciated!

2

Hi,
Before someone from elastic team comes to help you, let's check few things.
1- I think [beat.version] is changed to [agent.version] so I suggest you to change that.
2- Check your connection and config with filebeat test output -c CONFIG_PATH and filebeat test config -c CONFIG_PATH
3- Run ./filebeat setup -e to make sure changes are applied.

3

Hi Borna,
Thanks for your reply!

I have changed the configuration to have [agent.version] instead of [beat.version]. Unfortunately I still don't see any data being pushed to Elastic.

I have also run the commands you mentioned, here is the output:

> filebeat test output -c /usr/local/etc/filebeat/filebeat.yml
elasticsearch: http://localhost:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.9.2

The TLS disabled was expected since I'm running the OSS version.

> filebeat test config -c /usr/local/etc/filebeat/filebeat.yml 
Config OK

Thanks for your help, much appreciated!

4

I think the problem is your setup.template.pattern.
The index your filebeat is creating is something like : network-probe-7.9.2-* but your index template is expecting network-probe-template-*. so change it to network-probe-* and see if it works.

5

Ok, I made that change, the good news is there is progress!

I now get this error message:

2020-10-11T15:28:15.666+0200	INFO	template/load.go:89	Template network-probe-template already exists and will not be overwritten.

So I guess I'm not understanding how Filebeat uses templates. I was under the assumption that if I create a template Filebeat would use that template to create a new index. But from the error message it seems like Filebeat is trying to create a template itself. I don't understand that conceptually, what would Filebeat base the template on? I have not given it a definition anywhere.

Anyway, apparently I haven't configured it correctly, would you happen to know how I can configure Filebeat so it will use the template that I created earlier using my curl statement to Elasticsearch?

Thanks again :slight_smile:

6

Hi
2020-10-11T15:28:15.666+0200 INFO template/load.go:89 Template network-probe-template already exists and will not be overwritten.
It's because your filebeat is trying to import a template with mapping defined in fields.yml file (etc/filebeat/fields.yml). but you've already defined a template with that name in your elasticsearch.
So you could delete your template in elasticsearch and change the mapping inside the fields.yml according to your network-probe-template.

Read this document. it should answer most of your question. I'm new to elasticsearch so I'm afraid I might give you a wrong answer.

7

I'll take a look at overriding the fields.yml file.

Thanks again Borna for taking the time to help me out!

8

Your welcome,

Sorry I forgot to link the document. :slight_smile: