# Unable to get Filebeat setup correctly

**URL:** <https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 10, 2020, 6:13pm UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651 "2020-10-10T18:13:05Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonckvanderkogel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonckvanderkogel/32/76690_2.png) [@jonckvanderkogel](https://discuss.elastic.co/u/jonckvanderkogel)\
**Post date:** [October 10, 2020, 6:13pm UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/1 "2020-10-10T18:13:06Z")

</div>

Hi everyone,  
I'm trying to get Filebeat set up on a local Mac Mini so it will ingest log files of a process to Elasticsearch. The log files contain a custom JSON format. I got it sort of working but in Elasticsearch the timestamp of my log file was not being recognized as such so I wanted to create a custom index mapping where I explicitly declare the time field with it's format. However, I'm not able to get that working. I don't see any index being created in Elasticsearch with the setup that I have now.

First I created a custom index template in Elasticsearch as follows:

```auto
curl -X PUT "localhost:9200/_index_template/network-probe-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["network-probe-*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "context": {
          "type": "keyword"
        },
        "level": {
          "type": "keyword"
        },
        "logger": {
          "type": "keyword"
        },
        "mdc": {
          "properties": {
            "responseTime": { "type": "integer" },
            "reachable": { "type": "boolean"},
            "server": { "type": "keyword" }
          }
        },
        "message": {
          "type": "text"
        },
        "thread": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss.SSS"
        }
      }
    }
  }
}
'

```

Next, I have a filebeat.yml file which is as follows:

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/jonck/Documents/dev/network-probe/logs/pings.log
  json.keys_under_root: true
  json.add_error_key: true

setup.template.settings:
  index.number_of_shards: 1
setup.template.name: "network-probe-template"
setup.template.pattern: "network-probe-template-*"
setup.ilm.enabled: false # I set this to false according to what I read here: https://discuss.elastic.co/t/filebat-create-a-custom-index-on-elasticsearch/197741 

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "network-probe-%{[beat.version]}-%{+yyyy.MM.dd}"

```

I am on a Mac running Catalina 10.15.7, both Elasticsearch-oss and Filebeat-oss is installed using brew and running them using brew services. Both Elasticsearch and Filebeat are versions 7.9.

I am not getting any errors in my filebeat log file, but nevertheless I am not seeing an index being created in Elasticsearch and none of my logs are being shipped to Elasticsearch.

Any help pointing me in the right direction would be much appreciated!

---

<div class="post-metadata">

**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\
**Post date:** [October 11, 2020, 5:15am UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/2 "2020-10-11T05:15:14Z")

</div>

Hi,  
Before someone from elastic team comes to help you, let's check few things.  
1- I think `[beat.version]` is changed to `[agent.version]` so I suggest you to change that.  
2- Check your connection and config with `filebeat test output -c CONFIG_PATH` and `filebeat test config -c CONFIG_PATH`  
3- Run `./filebeat setup -e` to make sure changes are applied.

---

<div class="post-metadata">

**Author:** ![jonckvanderkogel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonckvanderkogel/32/76690_2.png) [@jonckvanderkogel](https://discuss.elastic.co/u/jonckvanderkogel)\
**Post date:** [October 11, 2020, 11:50am UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/3 "2020-10-11T11:50:32Z")

</div>

Hi Borna,  
Thanks for your reply!

I have changed the configuration to have [agent.version] instead of [beat.version]. Unfortunately I still don't see any data being pushed to Elastic.

I have also run the commands you mentioned, here is the output:

```auto
> filebeat test output -c /usr/local/etc/filebeat/filebeat.yml
elasticsearch: http://localhost:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.9.2

```

The TLS disabled was expected since I'm running the OSS version.

```auto
> filebeat test config -c /usr/local/etc/filebeat/filebeat.yml 
Config OK

```

Thanks for your help, much appreciated!

---

<div class="post-metadata">

**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\
**Post date:** [October 11, 2020, 12:22pm UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/4 "2020-10-11T12:22:18Z")

</div>

I think the problem is your `setup.template.pattern`.  
The index your filebeat is creating is something like : `network-probe-7.9.2-*` but your index template is expecting `network-probe-template-*`. so change it to `network-probe-*` and see if it works.

---

<div class="post-metadata">

**Author:** ![jonckvanderkogel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonckvanderkogel/32/76690_2.png) [@jonckvanderkogel](https://discuss.elastic.co/u/jonckvanderkogel)\
**Post date:** [October 11, 2020, 1:35pm UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/5 "2020-10-11T13:35:08Z")

</div>

Ok, I made that change, the good news is there is progress!

I now get this error message:

```auto
2020-10-11T15:28:15.666+0200	INFO	template/load.go:89	Template network-probe-template already exists and will not be overwritten.

```

So I guess I'm not understanding how Filebeat uses templates. I was under the assumption that if I create a template Filebeat would use that template to create a new index. But from the error message it seems like Filebeat is trying to create a template itself. I don't understand that conceptually, what would Filebeat base the template on? I have not given it a definition anywhere.

Anyway, apparently I haven't configured it correctly, would you happen to know how I can configure Filebeat so it will use the template that I created earlier using my curl statement to Elasticsearch?

Thanks again 🙂

---

<div class="post-metadata">

**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\
**Post date:** [October 12, 2020, 4:59am UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/6 "2020-10-12T04:59:32Z")

</div>

Hi  
`2020-10-11T15:28:15.666+0200	INFO	template/load.go:89	Template network-probe-template already exists and will not be overwritten.`  
It's because your filebeat is trying to import a template with mapping defined in `fields.yml` file (etc/filebeat/fields.yml). but you've already defined a template with that name in your elasticsearch.  
So you could delete your template in elasticsearch and change the mapping inside the fields.yml according to your `network-probe-template`.

> [@jonckvanderkogel](#):
>
> So I guess I'm not understanding how Filebeat uses templates.

Read this document. it should answer most of your question. I'm new to elasticsearch so I'm afraid I might give you a wrong answer.

---

<div class="post-metadata">

**Author:** ![jonckvanderkogel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonckvanderkogel/32/76690_2.png) [@jonckvanderkogel](https://discuss.elastic.co/u/jonckvanderkogel)\
**Post date:** [October 12, 2020, 9:04am UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/7 "2020-10-12T09:04:43Z")

</div>

I'll take a look at overriding the fields.yml file.

Thanks again Borna for taking the time to help me out!

---

<div class="post-metadata">

**Author:** ![borna\_talebi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/borna_talebi/32/76317_2.png) [@borna\_talebi](https://discuss.elastic.co/u/borna_talebi)\
**Post date:** [October 12, 2020, 9:09am UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/8 "2020-10-12T09:09:15Z")

</div>

Your welcome,

> [@borna\_talebi](#):
>
> Read this document.

Sorry I forgot to link the [document](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html). 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2020, 11:09am UTC](https://discuss.elastic.co/t/unable-to-get-filebeat-setup-correctly/251651/9 "2020-11-09T11:09:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.