# Send Log data from multiple files with different indices without logstash

**URL:** <https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 2, 2020, 8:57am UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648 "2020-03-02T08:57:35Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![CLehmann](https://avatars.discourse-cdn.com/v4/letter/c/e36b37/32.png) [@CLehmann](https://discuss.elastic.co/u/CLehmann)\
**Post date:** [March 2, 2020, 8:57am UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/1 "2020-03-02T08:57:36Z")

</div>

My goal is to use filebeat to take multiple log files, and send them to elastic search without logstash.

Here is my **filebeat.yml** :

```
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /usr/share/filebeat/logs/logs.json
    json.keys_under_root: true
    json.message_key: log
    encoding: utf-8
    fields:
      type: "logs1"

  - type: log
    enabled: true
    paths:
      - /usr/share/filebeat/logs/logs_business.json
    json.keys_under_root: true
    json.message_key: log
    encoding: utf-8
    fields:
      type: "logs2"

setup.kibana:
    host: localhost:5601

output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "http"
  index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

setup.template:
  name: "index-%{[beat.version]}"
  pattern: "index-%{[beat.version]}-*"
  overwrite: true
  enabled: false

setup.ilm.enabled: false

```

Somehow, in elastic search, the only generated index is **filebeat-7.6.0**.  
What am i missing?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 2, 2020, 12:40pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/2 "2020-03-02T12:40:36Z")

</div>

Hi @CLehmann and welcome to discuss 🙂

Since Beats 7.0 indexes are managed by ILM by default, you can find here these settings: [https://www.elastic.co/guide/en/beats/filebeat/7.6/ilm.html](https://www.elastic.co/guide/en/beats/filebeat/7.6/ilm.html)

You may need to disable ILM to achieve what you intend.

---

<div class="post-metadata">

**Author:** ![CLehmann](https://avatars.discourse-cdn.com/v4/letter/c/e36b37/32.png) [@CLehmann](https://discuss.elastic.co/u/CLehmann)\
**Post date:** [March 2, 2020, 12:56pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/3 "2020-03-02T12:56:12Z")

</div>

Hello @jsoriano, thank you for helping.

I have the following line in my filebeat.yml:

`setup.ilm.enabled: false`

This should disable ILM, shouldn't it? But the default index name is still used. ☹

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 2, 2020, 2:52pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/4 "2020-03-02T14:52:32Z")

</div>

Oh sorry, I didn't see the last line of the config 🤦‍♂️

Could you try to use `agent.version` instead of `beat.version`? This field was renamed in 7.0.

---

<div class="post-metadata">

**Author:** ![CLehmann](https://avatars.discourse-cdn.com/v4/letter/c/e36b37/32.png) [@CLehmann](https://discuss.elastic.co/u/CLehmann)\
**Post date:** [March 2, 2020, 3:07pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/5 "2020-03-02T15:07:26Z")

</div>

I changed all `agent.version` to `beat.version` but still no success. Do i have to do some configurations ins Kibana for it to work?

Edit: The indices somehow changed from ` filebeat-7.6.0` to `filebeat-7.6.0-2020.03.02-000001` ...

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 2, 2020, 4:45pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/6 "2020-03-02T16:45:10Z")

</div>

Do you have any error in your filebeat logs?

> [@CLehmann](#):
>
> Edit: The indices somehow changed from ` filebeat-7.6.0` to `filebeat-7.6.0-2020.03.02-000001` ...

These are the indexes created by ILM. Is it possible that you have some filebeats configured with ILM enabled?

---

<div class="post-metadata">

**Author:** ![CLehmann](https://avatars.discourse-cdn.com/v4/letter/c/e36b37/32.png) [@CLehmann](https://discuss.elastic.co/u/CLehmann)\
**Post date:** [March 2, 2020, 5:33pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/7 "2020-03-02T17:33:56Z")

</div>

It looks like, i have no errors (log below). But yes, i have a filebeat instance (different docker container) which has ILM enabled. I shut it down for test purposes and restartet everything but no change.

```
2020-03-02T16:59:02.640Z INFO instance/beat.go:622 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-03-02T16:59:02.666Z INFO instance/beat.go:630 Beat ID: ad1f8f26-d393-4574-990d-84902125f5ed
2020-03-02T16:59:02.666Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-03-02T16:59:02.666Z INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "ad1f8f26-d393-4574-990d-84902125f5ed"}}}
2020-03-02T16:59:02.666Z INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:45.000Z", "version": "7.6.0"}}}
2020-03-02T16:59:02.666Z INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.7"}}}
2020-03-02T16:59:02.667Z INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-20T08:30:19Z","containerized":true,"name":"b26d74cfe55b","ip":["127.0.0.1/8","172.24.0.3/16"],"kernel_version":"4.9.0-9-amd64","mac":["02:42:ac:18:00:03"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-03-02T16:59:02.668Z INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2020-03-02T16:59:01.130Z"}}}
2020-03-02T16:59:02.668Z INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.0
2020-03-02T16:59:02.668Z INFO elasticsearch/client.go:174 Elasticsearch url: http://192.168.128.172:9200
2020-03-02T16:59:02.668Z INFO [publisher] pipeline/module.go:110 Beat name: b26d74cfe55b
2020-03-02T16:59:02.669Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-03-02T16:59:02.669Z INFO instance/beat.go:439 filebeat start running.
2020-03-02T16:59:02.669Z INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-03-02T16:59:02.669Z INFO registrar/migrate.go:112 Initialize registry meta file
2020-03-02T16:59:02.683Z INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-03-02T16:59:02.698Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-03-02T16:59:02.698Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-03-02T16:59:02.698Z INFO crawler/crawler.go:72 Loading Inputs: 2
2020-03-02T16:59:02.698Z INFO log/input.go:152 Configured paths: [/usr/share/filebeat/logs/logs.json]
2020-03-02T16:59:02.706Z INFO input/input.go:114 Starting input of type: log; ID: 13050393368472405405
2020-03-02T16:59:02.706Z INFO log/input.go:152 Configured paths: [/usr/share/filebeat/logs/logs_business.json]
2020-03-02T16:59:02.706Z INFO input/input.go:114 Starting input of type: log; ID: 5255105000941106788
2020-03-02T16:59:02.706Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2
```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 2, 2020, 5:57pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/8 "2020-03-02T17:57:30Z")

</div>

Your configuration works for me after replacing `agent.version` with `beat.version`.

From the logs of your filebeat I cannot see that it is reading any file 🤔 Are these all the logs your filebeat is generating?

Do you see any log entry with messages starting with `Harvester started for file...`?

Once the file is being harvested there should be some messages about connecting to Elasticsearch.

---

<div class="post-metadata">

**Author:** ![CLehmann](https://avatars.discourse-cdn.com/v4/letter/c/e36b37/32.png) [@CLehmann](https://discuss.elastic.co/u/CLehmann)\
**Post date:** [March 2, 2020, 7:05pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/9 "2020-03-02T19:05:53Z")

</div>

Ah, yes. I got it to work. Since there was no `Harvester started for file ...` i figured, there was a problem with the provided files. Well i forgot i changed the naming of the files and did not change that in my filebeat.yml.

Very stupid mistake. Sorry for wasting your time. 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2020, 7:06pm UTC](https://discuss.elastic.co/t/send-log-data-from-multiple-files-with-different-indices-without-logstash/221648/10 "2020-03-30T19:06:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.