Send Log data from multiple files with different indices without logstash

CLehmann · March 2, 2020, 8:57am

My goal is to use filebeat to take multiple log files, and send them to elastic search without logstash.

Here is my filebeat.yml:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      -  /usr/share/filebeat/logs/logs.json
    json.keys_under_root: true
    json.message_key: log
    encoding: utf-8
    fields:
      type: "logs1"

  - type: log
    enabled: true
    paths:
      -  /usr/share/filebeat/logs/logs_business.json
    json.keys_under_root: true
    json.message_key: log
    encoding: utf-8
    fields:
      type: "logs2"

setup.kibana:
    host: localhost:5601

output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "http"
  index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

setup.template:
  name: "index-%{[beat.version]}"
  pattern: "index-%{[beat.version]}-*"
  overwrite: true
  enabled: false


setup.ilm.enabled: false

Somehow, in elastic search, the only generated index is filebeat-7.6.0.
What am i missing?

jsoriano · March 2, 2020, 12:40pm

Hi @CLehmann and welcome to discuss

Since Beats 7.0 indexes are managed by ILM by default, you can find here these settings: https://www.elastic.co/guide/en/beats/filebeat/7.6/ilm.html

You may need to disable ILM to achieve what you intend.

CLehmann · March 2, 2020, 12:56pm

Hello @jsoriano, thank you for helping.

I have the following line in my filebeat.yml:

setup.ilm.enabled: false

This should disable ILM, shouldn't it? But the default index name is still used.

jsoriano · March 2, 2020, 2:52pm

Oh sorry, I didn't see the last line of the config

Could you try to use agent.version instead of beat.version? This field was renamed in 7.0.

CLehmann · March 2, 2020, 3:07pm

I changed all agent.version to beat.version but still no success. Do i have to do some configurations ins Kibana for it to work?

Edit: The indices somehow changed from filebeat-7.6.0 to filebeat-7.6.0-2020.03.02-000001 ...

jsoriano · March 2, 2020, 4:45pm

Do you have any error in your filebeat logs?

These are the indexes created by ILM. Is it possible that you have some filebeats configured with ILM enabled?

CLehmann · March 2, 2020, 5:33pm

It looks like, i have no errors (log below). But yes, i have a filebeat instance (different docker container) which has ILM enabled. I shut it down for test purposes and restartet everything but no change.

2020-03-02T16:59:02.640Z        INFO    instance/beat.go:622    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-03-02T16:59:02.666Z        INFO    instance/beat.go:630    Beat ID: ad1f8f26-d393-4574-990d-84902125f5ed
2020-03-02T16:59:02.666Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-03-02T16:59:02.666Z        INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "ad1f8f26-d393-4574-990d-84902125f5ed"}}}
2020-03-02T16:59:02.666Z        INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:45.000Z", "version": "7.6.0"}}}
2020-03-02T16:59:02.666Z        INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.7"}}}
2020-03-02T16:59:02.667Z        INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-20T08:30:19Z","containerized":true,"name":"b26d74cfe55b","ip":["127.0.0.1/8","172.24.0.3/16"],"kernel_version":"4.9.0-9-amd64","mac":["02:42:ac:18:00:03"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-03-02T16:59:02.668Z        INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2020-03-02T16:59:01.130Z"}}}
2020-03-02T16:59:02.668Z        INFO    instance/beat.go:298    Setup Beat: filebeat; Version: 7.6.0
2020-03-02T16:59:02.668Z        INFO    elasticsearch/client.go:174     Elasticsearch url: http://192.168.128.172:9200
2020-03-02T16:59:02.668Z        INFO    [publisher]     pipeline/module.go:110  Beat name: b26d74cfe55b
2020-03-02T16:59:02.669Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-03-02T16:59:02.669Z        INFO    instance/beat.go:439    filebeat start running.
2020-03-02T16:59:02.669Z        INFO    registrar/migrate.go:104        No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-03-02T16:59:02.669Z        INFO    registrar/migrate.go:112        Initialize registry meta file
2020-03-02T16:59:02.683Z        INFO    registrar/registrar.go:108      No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-03-02T16:59:02.698Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-03-02T16:59:02.698Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2020-03-02T16:59:02.698Z        INFO    crawler/crawler.go:72   Loading Inputs: 2
2020-03-02T16:59:02.698Z        INFO    log/input.go:152        Configured paths: [/usr/share/filebeat/logs/logs.json]
2020-03-02T16:59:02.706Z        INFO    input/input.go:114      Starting input of type: log; ID: 13050393368472405405
2020-03-02T16:59:02.706Z        INFO    log/input.go:152        Configured paths: [/usr/share/filebeat/logs/logs_business.json]
2020-03-02T16:59:02.706Z        INFO    input/input.go:114      Starting input of type: log; ID: 5255105000941106788
2020-03-02T16:59:02.706Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 2

jsoriano · March 2, 2020, 5:57pm

Your configuration works for me after replacing agent.version with beat.version.

From the logs of your filebeat I cannot see that it is reading any file Are these all the logs your filebeat is generating?

Do you see any log entry with messages starting with Harvester started for file...?

Once the file is being harvested there should be some messages about connecting to Elasticsearch.

CLehmann · March 2, 2020, 7:05pm

Ah, yes. I got it to work. Since there was no Harvester started for file ... i figured, there was a problem with the provided files. Well i forgot i changed the naming of the files and did not change that in my filebeat.yml.

Very stupid mistake. Sorry for wasting your time.

system · March 30, 2020, 7:06pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.