Filebeat output to existing index

caraboy · September 15, 2021, 7:49pm

Hi everyone,
I'm trying to configure a filebeat output to an existing index but apparently i can't correctly set the template name and pattern.

output.elasticsearch.index: "sai5x"
setup.template.name: "sai5x"
setup.template.pattern: "sai5x"

DevOps created the index for me, and they seem not have used any template, just reindexed documents form an older ES, cause if i GET /sai5x no template is mentioned, and i know they used dynamic mapping.
I've tried creating de index_template i reference in setup.template.name via:

PUT /_index_template/sai5x
{
some_config...
}

But files still get indexed in filebeat-7.12.0.
What am I doing wrong?? Is there a default template used by elastic in these cases?
Thank you in advance.

warkolm · September 15, 2021, 10:01pm

It'd be useful if you shared your full Filebeat config.

Why, what happens when you do this?

caraboy · September 16, 2021, 11:50am

Nothing happens. I expressed my self wrong, it acknowledge: true, but i can't get that template applied to my existing index, cause files keep on going to the default index filebeat created.

caraboy · September 16, 2021, 12:20pm

Logs show index it's not being set correctly, but don't know why:

2021-09-16T12:05:38.516Z    INFO    instance/beat.go:660    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-09-16T12:05:38.525Z    INFO    instance/beat.go:668    Beat ID: e3b0f4f4-f4a8-4065-9bd4-1c8e0613698e
2021-09-16T12:05:38.526Z    INFO    [seccomp]   seccomp/seccomp.go:124  Syscall filter successfully installed
2021-09-16T12:05:38.526Z    INFO    [beat]  instance/beat.go:996    Beat info   {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "e3b0f4f4-f4a8-4065-9bd4-1c8e0613698e"}}}
2021-09-16T12:05:38.526Z    INFO    [beat]  instance/beat.go:1005   Build info  {"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:16:51.000Z", "version": "7.12.0"}}}
2021-09-16T12:05:38.526Z    INFO    [beat]  instance/beat.go:1008   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.15.8"}}}
2021-09-16T12:05:38.526Z    INFO    [beat]  instance/beat.go:1012   Host info   {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-16T11:13:58Z","containerized":true,"name":"0ee7a6d770a0","ip":["127.0.0.1/8","172.20.0.2/16"],"kernel_version":"5.4.0-84-generic","mac":["02:42:ac:14:00:02"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"44ccc339d95bd51386fcfc5d8f041927"}}}
2021-09-16T12:05:38.526Z    INFO    [beat]  instance/beat.go:1041   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 8, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-09-16T12:05:36.760Z"}}}
2021-09-16T12:05:38.527Z    INFO    instance/beat.go:304    Setup Beat: filebeat; Version: 7.12.0
2021-09-16T12:05:38.536Z    INFO    [index-management]  idxmgmt/std.go:184  Set output.elasticsearch.index to 'filebeat-7.12.0' as ILM is enabled.
2021-09-16T12:05:38.536Z    INFO    eslegclient/connection.go:99    elasticsearch url: http://xxx.xxx.xxx.xxx:9200
2021-09-16T12:05:38.537Z    INFO    [publisher] pipeline/module.go:113  Beat name: filebeat:prueba
2021-09-16T12:05:38.537Z    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-09-16T12:05:38.537Z    INFO    instance/beat.go:468    filebeat start running.
2021-09-16T12:05:38.538Z    INFO    memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-09-16T12:05:38.538Z    INFO    memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=7
2021-09-16T12:05:38.538Z    INFO    [registrar] registrar/registrar.go:109  States Loaded from registrar: 1
2021-09-16T12:05:38.538Z    INFO    [crawler]   beater/crawler.go:71    Loading Inputs: 1
2021-09-16T12:05:38.538Z    INFO    log/input.go:157    Configured paths: [/var/log/logger.json]
2021-09-16T12:05:38.538Z    INFO    [crawler]   beater/crawler.go:141   Starting input (ID: 5347375231622932818)
2021-09-16T12:05:38.538Z    INFO    [crawler]   beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 1
2021-09-16T12:06:08.540Z    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"0ee7a6d770a00ad9f44b6c9cc313453506a8ce140680f43183bb8c2f1c8d745a"},"cpuacct":{"id":"0ee7a6d770a00ad9f44b6c9cc313453506a8ce140680f43183bb8c2f1c8d745a","total":{"ns":257949678}},"memory":{"id":"0ee7a6d770a00ad9f44b6c9cc313453506a8ce140680f43183bb8c2f1c8d745a","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":43708416}}}},"cpu":{"system":{"ticks":60,"time":{"ms":63}},"total":{"ticks":180,"time":{"ms":192},"value":180},"user":{"ticks":120,"time":{"ms":129}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"e45ec394-5d14-423f-a572-55befcc58b4c","uptime":{"ms":30312}},"memstats":{"gc_next":18462496,"memory_alloc":9723624,"memory_sys":74793984,"memory_total":45335960,"rss":71954432},"runtime":{"goroutines":22}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4},"load":{"1":1.38,"15":1.84,"5":1.85,"norm":{"1":0.345,"15":0.46,"5":0.4625}}}}}}

warkolm · September 19, 2021, 10:10pm

I think it'd be useful if you could show your entire Filebeat config please.

caraboy · September 20, 2021, 1:23pm

Thanks for the answer!
Here you got. I've tried changing the order of the settings, placing index config upper and lower. Also changed from dot notation, to indent (YAML) notation, nothing works. I think the issue is when setting the template cause, like i said, it was created without specifying one.

name: "filebeat:prueba2"
tags: ["filebeat", "prueba"]


filebeat.inputs:
- type: log
  enabled: true
  #tail_files:true
  paths:
    - /var/log/logger.json
  json:  #storing logs as json
    add_error_key: true 
    keys_under_root: true

output.elasticsearch:
  hosts: ["XXXXXXXXXXX:9200"]
  username: "XXXXX"
  password: "XXXXXXXX"
  index: "sai5x-test"
setup.template:
  name: "sai5xa"
  pattern: "sai5xa*"
  type: "index"

logging.level: debug
logging.to_files: true
logging.files:
  path: /home/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

caraboy · September 23, 2021, 12:47pm

Solved! Had to set setup.ilm.enabled: false
Thank's for the help.

system · October 21, 2021, 2:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.