Functionbeat Disable ilm Policy

Elastic Stack Beats
1

Hi Team, I am trying to enable custom index in functionbeat using the below configuration.

output.elasticsearch:
     hosts: ["10.10.10.10:9200"]
     output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
     setup.template.name: "customname"
     setup.template.pattern: "customname-*"
     setup.ilm.enabled: false

I understand that for custom indices, we need to disable ILM. which i have done in the above and ive also disabled ILM in my elasticsearch as below.

POST _ilm/stop

This is still not ignoring my ILM settings and when i run my yml, below is the log im receiving.

[root@abce2a2elkiup0100 functionbeat-7.9.0-linux-x86_64]# ./functionbeat test config -e
2020-09-16T12:32:24.184Z        INFO    instance/beat.go:640    Home path: [/etc/functionbeat-7.9.0-linux-x86_64] Config path: [/etc/functionbeat-7.9.0-linux-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2020-09-16T12:32:24.185Z        INFO    instance/beat.go:648    Beat ID: 49b89005-e805-4389-ac87-3d538e773fe2
2020-09-16T12:32:24.186Z        INFO    [beat]  instance/beat.go:976    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/functionbeat-7.9.0-linux-x86_64", "data": "/tmp", "home": "/etc/functionbeat-7.9.0-linux-x86_64", "logs": "/tmp/logs"}, "type": "functionbeat", "uuid": "49b89005-e805-4389-ac87-3d538e773fe2"}}}
2020-09-16T12:32:24.186Z        INFO    [beat]  instance/beat.go:985    Build info      {"system_info": {"build": {"commit": "b2ee705fc4a59c023136c046803b56bc82a16c8d", "libbeat": "7.9.0", "time": "2020-08-11T19:44:55.000Z", "version": "7.9.0"}}}
2020-09-16T12:32:24.186Z        INFO    [beat]  instance/beat.go:988    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.14.4"}}}
2020-09-16T12:32:24.187Z        INFO    [beat]  instance/beat.go:992    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-09-03T07:54:06Z","containerized":false,"name":"abce2a2elkiup0100.abc.com","ip":["127.0.0.1/8","::1/128","10.10.10.10/24","fe80::412:38ff:fed8:7a48/64"],"kernel_version":"4.18.0-193.14.3.el8_2.x86_64","mac":["06:12:38:d8:7a:48"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux","version":"8.2 (Ootpa)","major":8,"minor":2,"patch":0,"codename":"Ootpa"},"timezone":"UTC","timezone_offset_sec":0,"id":"5b06dee4fd2a4f9b9bc0e325677a1a36"}}}
2020-09-16T12:32:24.187Z        INFO    [beat]  instance/beat.go:1021   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/functionbeat-7.9.0-linux-x86_64", "exe": "/etc/functionbeat-7.9.0-linux-x86_64/functionbeat", "name": "functionbeat", "pid": 111834, "ppid": 52141, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-09-16T12:32:23.170Z"}}}
2020-09-16T12:32:24.187Z        INFO    instance/beat.go:299    Setup Beat: functionbeat; Version: 7.9.0
***2020-09-16T12:32:24.187Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'functionbeat-7.9.0' as ILM is enabled.***
2020-09-16T12:32:24.187Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://10.10.10.10:9200
2020-09-16T12:32:24.187Z        INFO    [publisher]     pipeline/module.go:113  Beat name: abce2a2elkiup0100.abc.com
Config OK

How do i disable ilm ?..Please help me resolve this.

Thanks.

2

Did you try configuration options mentioned on this page: https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html ?

3

@mtojek ,Yes, I did, but unfortunately, it isnt reflecting.
I have set the ilm policy details as below which isnt considered by functionbeat.

setup.ilm.enabled: auto
setup.ilm.overwrite: true
setup.ilm.rollover_alias: "portfolio"
setup.ilm.pattern: "{now/d}-000001"

I also tried setting ovewrite to false, which is unuseful too.
How should i proceed ?

4

Hi,
I'm not sure but I think you shouldn't put it under output.elasticsearch settings.

Here is my configuration that's currently working (filebeat 7.8):

setup.ilm.enabled: false
output.elasticsearch:
  hosts: ["node-1:9200"]
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM}"
  indices:
    - index: "cisco-beat-%{+yyyy.MM}"
      when.contains:
        event.module: "cisco"
  protocol: "https"
5

According to this page: https://www.elastic.co/guide/en/beats/functionbeat/current/ilm.html , all options are available.

6

Hi @borna_talebi ,
Thank you very much.
Below configuration under elasticsearch output in functionbeat.yml worked for me.

setup.ilm.enabled: false
setup.template.name: "wmdevlogs"
setup.template.pattern: "wmdevlogs-*"
output.elasticsearch:
  # Array of hosts to connect to.
 hosts: ["0.0.0.0:9200"]
 index: "%{[wmdevlogs]}-%{[agent.version]}-%{+yyyy.MM.dd}"
 indices:
   - index: "portfolio-%{[agent.version]}-%{+yyyy.MM.dd}"
     when.contains:
        log_group:  "Portfolio"
1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.