Filebeat Setup isn't working after update to 7.9.0

Hello there,

we just upgraded our testsite (elastic, kibana, metricbeat, auditbeat and filebeat) to version 7.9.0. After the update filebeat setup fails with the following message:

[root@ELASTIC99004 ~]# filebeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
Exiting: 3 errors: Error setting up ML for apache_ecs: cannot set up ML with prefix: filebeat-apache_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}; Error setting up ML for apache_ecs: cannot set up ML with prefix: filebeat-apache_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}; Error setting up ML for nginx_ecs: cannot set up ML with prefix: filebeat-nginx_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}

Does anyone has an idea what to do? Auditbeat setup and metricbeat setup are working.

Thanks and kind regards
Boris Kunstleben

I was experiencing a similar error with the system module after upgrading all my servers filebeat from v7.8.1 to 7.9.0. I did some troubleshooting and noticed that filebeat wasn't opening a connection to elastic (v7.8.1) anymore. Upgrading the elastic deployment to the latest version has resolved the issue for me. I hope this helps

Hello David,
thanks for your answer, the error is still there, even after i upgraded all elastic nodes to 7.9.

1 Like

Can you run that again with debug enabled?

Hello warkolm,

is run it again with debug enabled, but the part concerning the problem hasn't changed:

filebeat -e setup
    2020-08-21T06:51:52.801+0200    INFO    instance/beat.go:640    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
        2020-08-21T06:51:52.801+0200    INFO    instance/beat.go:648    Beat ID: 9df4464c-4284-49fe-aed3-643815633979
        2020-08-21T06:51:52.803+0200    INFO    [beat]  instance/beat.go:976    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "9df4464c-4284-49fe-aed3-643815633979"}}}
        2020-08-21T06:51:52.803+0200    INFO    [beat]  instance/beat.go:985    Build info      {"system_info": {"build": {"commit": "b2ee705fc4a59c023136c046803b56bc82a16c8d", "libbeat": "7.9.0", "time": "2020-08-11T20:11:11.000Z", "version": "7.9.0"}}}
        2020-08-21T06:51:52.803+0200    INFO    [beat]  instance/beat.go:988    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.4"}}}
        2020-08-21T06:51:52.803+0200    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:89     add_cloud_metadata: hosting provider type not detected.
        2020-08-21T06:51:52.803+0200    INFO    [beat]  instance/beat.go:992    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-08-20T13:48:00+02:00","containerized":false,"name":"ELASTIC99004","ip":["127.0.0.1/8","::1/128","10.21.20.214/22","fe80::250:56ff:fea9:910f/64"],"kernel_version":"4.18.0-147.8.1.el8_1.x86_64","mac":["00:50:56:a9:91:0f"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"8 (Core)","major":8,"minor":1,"patch":1911,"codename":"Core"},"timezone":"CEST","timezone_offset_sec":7200,"id":"ffe6394fc21948ceac14b104c876a79e"}}}
        2020-08-21T06:51:52.804+0200    INFO    [beat]  instance/beat.go:1021   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 16670, "ppid": 2444, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-08-21T06:51:51.910+0200"}}}
        2020-08-21T06:51:52.804+0200    INFO    instance/beat.go:299    Setup Beat: filebeat; Version: 7.9.0
        2020-08-21T06:51:52.804+0200    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.9.0' as ILM is enabled.
        2020-08-21T06:51:52.804+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.211:9200
        2020-08-21T06:51:52.804+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.212:9200
        2020-08-21T06:51:52.804+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.213:9200
        2020-08-21T06:51:52.804+0200    INFO    [publisher]     pipeline/module.go:113  Beat name: ELASTIC99004
        2020-08-21T06:51:52.805+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.211:9200
        2020-08-21T06:51:52.805+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.212:9200
        2020-08-21T06:51:52.805+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.213:9200
        2020-08-21T06:51:52.885+0200    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.0
        2020-08-21T06:51:52.886+0200    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
        2020-08-21T06:51:53.094+0200    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
        2020-08-21T06:51:53.094+0200    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{filebeat-7.9.0 {now/d}-000001}' as ILM is enabled.
        2020-08-21T06:51:53.094+0200    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'filebeat-7.9.0-*' as ILM is enabled.
        2020-08-21T06:51:53.094+0200    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.9.0 {now/d}-000001} as ILM is enabled.
        2020-08-21T06:51:53.094+0200    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
        2020-08-21T06:51:53.096+0200    INFO    template/load.go:169    Existing template will be overwritten, as overwrite is enabled.
        2020-08-21T06:51:53.965+0200    INFO    template/load.go:109    Try loading template filebeat-7.9.0 to Elasticsearch
        2020-08-21T06:51:54.238+0200    INFO    template/load.go:101    template with name 'filebeat-7.9.0' loaded.
        2020-08-21T06:51:54.238+0200    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
        2020-08-21T06:51:54.239+0200    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
        Index setup finished.
        Loading dashboards (Kibana must be running and reachable)
        2020-08-21T06:51:54.240+0200    INFO    kibana/client.go:118    Kibana url: https://10.21.20.209:5601
        2020-08-21T06:51:55.453+0200    INFO    kibana/client.go:118    Kibana url: https://10.21.20.209:5601
        2020-08-21T06:53:02.318+0200    INFO    instance/beat.go:810    Kibana dashboards successfully loaded.
        Loaded dashboards
        2020-08-21T06:53:02.318+0200    WARN    [cfgwarn]       instance/beat.go:551    DEPRECATED: Setting up ML using Filebeat is going to be removed. Please use the ML app to setup jobs. Will be removed in version: 8.0.0
        Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
        See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
        2020-08-21T06:53:02.318+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.211:9200
        2020-08-21T06:53:02.318+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.212:9200
        2020-08-21T06:53:02.318+0200    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.21.20.213:9200
        2020-08-21T06:53:02.327+0200    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.0
        2020-08-21T06:53:02.328+0200    INFO    kibana/client.go:118    Kibana url: https://10.21.20.209:5601
        2020-08-21T06:53:02.410+0200    ERROR   instance/beat.go:951    Exiting: 3 errors: Error setting up ML for apache_ecs: cannot set up ML with prefix: filebeat-apache_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}; Error setting up ML for apache_ecs: cannot set up ML with prefix: filebeat-apache_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}; Error setting up ML for nginx_ecs: cannot set up ML with prefix: filebeat-nginx_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
        Exiting: 3 errors: Error setting up ML for apache_ecs: cannot set up ML with prefix: filebeat-apache_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}; Error setting up ML for apache_ecs: cannot set up ML with prefix: filebeat-apache_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}; Error setting up ML for nginx_ecs: cannot set up ML with prefix: filebeat-nginx_ecs-access-, response: {"statusCode":404,"error":"Not Found","message":"Not Found"}

Hello @warkolm
I got same error after upgraded to 7.9.1. And it was working fine in 7.8.1 before
Here is the log.

root@secure-dashboard:/home/cbarl# filebeat setup -e
2020-09-11T07:16:43.241Z        INFO    instance/beat.go:640    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-09-11T07:16:43.241Z        INFO    instance/beat.go:648    Beat ID: ed0bf284-1a80-421f-a8a7-d8f77cc961d4
2020-09-11T07:16:43.243Z        INFO    [beat]  instance/beat.go:976    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "ed0bf284-1a80-421f-a8a7-d8f77cc961d4"}}}
2020-09-11T07:16:43.243Z        INFO    [beat]  instance/beat.go:985    Build info      {"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T19:58:51.000Z", "version": "7.9.1"}}}
2020-09-11T07:16:43.243Z        INFO    [beat]  instance/beat.go:988    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.14.7"}}}
2020-09-11T07:16:43.244Z        INFO    [beat]  instance/beat.go:992    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-08-07T16:10:10Z","containerized":false,"name":"secure-dashboard","ip":["127.0.0.1/8","::1/128","10.142.0.31/32","fe80::4001:aff:fe8e:1f/64"],"kernel_version":"5.3.0-1032-gcp","mac":["42:01:0a:8e:00:1f"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.4 LTS (Bionic Beaver)","major":18,"minor":4,"patch":4,"codename":"bionic"},"timezone":"UTC","timezone_offset_sec":0,"id":"96aedd2e2630498599fa5826a99cfc92"}}}
2020-09-11T07:16:43.244Z        INFO    [beat]  instance/beat.go:1021   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/cbarl", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 22145, "ppid": 20558, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-09-11T07:16:42.190Z"}}}
2020-09-11T07:16:43.245Z        INFO    instance/beat.go:299    Setup Beat: filebeat; Version: 7.9.1
2020-09-11T07:16:43.245Z        INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.9.1' as ILM is enabled.
2020-09-11T07:16:43.245Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:16:43.246Z        INFO    [publisher]     pipeline/module.go:113  Beat name: secure-dashboard
2020-09-11T07:16:43.247Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:16:43.252Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2020-09-11T07:16:43.255Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:93     add_cloud_metadata: hosting provider type detected as gcp, metadata={"availability_zone":"us-east1-b","instance":{"id":"7983943716907689203","name":"secure-dashboard"},"machine":{"type":"custom-1-5376"},"project":{"id":"revamp-cyber"},"provider":"gcp"}
2020-09-11T07:16:43.277Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2020-09-11T07:16:43.282Z        INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-09-11T07:16:43.282Z        INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2020-09-11T07:16:43.282Z        INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{filebeat-7.9.1 {now/d}-000001}' as ILM is enabled.
2020-09-11T07:16:43.282Z        INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'filebeat-7.9.1-*' as ILM is enabled.
2020-09-11T07:16:43.282Z        INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.9.1 {now/d}-000001} as ILM is enabled.
2020-09-11T07:16:43.282Z        INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-09-11T07:16:43.284Z        INFO    template/load.go:169    Existing template will be overwritten, as overwrite is enabled.
2020-09-11T07:16:44.500Z        INFO    template/load.go:109    Try loading template filebeat-7.9.1 to Elasticsearch
2020-09-11T07:16:44.866Z        INFO    template/load.go:101    template with name 'filebeat-7.9.1' loaded.
2020-09-11T07:16:44.866Z        INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2020-09-11T07:16:44.869Z        INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-09-11T07:16:44.869Z        INFO    kibana/client.go:119    Kibana url: http://34.74.10.161:80
2020-09-11T07:16:47.075Z        INFO    kibana/client.go:119    Kibana url: http://34.74.10.161:80
2020-09-11T07:17:55.784Z        INFO    instance/beat.go:810    Kibana dashboards successfully loaded.
Loaded dashboards
2020-09-11T07:17:55.784Z        WARN    [cfgwarn]       instance/beat.go:551    DEPRECATED: Setting up ML using Filebeat is going to be removed. Please use the ML app to setup jobs. Will be removed in version: 8.0.0
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
2020-09-11T07:17:55.784Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:17:55.793Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
2020-09-11T07:17:55.793Z        INFO    kibana/client.go:119    Kibana url: http://34.74.10.161:80
2020-09-11T07:17:55.837Z        WARN    fileset/modules.go:421  X-Pack Machine Learning is not enabled
2020-09-11T07:17:55.863Z        WARN    fileset/modules.go:421  X-Pack Machine Learning is not enabled
2020-09-11T07:17:55.887Z        WARN    fileset/modules.go:421  X-Pack Machine Learning is not enabled
2020-09-11T07:17:55.917Z        WARN    fileset/modules.go:421  X-Pack Machine Learning is not enabled
Loaded machine learning job configurations
2020-09-11T07:17:55.917Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:17:55.920Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
2020-09-11T07:17:55.921Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:17:55.928Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
2020-09-11T07:17:56.045Z        INFO    fileset/pipelines.go:134        Elasticsearch pipeline with ID 'filebeat-7.9.1-mysql-slowlog-pipeline' loaded
2020-09-11T07:17:56.104Z        INFO    fileset/pipelines.go:134        Elasticsearch pipeline with ID 'filebeat-7.9.1-mysql-error-pipeline' loaded
2020-09-11T07:17:56.105Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:17:56.109Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
2020-09-11T07:17:56.233Z        INFO    fileset/pipelines.go:134        Elasticsearch pipeline with ID 'filebeat-7.9.1-nginx-access-pipeline' loaded
2020-09-11T07:17:56.275Z        INFO    fileset/pipelines.go:134        Elasticsearch pipeline with ID 'filebeat-7.9.1-nginx-error-pipeline' loaded
2020-09-11T07:17:56.277Z        INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-09-11T07:17:56.281Z        INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
2020-09-11T07:17:56.584Z        INFO    fileset/pipelines.go:134        Elasticsearch pipeline with ID 'filebeat-7.9.1-system-auth-pipeline' loaded
2020-09-11T07:17:56.659Z        INFO    fileset/pipelines.go:134        Elasticsearch pipeline with ID 'filebeat-7.9.1-system-syslog-pipeline' loaded
2020-09-11T07:17:56.660Z        INFO    cfgfile/reload.go:262   Loading of config files completed.
2020-09-11T07:17:56.660Z        INFO    [load]  cfgfile/list.go:124     Stopping 3 runners ...

so I didn't get any data in elasticsearch

root@secure-dashboard:/home/cbarl# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.