I'm getting an error whenever I run filebeat setup caused by filebeat being unable to setup ML jobs (that I don't even use). Here's the terminal output:
> sudo filebeat setup
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Exiting: 2 errors: Error setting up ML for apache2: 5 errors: [status_exception] A datafeed [datafeed-filebeat-apache2-access-response_code] already exists for job [filebeat-apache2-access-response_code]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-remote_ip_url_count] already exists for job [filebeat-apache2-access-remote_ip_url_count]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-visitor_rate] already exists for job [filebeat-apache2-access-visitor_rate]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-remote_ip_request_rate] already exists for job [filebeat-apache2-access-remote_ip_request_rate]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-low_request_rate] already exists for job [filebeat-apache2-access-low_request_rate]; Error setting up ML for nginx: 5 errors: [status_exception] A datafeed [datafeed-filebeat-nginx-access-visitor_rate] already exists for job [filebeat-nginx-access-visitor_rate]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-response_code] already exists for job [filebeat-nginx-access-response_code]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists for job [filebeat-nginx-access-remote_ip_request_rate]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists for job [filebeat-nginx-access-remote_ip_url_count]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-low_request_rate] already exists for job [filebeat-nginx-access-low_request_rate]
My filebeat user has the following roles:
- filebeat_client
- kibana_system
- machine_learning_admin
My filebeat_client role has the following cluster permissions:
- monitor
- manage_index_templates
- manage_ml
- manage_ingest_pipelines
And has all permissions for .kibana* and .filebeat* indexes.
I've been fussing with these permissions for a while trying to figure out what to do, so maybe these settings are pretty odd now. I would appreciate some help getting this sorted out! Honestly I'd be happy if i could just skip the ML jobs setup because I have no use for it.
It seems like this all came about because I added an index pattern to my config, and for some reason that causes ML jobs to be set up. I need the index pattern, but I really don't need the ML features.
I've tried running the setup in stages using the flags like filebeat setup --template etc and just skipping filebeat setup --machine-learning, but I still don't see any output in kibana.
I'm running elasticsearch & kibana 6.8.
Thanks 