Can't complete filebeat setup due to failed ML setup

Elastic Stack Beats
1

I'm getting an error whenever I run filebeat setup caused by filebeat being unable to setup ML jobs (that I don't even use). Here's the terminal output:

> sudo filebeat setup
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Exiting: 2 errors: Error setting up ML for apache2: 5 errors: [status_exception] A datafeed [datafeed-filebeat-apache2-access-response_code] already exists for job [filebeat-apache2-access-response_code]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-remote_ip_url_count] already exists for job [filebeat-apache2-access-remote_ip_url_count]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-visitor_rate] already exists for job [filebeat-apache2-access-visitor_rate]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-remote_ip_request_rate] already exists for job [filebeat-apache2-access-remote_ip_request_rate]; [status_exception] A datafeed [datafeed-filebeat-apache2-access-low_request_rate] already exists for job [filebeat-apache2-access-low_request_rate]; Error setting up ML for nginx: 5 errors: [status_exception] A datafeed [datafeed-filebeat-nginx-access-visitor_rate] already exists for job [filebeat-nginx-access-visitor_rate]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-response_code] already exists for job [filebeat-nginx-access-response_code]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-remote_ip_request_rate] already exists for job [filebeat-nginx-access-remote_ip_request_rate]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-remote_ip_url_count] already exists for job [filebeat-nginx-access-remote_ip_url_count]; [status_exception] A datafeed [datafeed-filebeat-nginx-access-low_request_rate] already exists for job [filebeat-nginx-access-low_request_rate]

My filebeat user has the following roles:

  • filebeat_client
  • kibana_system
  • machine_learning_admin

My filebeat_client role has the following cluster permissions:

  • monitor
  • manage_index_templates
  • manage_ml
  • manage_ingest_pipelines

And has all permissions for .kibana* and .filebeat* indexes.

I've been fussing with these permissions for a while trying to figure out what to do, so maybe these settings are pretty odd now. I would appreciate some help getting this sorted out! Honestly I'd be happy if i could just skip the ML jobs setup because I have no use for it.

It seems like this all came about because I added an index pattern to my config, and for some reason that causes ML jobs to be set up. I need the index pattern, but I really don't need the ML features.

I've tried running the setup in stages using the flags like filebeat setup --template etc and just skipping filebeat setup --machine-learning, but I still don't see any output in kibana.

I'm running elasticsearch & kibana 6.8.

Thanks :slight_smile:

2

Update: This might not have been the problem that was actually stopping my data from being indexed. I disabled indexing for my system logs and everything is working fine now. Seems like there were several services contending for access to auth.log, and that issue was just being masked by this setup failing, making me think that this was the main issue.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.