One filebeat instance marking output cnx failed

[stefws (Steffen Winther Sørensen)]

Hi

Got multiple filebeat instance +50, all deployed from same home built rpm, now suddenly one instance are failing to ingest it filestreams, complaining like this:

{"log.level":"error","@timestamp":"2024-03-15T12:44:54.629+0100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://<redacted>:9200)): Connection marked as failed because the onConnect callback failed: 1 error: error loading pipeline for fileset elasticsearch/deprecation: couldn't load pipeline: couldn't load json. Error: 403 Forbidden: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [epj_ingester] with effective roles [epj_ingesting], this action is granted by the cluster privileges [manage_ingest_pipelines,manage_pipeline,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [epj_ingester] with effective roles [epj_ingesting], this action is granted by the cluster privileges [manage_ingest_pipelines,manage_pipeline,manage,all]\"},\"status\":403}. Response body: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [epj_ingester] with effective roles [epj_ingesting], this action is granted by the cluster privileges [manage_ingest_pipelines,manage_pipeline,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ingest/pipeline/put] is unauthorized for user [epj_ingester] with effective roles [epj_ingesting], this action is granted by the cluster privileges [manage_ingest_pipelines,manage_pipeline,manage,all]\"},\"status\":403}","service.name":"filebeat","ecs.version":"1.6.0"}

Appreciate hints on how to figure out why this instance now suddenly seems need more privilleges like to manage pipelines, as all filebeat instances are using the same user to ingest by and ingesting just fine.

This instance are running on a server that resently change it's IP address, but so are other instances and they are ingesting just fine after IP address change.

TIA

[rtwolfe94022 (Tim Wolfe)]

Steffen,
The error indicates that the user epj_ingester lacks permission to manage Elasticsearch ingest pipelines. To fix this, ensure the user has the necessary permissions (manage_ingest_pipelines , manage_pipeline , manage , all ). Check the user's roles and permissions in Kibana or via the Elasticsearch API, and adjust as needed to grant the correct privileges. Also, verify that the Filebeat configuration is consistent across all instances, especially regarding pipeline usage, and ensure no network or firewall settings are affecting connectivity after the IP address change.

[stefws (Steffen Winther Sørensen)]

@rtwolfe94022

manage_ingest_pipelines

Thanks, Tim Wolf, I've added this privilige, it seems to fix it, just wondering as this role hasn't changed and been used for months ingesting from all my ingest instances of either filebeat or metricbeat and only now found this single filebeat instance having this issue and it deployed via ansible as all my instances are and from the same rpm package.

Tried yesterday with a 'dnf erase + ansible reployment', and this made it ingest but it still complained once in the filebeat log though, but it stopped ingesting after a 'systemctl restart' which also happened after the IP change + reboot. just wondering why the suddenly if seems to need more priviliges to ingest

But anyway now it's ingesting again, which is the main thing right now... would just like to keep priviliges at bare minimum