Filebeat upload log to es5 , 403

Elastic Stack Elasticsearch
1

I use es 5.6.4 , filebeat upload log to es, return 403 ,What could cause this?
The log is as follows:

2020-09-11T14:14:25+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:25+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:25+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:25+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:26+08:00 INFO Connected to Elasticsearch version 5.6.4

2020-09-11T14:14:26+08:00 INFO Connected to Elasticsearch version 5.6.4

2020-09-11T14:14:26+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:26+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:26+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:26+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:27+08:00 INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=33756 libbeat.es.call_count.PublishEvents=58 libbeat.es.publish.write_bytes=19695408

2020-09-11T14:14:27+08:00 INFO Connected to Elasticsearch version 5.6.4

2020-09-11T14:14:27+08:00 INFO Connected to Elasticsearch version 5.6.4

2020-09-11T14:14:27+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:27+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:27+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:27+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:28+08:00 INFO Connected to Elasticsearch version 5.6.4

2020-09-11T14:14:28+08:00 INFO Connected to Elasticsearch version 5.6.4

2020-09-11T14:14:28+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:28+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:28+08:00 ERR Failed to perform any bulk index operations: 403 Forbidden

2020-09-11T14:14:28+08:00 INFO Error publishing events (retrying): 403 Forbidden

2020-09-11T14:14:29+08:00 INFO Connected to Elasticsearch version 5.6.4

2020
2

Please check your Elasticsearch logs. Is it possible that the cluster has run out of space and gone into read-only mode?

3

There is plenty of space, are there other possibilities?

4

Is there anything else in the Elasticsearch logs?

5 
[2020-09-11T14:41:39,450][INFO ][o.e.p.o.OPackActionFilter] [node1] forbidden request: { ID:51f50c69e1fc4bf4b932eb4cdb67e4b5, TYP:BulkRequest, USR:tsf-es-1, BRS:true, ACT:indices:data/write/bulk, OA:22.188.116.201, IDX:filebeat-2020.09.11, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=334391>, HDR:Authorization,Accept,User-Agent,Host,Accept-Encoding,Content-Length,Content-Type, EFF:0 } Reason: null
[2020-09-11T14:41:39,457][INFO ][o.e.p.o.OPackActionFilter] [node1] forbidden request: { ID:c527c41c71da488fb4515c9b18de852b, TYP:BulkRequest, USR:tsf-es-1, BRS:true, ACT:indices:data/write/bulk, OA:22.188.116.216, IDX:filebeat-2020.09.11, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=47541>, HDR:Authorization,Accept,User-Agent,Host,Accept-Encoding,Content-Length,Content-Type, EFF:0 } Reason: null
[2020-09-11T14:41:39,564][INFO ][o.e.p.o.OPackActionFilter] [node1] forbidden request: { ID:221f1e0a05284003adf0d2966279b9e2, TYP:BulkRequest, USR:tsf-es-1, BRS:true, ACT:indices:data/write/bulk, OA:22.188.116.203, IDX:filebeat-2020.09.11, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=22007>, HDR:Authorization,Accept,User-Agent,Host,Accept-Encoding,Content-Length,Content-Type, EFF:0 } Reason: null
[2020-09-11T14:41:39,566][INFO ][o.e.p.o.OPackActionFilter] [node1] forbidden request: { ID:e859d0ab25564c079674d20712481bcd, TYP:BulkRequest, USR:tsf-es-1, BRS:true, ACT:indices:data/write/bulk, OA:22.188.116.217, IDX:filebeat-2020.09.11, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=20775>, HDR:Authorization,Accept,User-Agent,Host,Accept-Encoding,Content-Length,Content-Type, EFF:0 } Reason: null
[2020-09-11T14:41:39,569][INFO ][o.e.p.o.OPackActionFilter] [node1] forbidden request: { ID:61ee1415444842f8a8bf33745af1ffc6, TYP:BulkRequest, USR:tsf-es-1, BRS:true, ACT:indices:data/write/bulk, OA:22.188.116.203, IDX:filebeat-2020.09.11, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=22007>, HDR:Authorization,Accept,User-Agent,Host,Accept-Encoding,Content-Length,Content-Type, EFF:0 } Reason: null
[2020-09-11T14:41:39,569][INFO ][o.e.p.o.OPackActionFilte
6 
[2020-09-11T14:43:23,562][DEBUG][o.e.a.b.TransportBulkAction] [node1] failed to execute pipeline [tsf-business-1-l6ymbvgd] for document [filebeat-2020.09.11/log/null]
java.lang.IllegalArgumentException: pipeline with id [tsf-business-1-l6ymbvgd] does not exist
        at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:194) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.4.jar:5.6.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
[2020-09-11T14:43:23,563][DEBUG][o.e.a.b.TransportBulkAction] [node1] failed to execute pipeline [tsf-business-1-l6ymbvgd] for document [filebeat-2020.09.11/log/null]
java.lang.IllegalArgumentException: pipeline with id [tsf-business-1-l6ymbvgd] does not exist
        at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:194) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
7

Which version of Filebeat are you using to ingest data into the cluster?

It seems the bulk requests specify an invalid ingest pipeline.

8

filebeat 5.0.0

9

Has the ingest pipeline mentioned in the logs been deployed?

10

Welcome to our community! :smiley:

Pleaser look at upgrading your stack ASAP. 5.X has been EOL for quite some time and is no longer supported.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.