[logstash.outputs.opensearch][main][9182 Retrying individual bulk actions that failed or were rejected by the previous bulk request {:count=>125}

Elastic Stack Elasticsearch
1

bin/logstash -f /opt/logstash-8.9.0/config/conf.d/pipe.conf

[2024-06-13T12:42:19,876][INFO ][logstash.outputs.opensearch][main][91823b58f6fb0958c7671051e63dcf1ae4101824338810d25e044a5a52cd36e4] Retrying failed action {:status=>429, :action=>["index", {:_id=>nil, :_index=>"rpmappserver-01ribbideo_logs", :routing=>nil}, {"event"=>{"original"=>"10.30.1.132 - - [10/Sep/2023:01:00:02 -0700] "GET /rx/api/config HTTP/1.1" 200 2\n"}, "message"=>"10.30.1.132 - - [10/Sep/2023:01:00:02 -0700] "GET /rx/api/config HTTP/1.1" 200 2\n", "@version"=>"1", "type"=>"s3", "@timestamp"=>2024-06-13T12:40:10.852904110Z}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [rpmappserver-01ribbideo_logs] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}

[logstash.outputs.opensearch][main][91823b58f6fb0958c7671051e63dcf1ae4101824338810d25e044a5a52cd36e4] Retrying individual bulk actions that failed or were rejected by the previous bulk request {:count=>125}

cd /opt/logstash-8.9.0/config/conf.d
vi pipe.conf
input {
s3 {
access_key_id => "AKIAUNWJFISALFXWOPVQ"
secret_access_key => "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"
bucket => "serverlogs-01backup"
region => "us-west-2"
type => "s3"
prefix => "appserver/rpmappserver-01/ribbideo_logs/"
interval => 60
}

s3 {
access_key_id => "AKIAUNWJFISALFXWOPVQ"
secret_access_key => "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"
bucket => "serverlogs-01backup"
region => "us-west-2"
type => "s3"
prefix => "appserver/rpmappserver-01/tomcat8_logs/"
interval => 60
}
}

filter {

Add any necessary filters here to parse or enrich your data

}

output {
opensearch {
hosts => ["https://localhost:9200"]
index => "rpmappserver-01ribbideo_logs"
user => "admin"
password => "admin"
ssl_certificate_verification => false
}

opensearch {
hosts => ["https://localhost:9200"]
index => "rpmappserver-01tomcat8_logs"
user => "admin"
password => "admin"
ssl_certificate_verification => false
}
}

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

pls help me , how to fix this

4

Opensearch is not supported here, but this is your issue, your disk does not have enough space, you need to clean up some space on your disk.

5

Looks like your server has run out of disk space.

6

already i added new volume to my server

/dev/xvdd 69G 53M 66G 1% /opt/new_volume

7

-bash-4.2# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 7.9G 528K 7.9G 1% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/xvda1 71G 67G 4.8G 94% /
/dev/xvdc 12G 893M 11G 8% /var/lib/mongo
/dev/xvdb 12G 41M 12G 1% /usr/local/ribb ideo/logs
/dev/xvdf 12G 2.1G 10G 18% /data
tmpfs 7.9G 249M 7.6G 4% /tmp
fs-56b3fdfc.efs.us-west-2.amazonaws.com:confused: 8.0E 1.8G 8.0E 1% /mnt/staging/ef s
/dev/xvdd 69G 53M 66G 1% /opt/new_volume
tmpfs 1.6G 0 1.6G 0% /run/user/1000

8

I don't think this makes any difference, the issue is related to the space where your data is being saved.

As mentioned Opensearch is not supported here, you need to check where the data is being saved and increase that partition or remove any data.

I would assume that it is saving under /var/lib.