Snapshot creation to external fs runs into open file limit

Hi

Elastic 7.16

How can I limit open file descriptors while creating snapshots..?

My issue is, the external file share (hpe storeonce nfs share), we are using where to put our snapshots, it has open file handle limit. And we very often run into this limit and snapshots fails.

I have tried reduce'ing max_snapshot_bytes_per_sec from 40mb to 10mb. But when monitoring those handles, I do not see much difference.

Does Elasticsearch do index snapshots in parallel, or one index as a time..?
If parallel, isn't there a possibility to make snapshots only in serial, so if cluster has 10 indices, then it will do this one indice at a time, so when first one finishes, then the second, third, .. etc.

Regards
Raul

What is the file handle limit you're running into? There's a certain amount of concurrency but it's only max 5 files at once on each node, hardly a lot.

Limit is 600-ish.

In 7.16 single node development environment, where I only have less than 5 user created indexes (and others are system indices). When I take snapshot, I can easily see these file handles increasing to 120 already. Reducing max_snapshot_bytes_per_sec does not reduce either.

The issue for me actually happens on prod cluster, where I have 100gb data, it is version 6.8.16
When I have no snapshots present, and I create a first snapshot, these file handles immediately get into 600-ish.

Raul

Hmm 120 files open on a single node sounds suspicious. Can you run lsof on the Elasticsearch process and share the output?

Yes I can do that.
I will get to conputer within few hours.

Is there anything else you need me to share from that environment..?

Raul

A lsof showing all ~120 files open in the snapshot repository might be enough, let's see what that shows.

I did repository to localhost (not external share) and set max_bytes_per_sec to just 1mb at the moment, so I can run this lsof command multiple times (otherwise this snapshot just finished within 3 seconds).
I run this command 9 times, there were roughlt ~110 file handles for all the time.

Here is one of theose examples:

COMMAND      PID    TID TASKCMD            USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
java      374031                  elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374214 java      elasticsearch  358w      REG              253,0   2752512    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374215 GC\x20Thr elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374216 G1\x20Mai elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374217 G1\x20Con elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374218 GC\x20Thr elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374219 G1\x20Ref elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374220 G1\x20Ser elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374221 VM\x20Thr elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374222 Reference elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374223 Finalizer elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374224 Signal    elasticsearch  358w      REG              253,0   2883584    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374225 Service   elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374226 Monitor   elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374227 C2\x20Com elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374228 C1\x20Com elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374229 Sweeper   elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374230 Common-Cl elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374231 Notificat elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374232 VM\x20Per elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374241 process   elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374257 elasticse elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374258 elasticse elasticsearch  358w      REG              253,0   3014656    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374274 ml-cpp-lo elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374278 Connectio elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374279 elasticse elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374280 ticker-sc elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374281 elasticse elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374282 elasticse elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374288 elasticse elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374289 elasticse elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374290 G1\x20Ref elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374291 elasticse elasticsearch  358w      REG              253,0   3145728    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374292 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374295 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374296 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374297 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374298 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374299 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374300 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374301 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374303 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374306 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374307 elasticse elasticsearch  358w      REG              253,0   3276800    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374308 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374309 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374310 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374311 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374312 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374313 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374319 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374320 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374321 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374322 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374323 elasticse elasticsearch  358w      REG              253,0   3407872    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374324 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374325 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374326 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374327 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374328 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374329 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374330 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374331 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374332 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374333 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374334 elasticse elasticsearch  358w      REG              253,0   3538944    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374335 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374336 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374337 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374338 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374339 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374340 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374341 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374347 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374348 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374387 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374467 elasticse elasticsearch  358w      REG              253,0   3670016    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374476 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374485 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374506 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374732 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 374733 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 375387 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 379544 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 379545 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 379546 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458626 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458645 elasticse elasticsearch  358w      REG              253,0   3801088    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458647 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458665 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458666 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458877 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458878 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458879 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458880 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458881 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458882 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458883 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458884 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458889 elasticse elasticsearch  358w      REG              253,0   3932160    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458890 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458901 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458902 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458903 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458907 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458908 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 458909 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 459369 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 459388 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 459757 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g
java      374031 459766 elasticse elasticsearch  358w      REG              253,0   4063232    9851798 /mnt/backup/elastic/arendus-dtelastic/data/indices/0T759bC3Rti1TFUaNegs2w/0/__-LXxFAKYRwOB75VJKBb_3g

Snapshot that was taken, was 120mb in size.
And has 1192 file handlers, calculated like this
find /mnt/backup/elastic/arendus-dtelastic/data/ -type f | wc -l

Regards
Raul

But this is just one file descriptor, number 358. In a multithreaded system like Elasticsearch lsof reports every file against every thread.

Anyhow, seems like hpe backup solution cannot handle this stuff at all.

This alert is from 6.8.18 cluster, with just 100g data.
It will take it up to 600 and then alerts will start..

Subject: HPE StoreOnce 5100 System d2d.services_alert event

The following event occurred: d2d.services_alert

Severity:       ALERT
Event Code:     E08020016
Date:           Mon Feb 14 10:59:10 EET 2022
System URL(s):  


Event Message:	[ssid1] : Too many files open file handles on NAS share backup:/elastic/etelastic/data/indices/swMz6uMqRpOK5xwSBTEJvQ/0/__c59MXszSS1Gknr14Lz_joQ 
 Root Cause:		The NAS Share has too many files open. This is an internally configured value that cannot be modified by the customer. 
 Recommended Action:		Reduce the number of files open by the backup application by reducing the number of concurrent streams or number of simultaneous backup jobs

Regards
Raul

There's no denying that your storage is struggling, but is this really Elasticsearch's doing? If you can't find Elasticsearch holding too many file descriptors open with lsof then that would indicate Elasticsearch is not doing a lot of concurrent access so the problem lies elsewhere.

Your analysis might get further with the repository analysis API which has an explicit ?concurrency parameter.

Are those "_analyze" api parameters also allowed when creating the snapshot, or not..?
If not then these are no use unfortunately.

I have tested with this analyze property
And seems like the problem is those very tiny files (few kb or mb in size). When these are backed up, file handles change very rapidly, since these are so tiny. And, although Elasticsearch has already opened/closed like N amount of these files, our storage thinks many of these files are still opened.

For example, when I tested like this, I could clearly see these file handles being increased like 20-30.
curl -XPOST 'http://localhost:9200/_snapshot/arendus-dtelastic/_analyze?blob_count=100&max_blob_size=1mb&timeout=120s&pretty';

Adding &concurrency=1, does not change anything.

On the contrary, when I executed the command with these parameters, I could hardly see any difference regarding open file handle count:
?blob_count=10&max_blob_size=200mb

So moving larger chunks of data, is no issue.

Is there any hint, how to limit those tiny files being created..?

Regards
Raul

They're not, they're only there to help you characterise problems in your storage system.

If your storage system is reporting that files are still open even after they're closed then you'll need to report this to HPE and work with them to resolve it. There is no way to limit the rate of file creation in Elasticsearch - it's just not something that we expect properly-configured snapshot storage to care about.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.