Too many open files even after increasing limit

Elastic Stack Elasticsearch
1

Hi all,

I'm running into a 'Too many open files' issue even after I've greatly
increased the file limit on each node of the ES cluster. The set up is 6
storage nodes with around 630 million total indexed documents.

Here's the node output showing that the limit is indeed 65,535 per node
(500,000 on one where I was testing):

{
"ok" : true,
"cluster_name" : "Odine",
"nodes" : {
"J_vVn1gKQMmcDYADbcPlfw" : {
"name" : "rap-es-1",
"transport_address" : "inet[/141.161.88.174:9300]",
"hostname" : "rap-es-1.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.174:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 31729,
"max_file_descriptors" : 65535
}
},
"hT_Z-JneS6uCem0M-7nTEQ" : {
"name" : "rap-es",
"transport_address" : "inet[/141.161.88.169:9300]",
"hostname" : "rap-es.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.169:9200]",
"attributes" : {
"data" : "false"
},
"os" : {
"refresh_interval" : 1000,
"available_processors" : 2,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 2,
"total_sockets" : 2,
"cores_per_socket" : 1,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255303680
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 10282,
"max_file_descriptors" : 500000
}
},
"-76vB5ISTUq4M7ESizP7og" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9301]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2462,
"max_file_descriptors" : 65535
}
},
"xssBGLKITpicDcxdeEjr4A" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9300]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2608,
"max_file_descriptors" : 65535
}
},
"wg_I5wBfRxqs6kdH-S_kww" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9301]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2570,
"max_file_descriptors" : 65535
}
},
"lG0sqr95Q02Lt-iTCcg6qg" : {
"name" : "rap-es-5",
"transport_address" : "inet[/141.161.88.178:9300]",
"hostname" : "rap-es-5.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.178:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 22580,
"max_file_descriptors" : 65535
}
},
"biDQwwn4R9GTkesZUlNW1g" : {
"name" : "rap-es-2",
"transport_address" : "inet[/141.161.88.175:9300]",
"hostname" : "rap-es-2.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.175:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 5190,
"max_file_descriptors" : 65535
}
},
"A7E7I7R5SIib6RJPA3RNug" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9300]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2500,
"max_file_descriptors" : 65535
}
},
"oQG-yfURTTWZiFC6eOwjAg" : {
"name" : "rap-es-3",
"transport_address" : "inet[/141.161.88.176:9300]",
"hostname" : "rap-es-3.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.176:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 1291,
"max_file_descriptors" : 65535
}
}
}
}

Any idea why I'm still getting this error?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Probably too many shards per node, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 20:00, Aaron Troy atoy3731@gmail.com a écrit :

Hi all,

I'm running into a 'Too many open files' issue even after I've greatly increased the file limit on each node of the ES cluster. The set up is 6 storage nodes with around 630 million total indexed documents.

Here's the node output showing that the limit is indeed 65,535 per node (500,000 on one where I was testing):

{
"ok" : true,
"cluster_name" : "Odine",
"nodes" : {
"J_vVn1gKQMmcDYADbcPlfw" : {
"name" : "rap-es-1",
"transport_address" : "inet[/141.161.88.174:9300]",
"hostname" : "rap-es-1.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.174:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 31729,
"max_file_descriptors" : 65535
}
},
"hT_Z-JneS6uCem0M-7nTEQ" : {
"name" : "rap-es",
"transport_address" : "inet[/141.161.88.169:9300]",
"hostname" : "rap-es.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.169:9200]",
"attributes" : {
"data" : "false"
},
"os" : {
"refresh_interval" : 1000,
"available_processors" : 2,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 2,
"total_sockets" : 2,
"cores_per_socket" : 1,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255303680
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 10282,
"max_file_descriptors" : 500000
}
},
"-76vB5ISTUq4M7ESizP7og" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9301]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2462,
"max_file_descriptors" : 65535
}
},
"xssBGLKITpicDcxdeEjr4A" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9300]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2608,
"max_file_descriptors" : 65535
}
},
"wg_I5wBfRxqs6kdH-S_kww" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9301]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2570,
"max_file_descriptors" : 65535
}
},
"lG0sqr95Q02Lt-iTCcg6qg" : {
"name" : "rap-es-5",
"transport_address" : "inet[/141.161.88.178:9300]",
"hostname" : "rap-es-5.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.178:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 22580,
"max_file_descriptors" : 65535
}
},
"biDQwwn4R9GTkesZUlNW1g" : {
"name" : "rap-es-2",
"transport_address" : "inet[/141.161.88.175:9300]",
"hostname" : "rap-es-2.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.175:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 5190,
"max_file_descriptors" : 65535
}
},
"A7E7I7R5SIib6RJPA3RNug" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9300]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2500,
"max_file_descriptors" : 65535
}
},
"oQG-yfURTTWZiFC6eOwjAg" : {
"name" : "rap-es-3",
"transport_address" : "inet[/141.161.88.176:9300]",
"hostname" : "rap-es-3.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.176:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 1291,
"max_file_descriptors" : 65535
}
}
}
}

Any idea why I'm still getting this error?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Ah, perhaps that is the issue.

Right now I have 48 indexes, each with 12 shards randomly allocated across
the nodes. Would reducing the number of shards per index (perhaps 6) help
the issue with open files?

On Monday, November 11, 2013 2:59:48 PM UTC-5, David Pilato wrote:

Probably too many shards per node, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 20:00, Aaron Troy <atoy...@gmail.com <javascript:>> a
écrit :

Hi all,

I'm running into a 'Too many open files' issue even after I've greatly
increased the file limit on each node of the ES cluster. The set up is 6
storage nodes with around 630 million total indexed documents.

Here's the node output showing that the limit is indeed 65,535 per node
(500,000 on one where I was testing):

{
"ok" : true,
"cluster_name" : "Odine",
"nodes" : {
"J_vVn1gKQMmcDYADbcPlfw" : {
"name" : "rap-es-1",
"transport_address" : "inet[/141.161.88.174:9300]",
"hostname" : "rap-es-1.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.174:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 31729,
"max_file_descriptors" : 65535
}
},
"hT_Z-JneS6uCem0M-7nTEQ" : {
"name" : "rap-es",
"transport_address" : "inet[/141.161.88.169:9300]",
"hostname" : "rap-es.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.169:9200]",
"attributes" : {
"data" : "false"
},
"os" : {
"refresh_interval" : 1000,
"available_processors" : 2,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 2,
"total_sockets" : 2,
"cores_per_socket" : 1,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255303680
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 10282,
"max_file_descriptors" : 500000
}
},
"-76vB5ISTUq4M7ESizP7og" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9301]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2462,
"max_file_descriptors" : 65535
}
},
"xssBGLKITpicDcxdeEjr4A" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9300]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2608,
"max_file_descriptors" : 65535
}
},
"wg_I5wBfRxqs6kdH-S_kww" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9301]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2570,
"max_file_descriptors" : 65535
}
},
"lG0sqr95Q02Lt-iTCcg6qg" : {
"name" : "rap-es-5",
"transport_address" : "inet[/141.161.88.178:9300]",
"hostname" : "rap-es-5.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.178:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 22580,
"max_file_descriptors" : 65535
}
},
"biDQwwn4R9GTkesZUlNW1g" : {
"name" : "rap-es-2",
"transport_address" : "inet[/141.161.88.175:9300]",
"hostname" : "rap-es-2.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.175:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 5190,
"max_file_descriptors" : 65535
}
},
"A7E7I7R5SIib6RJPA3RNug" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9300]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2500,
"max_file_descriptors" : 65535
}
},
"oQG-yfURTTWZiFC6eOwjAg" : {
"name" : "rap-es-3",
"transport_address" : "inet[/141.161.88.176:9300]",
"hostname" : "rap-es-3.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.176:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 1291,
"max_file_descriptors" : 65535
}
}
}
}

Any idea why I'm still getting this error?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

And also how many replica do you have?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 21:17, Aaron Troy atoy3731@gmail.com a écrit :

Ah, perhaps that is the issue.

Right now I have 48 indexes, each with 12 shards randomly allocated across the nodes. Would reducing the number of shards per index (perhaps 6) help the issue with open files?

On Monday, November 11, 2013 2:59:48 PM UTC-5, David Pilato wrote:

Probably too many shards per node, right?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 20:00, Aaron Troy atoy...@gmail.com a écrit :

Hi all,

I'm running into a 'Too many open files' issue even after I've greatly increased the file limit on each node of the ES cluster. The set up is 6 storage nodes with around 630 million total indexed documents.

Here's the node output showing that the limit is indeed 65,535 per node (500,000 on one where I was testing):

{
"ok" : true,
"cluster_name" : "Odine",
"nodes" : {
"J_vVn1gKQMmcDYADbcPlfw" : {
"name" : "rap-es-1",
"transport_address" : "inet[/141.161.88.174:9300]",
"hostname" : "rap-es-1.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.174:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 31729,
"max_file_descriptors" : 65535
}
},
"hT_Z-JneS6uCem0M-7nTEQ" : {
"name" : "rap-es",
"transport_address" : "inet[/141.161.88.169:9300]",
"hostname" : "rap-es.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.169:9200]",
"attributes" : {
"data" : "false"
},
"os" : {
"refresh_interval" : 1000,
"available_processors" : 2,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 2,
"total_sockets" : 2,
"cores_per_socket" : 1,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255303680
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 10282,
"max_file_descriptors" : 500000
}
},
"-76vB5ISTUq4M7ESizP7og" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9301]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2462,
"max_file_descriptors" : 65535
}
},
"xssBGLKITpicDcxdeEjr4A" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9300]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2608,
"max_file_descriptors" : 65535
}
},
"wg_I5wBfRxqs6kdH-S_kww" : {
"name" : "rap-es-6",
"transport_address" : "inet[/141.161.88.179:9301]",
"hostname" : "rap-es-6.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.179:9201]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2570,
"max_file_descriptors" : 65535
}
},
"lG0sqr95Q02Lt-iTCcg6qg" : {
"name" : "rap-es-5",
"transport_address" : "inet[/141.161.88.178:9300]",
"hostname" : "rap-es-5.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.178:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 22580,
"max_file_descriptors" : 65535
}
},
"biDQwwn4R9GTkesZUlNW1g" : {
"name" : "rap-es-2",
"transport_address" : "inet[/141.161.88.175:9300]",
"hostname" : "rap-es-2.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.175:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 5190,
"max_file_descriptors" : 65535
}
},
"A7E7I7R5SIib6RJPA3RNug" : {
"name" : "rap-es-4",
"transport_address" : "inet[/141.161.88.177:9300]",
"hostname" : "rap-es-4.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.177:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 2500,
"max_file_descriptors" : 65535
}
},
"oQG-yfURTTWZiFC6eOwjAg" : {
"name" : "rap-es-3",
"transport_address" : "inet[/141.161.88.176:9300]",
"hostname" : "rap-es-3.uis.georgetown.edu",
"version" : "0.90.3",
"http_address" : "inet[/141.161.88.176:9200]",
"os" : {
"refresh_interval" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 1995,
"total_cores" : 4,
"total_sockets" : 1,
"cores_per_socket" : 4,
"cache_size" : "18kb",
"cache_size_in_bytes" : 18432
},
"mem" : {
"total" : "7.6gb",
"total_in_bytes" : 8255045632
},
"swap" : {
"total" : "7.9gb",
"total_in_bytes" : 8589926400
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 1291,
"max_file_descriptors" : 65535
}
}
}
}

Any idea why I'm still getting this error?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Only one replica per shard.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

So you have 1152 shards on 6 nodes.
About 200 shards per node.

Each shard needs many file descriptors for files but for socket as well.

I would really reduce the number of shard per node. Which means to me:
Decrease the number of shard (don't over allocate)
Increase the number of nodes

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 21:40, Aaron Troy atoy3731@gmail.com a écrit :

Only one replica per shard.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Thanks for the input, David. As the system is still ingesting and
collecting data, I think decreasing the number of shards per index is the
way to go as of now.

Guess I'd better start reindexing.

Thanks again!

On Monday, November 11, 2013 3:55:26 PM UTC-5, David Pilato wrote:

So you have 1152 shards on 6 nodes.
About 200 shards per node.

Each shard needs many file descriptors for files but for socket as well.

I would really reduce the number of shard per node. Which means to me:
Decrease the number of shard (don't over allocate)
Increase the number of nodes

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 21:40, Aaron Troy <atoy...@gmail.com <javascript:>> a
écrit :

Only one replica per shard.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Aaron - also, if you aren't already, be sure to checkout some of the site
plugins available to help review various performance metrics:
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, November 11, 2013 4:10:22 PM UTC-5, Aaron Troy wrote:

Thanks for the input, David. As the system is still ingesting and
collecting data, I think decreasing the number of shards per index is the
way to go as of now.

Guess I'd better start reindexing.

Thanks again!

On Monday, November 11, 2013 3:55:26 PM UTC-5, David Pilato wrote:

So you have 1152 shards on 6 nodes.
About 200 shards per node.

Each shard needs many file descriptors for files but for socket as well.

I would really reduce the number of shard per node. Which means to me:
Decrease the number of shard (don't over allocate)
Increase the number of nodes

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 nov. 2013 à 21:40, Aaron Troy atoy...@gmail.com a écrit :

Only one replica per shard.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.