# ES 5.2.2 crashes daily in docker **URL:** \ **Category:** Elasticsearch\ **Created:** [March 27, 2017, 8:45am UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096 "2017-03-27T08:45:07Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![expert](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@expert](https://discuss.elastic.co/u/expert)\ **Post date:** [March 27, 2017, 8:45am UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/1 "2017-03-27T08:45:07Z")
Hey guys, For some reason ES 5.2.2 crashes daily in docker for me. Here is how I launch it > docker run --name search -d --net=host -p 9200:9200 -p 9300:9300 -e "http.host=127.0.0.1" -e "transport.host=127.0.0.1" -v /jdata/elastic/data:/usr/share/elasticsearch/data -v /jdata/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml [docker.elastic.co/elasticsearch/elasticsearch:5.2.2](http://docker.elastic.co/elasticsearch/elasticsearch:5.2.2) my config doesn't have much > transport.tcp.port: 9300 > http.port: 9200 > client.transport.ignore\_cluster\_name: false > client.transport.sniff: false > discovery.zen.minimum\_master\_nodes: 1 > xpack.security.enabled: false and here what I have in log before it dies. > [2017-03-26T19:08:07,160][INFO][o.e.n.Node] initializing ... > [2017-03-26T19:08:07,317][INFO][o.e.e.NodeEnvironment] [1WEhN6j] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdb)]], net usable\_space [294.6gb], net total\_space [492gb], spins? [possibly], types [ext3] > [2017-03-26T19:08:07,318][INFO][o.e.e.NodeEnvironment] [1WEhN6j] heap size [1.9gb], compressed ordinary object pointers [true] > [2017-03-26T19:08:07,368][INFO][o.e.n.Node] node name [1WEhN6j] derived from node ID [1WEhN6juR--PB5kOwCTJQA]; set [node.name] to override > [2017-03-26T19:08:07,374][INFO][o.e.n.Node] version[5.2.2], pid[1], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/4.4.19-29.55.amzn1.x86\_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_92-internal/25.92-b14] > [2017-03-26T19:08:12,275][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [aggs-matrix-stats] > [2017-03-26T19:08:12,275][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [ingest-common] > [2017-03-26T19:08:12,276][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [lang-expression] > [2017-03-26T19:08:12,276][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [lang-groovy] > [2017-03-26T19:08:12,276][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [lang-mustache] > [2017-03-26T19:08:12,276][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [lang-painless] > [2017-03-26T19:08:12,277][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [percolator] > [2017-03-26T19:08:12,277][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [reindex] > [2017-03-26T19:08:12,277][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [transport-netty3] > [2017-03-26T19:08:12,278][INFO][o.e.p.PluginsService] [1WEhN6j] loaded module [transport-netty4] > [2017-03-26T19:08:12,280][INFO][o.e.p.PluginsService] [1WEhN6j] loaded plugin [x-pack] > [2017-03-26T19:08:13,382][WARN][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead > [2017-03-26T19:08:21,064][INFO][o.e.n.Node] initialized > [2017-03-26T19:08:21,066][INFO][o.e.n.Node] [1WEhN6j] starting ... > [2017-03-26T19:08:21,976][WARN][i.n.u.i.MacAddressUtil] Failed to find a usable hardware address from the network interfaces; using random bytes: a8:90:75:a9:3a:9a:65:f3 > [2017-03-26T19:08:22,269][INFO][o.e.t.TransportService] [1WEhN6j] publish\_address {127.0.0.1:9300}, bound\_addresses {127.0.0.1:9300} > [2017-03-26T19:08:22,298][WARN][o.e.b.BootstrapChecks] [1WEhN6j] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] > [2017-03-26T19:08:22,298][WARN][o.e.b.BootstrapChecks] [1WEhN6j] max virtual memory areas vm.max\_map\_count [65530] is too low, increase to at least [262144] > [2017-03-26T19:08:25,499][INFO][o.e.c.s.ClusterService] [1WEhN6j] new\_master {1WEhN6j}{1WEhN6juR--PB5kOwCTJQA}{f3tBrOMKQYqTptKjSgwDgg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) > [2017-03-26T19:08:25,585][INFO][o.e.h.HttpServer] [1WEhN6j] publish\_address {127.0.0.1:9200}, bound\_addresses {127.0.0.1:9200} > [2017-03-26T19:08:25,586][INFO][o.e.n.Node] [1WEhN6j] started > [2017-03-26T19:08:27,631][INFO][o.e.l.LicenseService] [1WEhN6j] license [fba9ada3-6300-45ee-98b5-bbf355d8cfe3] mode [trial] - valid > [2017-03-26T19:08:27,670][INFO][o.e.g.GatewayService] [1WEhN6j] recovered [6] indices into cluster\_state > [2017-03-26T19:09:32,636][ERROR][o.e.x.m.AgentService] [1WEhN6j] exception when exporting documents > org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks > at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.2.2.jar:5.2.2] > at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.2.2.jar:5.2.2] > at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:183) ~[x-pack-5.2.2.jar:5.2.2] > at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:196) [x-pack-5.2.2.jar:5.2.2] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_92-internal] > Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default\_local] > at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?] > at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?] > at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?] > ... 4 more > Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default\_local] reports failures when exporting documents > at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:121) ~[?:?] > at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:111) ~[?:?] > at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?] > at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?] > ... 4 more > [2017-03-26T19:10:15,948][INFO][o.e.c.r.a.AllocationService] [1WEhN6j] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-2-2017.03.26][0]] ...]). I have ~76M small documents in my ES. Total size of DB files is ~35GB What should I do ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 27, 2017, 9:19am UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/2 "2017-03-27T09:19:38Z")
I'd probably start by fixing all the warnings which appears in your logs (bootstrap) ---
**Author:** ![expert](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@expert](https://discuss.elastic.co/u/expert)\ **Post date:** [March 27, 2017, 9:53am UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/3 "2017-03-27T09:53:49Z")
Setting > sudo sysctl -w vm.max\_map\_count=262144 helped to get rid of > max virtual memory areas vm.max\_map\_count [65530] is too low, increase to at least [262144] but I still can't find how to increase max file descriptors. In the log it says > max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] but I don't see such number in host system > [expert@ip-172-30-0-246 ~]$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 31863 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 31863 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited Thoughts ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 27, 2017, 12:29pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/4 "2017-03-27T12:29:13Z")
This can help? [https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html) ---
**Author:** ![expert](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@expert](https://discuss.elastic.co/u/expert)\ **Post date:** [March 27, 2017, 12:51pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/5 "2017-03-27T12:51:51Z")
I fixed limit with docker's `--ulimit nofile=65536:65536`. So you think ES was crashing because of these limits ? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 27, 2017, 1:25pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/6 "2017-03-27T13:25:33Z")
Probably. At least going in production without those settings is really bad. ---
**Author:** ![expert](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@expert](https://discuss.elastic.co/u/expert)\ **Post date:** [April 2, 2017, 5:39pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/7 "2017-04-02T17:39:40Z")
Well, sadly it didn't help. But I got exception in the log. Maybe it's related ? ``` [2017-04-01T20:05:49,483][INFO][o.e.g.GatewayService] [1WEhN6j] recovered [9] indices into cluster_state [2017-04-01T20:06:54,318][ERROR][o.e.x.m.AgentService] [1WEhN6j] exception when exporting documents org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.2.2.jar:5.2.2] at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.2.2.jar:5.2.2] at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:183) ~[x-pack-5.2.2.jar:5.2.2] at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:196) [x-pack-5.2.2.jar:5.2.2] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92-internal] Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?] ... 4 more Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:121) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:111) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?] ... 4 more [2017-04-01T20:07:00,129][INFO][o.e.c.r.a.AllocationService] [1WEhN6j] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-2-2017.04.01][0]] ...]). [2017-04-02T00:00:03,652][INFO][o.e.c.m.MetaDataCreateIndexService] [1WEhN6j] [.monitoring-es-2-2017.04.02] creating index, cause [auto(bulk api)], templates [.monitoring-es-2], shards [1]/[1], mappings [shards, _default_, node, index_stats, index_recovery, cluster_state, cluster_stats, indices_stats, node_stats] [2017-04-02T00:00:03,765][INFO][o.e.c.m.MetaDataMappingService] [1WEhN6j] [.monitoring-es-2-2017.04.02/lZ9x2xIWTRywMr1HbyqYxw] update_mapping [cluster_stats] [2017-04-02T00:00:03,816][INFO][o.e.c.m.MetaDataMappingService] [1WEhN6j] [.monitoring-es-2-2017.04.02/lZ9x2xIWTRywMr1HbyqYxw] update_mapping [indices_stats] [2017-04-02T00:00:03,853][INFO][o.e.c.m.MetaDataMappingService] [1WEhN6j] [.monitoring-es-2-2017.04.02/lZ9x2xIWTRywMr1HbyqYxw] update_mapping [index_stats] [2017-04-02T00:00:03,916][INFO][o.e.c.m.MetaDataMappingService] [1WEhN6j] [.monitoring-es-2-2017.04.02/lZ9x2xIWTRywMr1HbyqYxw] update_mapping [node_stats] [2017-04-02T01:00:00,010][INFO][o.e.x.m.e.l.LocalExporter] cleaning up [1] old indices [2017-04-02T01:00:00,017][INFO][o.e.c.m.MetaDataDeleteIndexService] [1WEhN6j] [.monitoring-es-2-2017.03.26/u3UAMAUgRJuTN66fmhqZlw] deleting index [2017-04-02T07:38:24,250][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [1WEhN6j] collector [cluster-stats-collector] timed out when collecting data [2017-04-02T07:52:14,974][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [1WEhN6j] collector [cluster-stats-collector] timed out when collecting data ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 2, 2017, 6:01pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/8 "2017-04-02T18:01:09Z")
But does it stop the node? ---
**Author:** ![expert](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@expert](https://discuss.elastic.co/u/expert)\ **Post date:** [April 2, 2017, 6:12pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/9 "2017-04-02T18:12:49Z")
Yep. Unfortunately it does. I just tried to uninstall X-Pack. We'll see if it helps. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 30, 2017, 6:13pm UTC](https://discuss.elastic.co/t/es-5-2-2-crashes-daily-in-docker/80096/10 "2017-04-30T18:13:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.