Hey guys,
For some reason ES 5.2.2 crashes daily in docker for me. Here is how I launch it
docker run --name search -d --net=host -p 9200:9200 -p 9300:9300 -e "http.host=127.0.0.1" -e "transport.host=127.0.0.1" -v /jdata/elastic/data:/usr/share/elasticsearch/data -v /jdata/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:5.2.2
my config doesn't have much
transport.tcp.port: 9300
http.port: 9200
client.transport.ignore_cluster_name: false
client.transport.sniff: false
discovery.zen.minimum_master_nodes: 1
xpack.security.enabled: false
and here what I have in log before it dies.
[2017-03-26T19:08:07,160][INFO ][o.e.n.Node ] initializing ...
[2017-03-26T19:08:07,317][INFO ][o.e.e.NodeEnvironment ] [1WEhN6j] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdb)]], net usable_space [294.6gb], net total_space [492gb], spins? [possibly], types [ext3]
[2017-03-26T19:08:07,318][INFO ][o.e.e.NodeEnvironment ] [1WEhN6j] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-03-26T19:08:07,368][INFO ][o.e.n.Node ] node name [1WEhN6j] derived from node ID [1WEhN6juR--PB5kOwCTJQA]; set [node.name] to override
[2017-03-26T19:08:07,374][INFO ][o.e.n.Node ] version[5.2.2], pid[1], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/4.4.19-29.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
[2017-03-26T19:08:12,275][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [aggs-matrix-stats]
[2017-03-26T19:08:12,275][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [ingest-common]
[2017-03-26T19:08:12,276][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [lang-expression]
[2017-03-26T19:08:12,276][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [lang-groovy]
[2017-03-26T19:08:12,276][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [lang-mustache]
[2017-03-26T19:08:12,276][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [lang-painless]
[2017-03-26T19:08:12,277][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [percolator]
[2017-03-26T19:08:12,277][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [reindex]
[2017-03-26T19:08:12,277][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [transport-netty3]
[2017-03-26T19:08:12,278][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded module [transport-netty4]
[2017-03-26T19:08:12,280][INFO ][o.e.p.PluginsService ] [1WEhN6j] loaded plugin [x-pack]
[2017-03-26T19:08:13,382][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2017-03-26T19:08:21,064][INFO ][o.e.n.Node ] initialized
[2017-03-26T19:08:21,066][INFO ][o.e.n.Node ] [1WEhN6j] starting ...
[2017-03-26T19:08:21,976][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: a8:90:75:a9:3a:9a:65:f3
[2017-03-26T19:08:22,269][INFO ][o.e.t.TransportService ] [1WEhN6j] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-03-26T19:08:22,298][WARN ][o.e.b.BootstrapChecks ] [1WEhN6j] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2017-03-26T19:08:22,298][WARN ][o.e.b.BootstrapChecks ] [1WEhN6j] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-03-26T19:08:25,499][INFO ][o.e.c.s.ClusterService ] [1WEhN6j] new_master {1WEhN6j}{1WEhN6juR--PB5kOwCTJQA}{f3tBrOMKQYqTptKjSgwDgg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-03-26T19:08:25,585][INFO ][o.e.h.HttpServer ] [1WEhN6j] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2017-03-26T19:08:25,586][INFO ][o.e.n.Node ] [1WEhN6j] started
[2017-03-26T19:08:27,631][INFO ][o.e.l.LicenseService ] [1WEhN6j] license [fba9ada3-6300-45ee-98b5-bbf355d8cfe3] mode [trial] - valid
[2017-03-26T19:08:27,670][INFO ][o.e.g.GatewayService ] [1WEhN6j] recovered [6] indices into cluster_state
[2017-03-26T19:09:32,636][ERROR][o.e.x.m.AgentService ] [1WEhN6j] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.2.2.jar:5.2.2]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.2.2.jar:5.2.2]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:183) ~[x-pack-5.2.2.jar:5.2.2]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:196) [x-pack-5.2.2.jar:5.2.2]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92-internal]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:121) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:111) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
[2017-03-26T19:10:15,948][INFO ][o.e.c.r.a.AllocationService] [1WEhN6j] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-2-2017.03.26][0]] ...]).
I have ~76M small documents in my ES. Total size of DB files is ~35GB
What should I do ?