Elasticsearch 7.17.4 crashes without an error message, while indexing large chunks of data on Windows

simon137 · September 19, 2022, 11:48am

Hello,

I tried multiple java heap configurations for both logstash and elasticsearch, and also using a persisted queue (with space up to 40gb) instead of a memory queue, but it's always the same:

If I take a large chunk of data (8 parallel input pipelines from a Microsoft SQL-Server) and throw it into one index, Elasticsearch crashes suddenly, always at different times and with no error message whatsoever. Everything that's logged right before the crash is this:

[2022-09-19T13:18:55,523][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch] [gc][886] overhead, spent [358ms] collecting in the last [1.1s]
[2022-09-19T13:19:17,044][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [elasticsearch] attempting to trigger G1GC due to high heap usage [2076390392]
[2022-09-19T13:19:17,113][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [elasticsearch] GC did bring memory usage down, before [2076390392], after [1878888952], allocations [5], duration [69]
[2022-09-19T13:20:07,026][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch] [gc][955] overhead, spent [508ms] collecting in the last [1.1s]

I tried my virtual windows server with many different memory configurations, the last one, where the above log is from, having 32gb of RAM in total and 8 processors and ONLY running elasticsearch, logstash and kibana.

Is this maybe a known issue in 7.17.4 or something like that? Or have you any other idea, where that error might be coming from?

Thanks in advance for your help.
Simon

leandrojmp · September 19, 2022, 12:33pm

What are the configuration for each service? What is the configured Java Heap for both elasticsearch and Logstash?

If these are the last lines in the log, then the elasticsearch process is being killed by the system. How are you running it? As a service?

You need to check further in the system logs, probably in the event viewer, to see if you can get more information.

simon137 · September 19, 2022, 1:12pm

For logstash it's 2GB of heap for elasticsearch i usually don't configure it, to let it determine the heap it needs dynamically at startup, but like I said I also tried various configurations from 1GB up to 16GB of heap for elasticsearch, always resulting in the same behaviour, but only for the pipeline with 8 different inputs at the first indexing with the huge data amounts, the other smaller pipelines are working perfectly fine.

I run it as a service with nssm on windows as explained in the tutorial.
I also checked the eventlog, but there is literally no information on the reason of the shutdown of the service at all. The only info is in the "system"-event log and its only the standard from the source "Service Control Manager" saying that "the elasticsearch service is now in the state "terminated"" (translated from the original german message), The Event-ID is 7036

That's the XML-Event Data for the event:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" /> 
  <EventID Qualifiers="16384">7036</EventID> 
  <Version>0</Version> 
  <Level>4</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x8080000000000000</Keywords> 
  <TimeCreated SystemTime="2022-09-19T11:20:07.0266984Z" /> 
  <EventRecordID>151722</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="916" ThreadID="4360" /> 
  <Channel>System</Channel> 
  <Computer>NOT DISCLOSED</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data Name="param1">Elasticsearch 7.17.4 (elasticsearch-service-x64)</Data> 
  <Data Name="param2">Beendet</Data> 
  <Binary>65006C00610073007400690063007300650061007200630068002D0073006500720076006900630065002D007800360034002F0031000000</Binary> 
  </EventData>
  </Event>

warkolm · September 21, 2022, 1:37am

What does your full Elasticsearch log show?
What is your elasticsearch.yml?

simon137 · September 21, 2022, 9:56am

The elasticsearch.yml is:

cluster.name: "elasticsearch-cluster"
node.name: "elasticsearch"
path.data: "D:\\Elasticsearch_Index\\data"
path.logs: "D:\\Elasticsearch_Index\\logs"
network.host: 0.0.0.0
discovery.type: single-node
xpack.security.enabled: true

The full log from start to finish shows this:

[2022-09-19T13:03:06,528][INFO ][o.e.n.Node               ] [elasticsearch] version[7.17.4], pid[736], build[default/zip/79878662c54c886ae89206c685d9f1051a9d6411/2022-05-18T18:04:20.964345128Z], OS[Windows Server 2022/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]
[2022-09-19T13:03:06,544][INFO ][o.e.n.Node               ] [elasticsearch] JVM home [C:\Program Files (x86)\ElasticSearch\elasticsearch\jdk], using bundled JDK [true]
[2022-09-19T13:03:06,544][INFO ][o.e.n.Node               ] [elasticsearch] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=C:\Users\Test\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Delasticsearch, -Des.path.home=C:\Program Files (x86)\ElasticSearch\elasticsearch, -Des.path.conf=C:\Program Files (x86)\ElasticSearch\elasticsearch\config, -Des.distribution.flavor=default, -Des.distribution.type=zip, -Des.bundled_jdk=true, exit, abort, -Xms2047m, -Xmx2047m, -Xss1024k]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [aggs-matrix-stats]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [analysis-common]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [constant-keyword]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [frozen-indices]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-common]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-geoip]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-user-agent]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [kibana]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-expression]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-mustache]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-painless]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [legacy-geo]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [mapper-extras]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [mapper-version]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [parent-join]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [percolator]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [rank-eval]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [reindex]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repositories-metering-api]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repository-encrypted]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repository-url]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [runtime-fields-common]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [search-business-rules]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [searchable-snapshots]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [snapshot-repo-test-kit]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [spatial]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [transform]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [transport-netty4]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [unsigned-long]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [vector-tile]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [vectors]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [wildcard]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-aggregate-metric]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-analytics]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-async]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-async-search]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-autoscaling]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ccr]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-core]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-data-streams]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-deprecation]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-enrich]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-eql]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-fleet]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-graph]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-identity-provider]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ilm]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-logstash]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ml]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-monitoring]
[2022-09-19T13:03:09,934][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ql]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-rollup]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-security]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-shutdown]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-sql]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-stack]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-text-structure]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-voting-only-node]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-watcher]
[2022-09-19T13:03:09,947][INFO ][o.e.p.PluginsService     ] [elasticsearch] no plugins loaded
[2022-09-19T13:03:10,200][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch] using [1] data paths, mounts [[Daten (D:)]], net usable_space [73.4gb], net total_space [99.9gb], types [NTFS]
[2022-09-19T13:03:10,200][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch] heap size [2gb], compressed ordinary object pointers [true]
[2022-09-19T13:03:10,372][INFO ][o.e.n.Node               ] [elasticsearch] node name [elasticsearch], node ID [swKRfL_WSuurMnmCRTabdw], cluster name [elasticsearch-cluster], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-09-19T13:03:16,153][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elasticsearch] [controller/5840] [Main.cc@122] controller (64 bit): Version 7.17.4 (Build 57e42dc012e06b) Copyright (c) 2022 Elasticsearch BV
[2022-09-19T13:03:16,809][INFO ][o.e.x.s.a.Realms         ] [elasticsearch] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-09-19T13:03:16,825][INFO ][o.e.x.s.a.s.FileRolesStore] [elasticsearch] parsed [0] roles from file [C:\Program Files (x86)\ElasticSearch\elasticsearch\config\roles.yml]
[2022-09-19T13:03:17,559][INFO ][o.e.i.g.ConfigDatabases  ] [elasticsearch] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [C:\Program Files (x86)\ElasticSearch\elasticsearch\config\ingest-geoip] for changes
[2022-09-19T13:03:17,559][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-ASN.mmdb]
[2022-09-19T13:03:17,559][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-ASN.mmdb_COPYRIGHT.txt]
[2022-09-19T13:03:17,559][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt]
[2022-09-19T13:03:17,559][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-ASN.mmdb_LICENSE.txt]
[2022-09-19T13:03:17,559][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb_COPYRIGHT.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb_LICENSE.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb_README.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-Country.mmdb]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-Country.mmdb_COPYRIGHT.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] deleting stale file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-Country.mmdb_LICENSE.txt]
[2022-09-19T13:03:17,575][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] initialized database registry, using geoip-databases directory [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw]
[2022-09-19T13:03:18,350][INFO ][o.e.t.NettyAllocator     ] [elasticsearch] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-09-19T13:03:18,382][INFO ][o.e.i.r.RecoverySettings ] [elasticsearch] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-09-19T13:03:18,445][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch] using discovery type [single-node] and seed hosts providers [settings]
[2022-09-19T13:03:19,029][INFO ][o.e.g.DanglingIndicesState] [elasticsearch] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-09-19T13:03:19,949][INFO ][o.e.n.Node               ] [elasticsearch] initialized
[2022-09-19T13:03:19,949][INFO ][o.e.n.Node               ] [elasticsearch] starting ...
[2022-09-19T13:03:19,981][INFO ][o.e.x.s.c.f.PersistentCache] [elasticsearch] persistent cache index loaded
[2022-09-19T13:03:19,981][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [elasticsearch] deprecation component started
[2022-09-19T13:03:20,168][INFO ][o.e.t.TransportService   ] [elasticsearch] publish_address {10.1.1.61:9300}, bound_addresses {[::]:9300}
[2022-09-19T13:03:21,201][WARN ][o.e.b.BootstrapChecks    ] [elasticsearch] Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2022-09-19T13:03:21,201][INFO ][o.e.c.c.Coordinator      ] [elasticsearch] cluster UUID [tGmSkZxhRzWOARn6D1wErw]
[2022-09-19T13:03:21,391][INFO ][o.e.c.s.MasterService    ] [elasticsearch] elected-as-master ([1] nodes joined)[{elasticsearch}{swKRfL_WSuurMnmCRTabdw}{jusZrYVnQkyNr6OknBy1BA}{10.1.1.61}{10.1.1.61:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 30, version: 5603, delta: master node changed {previous [], current [{elasticsearch}{swKRfL_WSuurMnmCRTabdw}{jusZrYVnQkyNr6OknBy1BA}{10.1.1.61}{10.1.1.61:9300}{cdfhilmrstw}]}
[2022-09-19T13:03:21,550][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch] master node changed {previous [], current [{elasticsearch}{swKRfL_WSuurMnmCRTabdw}{jusZrYVnQkyNr6OknBy1BA}{10.1.1.61}{10.1.1.61:9300}{cdfhilmrstw}]}, term: 30, version: 5603, reason: Publication{term=30, version=5603}
[2022-09-19T13:03:21,630][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch] publish_address {10.1.1.61:9200}, bound_addresses {[::]:9200}
[2022-09-19T13:03:21,630][INFO ][o.e.n.Node               ] [elasticsearch] started
[2022-09-19T13:03:21,792][INFO ][o.e.c.s.ClusterSettings  ] [elasticsearch] updating [xpack.monitoring.elasticsearch.collection.enabled] from [true] to [false]
[2022-09-19T13:03:21,792][INFO ][o.e.c.s.ClusterSettings  ] [elasticsearch] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2022-09-19T13:03:22,260][INFO ][o.e.l.LicenseService     ] [elasticsearch] license [17597b51-bd1b-4a56-8b82-239000c41f08] mode [basic] - valid
[2022-09-19T13:03:22,260][INFO ][o.e.x.s.a.Realms         ] [elasticsearch] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-09-19T13:03:22,260][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [elasticsearch] Active license is now [BASIC]; Security is enabled
[2022-09-19T13:03:22,272][INFO ][o.e.g.GatewayService     ] [elasticsearch] recovered [46] indices into cluster_state
[2022-09-19T13:03:22,366][ERROR][o.e.x.s.a.e.ReservedRealm] [elasticsearch] failed to retrieve password hash for reserved user [kibana_system]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
	at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:147) ~[x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:605) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:231) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:109) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:200) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:148) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:84) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:171) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:165) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:143) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:104) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) [x-pack-security-7.17.4.jar:7.17.4]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:327) [elasticsearch-7.17.4.jar:7.17.4]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:393) [elasticsearch-7.17.4.jar:7.17.4]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:245) [elasticsearch-7.17.4.jar:7.17.4]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.17.4.jar:7.17.4]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.17.4.jar:7.17.4]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.17.4.jar:7.17.4]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35) [transport-netty4-client-7.17.4.jar:7.17.4]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19) [transport-netty4-client-7.17.4.jar:7.17.4]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.17.4.jar:7.17.4]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:03:22,381][INFO ][o.e.x.s.a.RealmsAuthenticator] [elasticsearch] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2022-09-19T13:03:22,910][INFO ][o.e.i.g.GeoIpDownloader  ] [elasticsearch] updating geoip databases
[2022-09-19T13:03:22,910][INFO ][o.e.i.g.GeoIpDownloader  ] [elasticsearch] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2022-09-19T13:03:23,692][INFO ][o.e.i.g.GeoIpDownloader  ] [elasticsearch] geoip database [GeoLite2-ASN.mmdb] is up to date, updated timestamp
[2022-09-19T13:03:24,008][INFO ][o.e.i.g.GeoIpDownloader  ] [elasticsearch] geoip database [GeoLite2-City.mmdb] is up to date, updated timestamp
[2022-09-19T13:03:24,087][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-Country.mmdb.tmp.gz]
[2022-09-19T13:03:24,087][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-19T13:03:24,097][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb.tmp.gz]
[2022-09-19T13:03:24,267][INFO ][o.e.i.g.GeoIpDownloader  ] [elasticsearch] geoip database [GeoLite2-Country.mmdb] is up to date, updated timestamp
[2022-09-19T13:03:24,432][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] successfully reloaded changed geoip database file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-Country.mmdb]
[2022-09-19T13:03:24,482][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] successfully reloaded changed geoip database file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-ASN.mmdb]
[2022-09-19T13:03:26,039][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] successfully reloaded changed geoip database file [C:\Users\Test\AppData\Local\Temp\elasticsearch\geoip-databases\swKRfL_WSuurMnmCRTabdw\GeoLite2-City.mmdb]
[2022-09-19T13:03:29,832][INFO ][o.e.c.r.a.AllocationService] [elasticsearch] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test_index_1][0]]]).
[2022-09-19T13:05:12,093][INFO ][o.e.c.m.MetadataCreateIndexService] [elasticsearch] [large_index_test] creating index, cause [auto(bulk api)], templates [test_template], shards [1]/[0]
[2022-09-19T13:05:13,691][INFO ][o.e.c.m.MetadataMappingService] [elasticsearch] [large_index_test/_TLSMt23RnWQPuvLgvRSXQ] update_mapping [_doc]
[2022-09-19T13:05:14,358][INFO ][o.e.c.m.MetadataMappingService] [elasticsearch] [large_index_test/_TLSMt23RnWQPuvLgvRSXQ] update_mapping [_doc]
[2022-09-19T13:12:49,174][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch] [gc][540] overhead, spent [269ms] collecting in the last [1s]
[2022-09-19T13:13:17,478][INFO ][o.e.i.IndexingMemoryController] [elasticsearch] now throttling indexing for shard [[large_index_test][0]]: segment writing can't keep up
[2022-09-19T13:13:21,599][INFO ][o.e.i.IndexingMemoryController] [elasticsearch] stop throttling indexing for shard [[large_index_test][0]]
[2022-09-19T13:13:22,116][INFO ][o.e.x.i.IndexLifecycleRunner] [elasticsearch] policy [metricbeat] for index [metricbeat-7.17.5-2022.08.23-000001] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1945]
[2022-09-19T13:18:55,523][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch] [gc][886] overhead, spent [358ms] collecting in the last [1.1s]
[2022-09-19T13:19:17,044][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [elasticsearch] attempting to trigger G1GC due to high heap usage [2076390392]
[2022-09-19T13:19:17,113][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [elasticsearch] GC did bring memory usage down, before [2076390392], after [1878888952], allocations [5], duration [69]
[2022-09-19T13:20:07,026][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch] [gc][955] overhead, spent [508ms] collecting in the last [1.1s]

The "large_index_test" is the index I was trying to index from scratch here.

warkolm · September 21, 2022, 10:05am

Unfortunately there's nothing in there that shows Elasticsearch has stopped or crashed, how are you confirming that it's crashed?

simon137 · September 21, 2022, 10:38am

I know, that's exactely my problem!
I can confirm that it crashes, because I see that it does. The windows service just terminates and as I wrote above the windows eventlog shows that the service has stopped, but not why.

simon137 · October 11, 2022, 7:38am

It seems the solution is simply to update both Elasticsearch and Logstash to Version 8.4.3.
With this newest version everything seems to work smoothly, although I still don't really know what the problem was in the first place.

system · November 8, 2022, 7:38am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.