Crash after upgrading to es 8.0.0

ilkkao · February 21, 2022, 8:43am

I was about to add a github issue but as I'm not sure what counts as a confirmed bug I report my issue here:

I was running es & kibana 7.17.0 and used kibana migration assistant to re-index my main index called "messages". Assistant was then all green. After that I upgraded to es 8.0.0 and got the following crash:

java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/bq-_Zh3KS0-Q_ORWnVekDA]]
Likely root cause: java.lang.IllegalArgumentException: Unknown pattern letter: t
	at java.base/java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1882)
	at java.base/java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1772)
	at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2135)
	at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116)
	at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:277)
	at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:331)
	at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:220)
	at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:100)
	at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:101)
	at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:149)
	at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94)

Here are the logs the server manages to print before crashing:

 $ kubectl -n myproject logs elasticsearch-59588dcd4b-68ml8 | jq ".message"
"version[8.0.0], pid[7], build[default/docker/1b6a7ece17463df5ff54a3e1302d825889aa1161/2022-02-03T16:47:57.507843096Z], OS[Linux/5.4.0-91-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]"
"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]"
"JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-358308377955594670, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]"
"SLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\"."
"SLF4J: Defaulting to no-operation (NOP) logger implementation"
"SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details."
"loaded module [aggs-matrix-stats]"
"loaded module [analysis-common]"
"loaded module [constant-keyword]"
"loaded module [frozen-indices]"
"loaded module [ingest-common]"
"loaded module [ingest-geoip]"
"loaded module [ingest-user-agent]"
"loaded module [kibana]"
"loaded module [lang-expression]"
"loaded module [lang-mustache]"
"loaded module [lang-painless]"
"loaded module [legacy-geo]"
"loaded module [mapper-extras]"
"loaded module [mapper-version]"
"loaded module [parent-join]"
"loaded module [percolator]"
"loaded module [rank-eval]"
"loaded module [reindex]"
"loaded module [repositories-metering-api]"
"loaded module [repository-azure]"
"loaded module [repository-encrypted]"
"loaded module [repository-gcs]"
"loaded module [repository-s3]"
"loaded module [repository-url]"
"loaded module [runtime-fields-common]"
"loaded module [search-business-rules]"
"loaded module [searchable-snapshots]"
"loaded module [snapshot-based-recoveries]"
"loaded module [snapshot-repo-test-kit]"
"loaded module [spatial]"
"loaded module [transform]"
"loaded module [transport-netty4]"
"loaded module [unsigned-long]"
"loaded module [vector-tile]"
"loaded module [vectors]"
"loaded module [wildcard]"
"loaded module [x-pack-aggregate-metric]"
"loaded module [x-pack-analytics]"
"loaded module [x-pack-async]"
"loaded module [x-pack-async-search]"
"loaded module [x-pack-autoscaling]"
"loaded module [x-pack-ccr]"
"loaded module [x-pack-core]"
"loaded module [x-pack-data-streams]"
"loaded module [x-pack-deprecation]"
"loaded module [x-pack-enrich]"
"loaded module [x-pack-eql]"
"loaded module [x-pack-fleet]"
"loaded module [x-pack-graph]"
"loaded module [x-pack-identity-provider]"
"loaded module [x-pack-ilm]"
"loaded module [x-pack-logstash]"
"loaded module [x-pack-ml]"
"loaded module [x-pack-monitoring]"
"loaded module [x-pack-ql]"
"loaded module [x-pack-rollup]"
"loaded module [x-pack-security]"
"loaded module [x-pack-shutdown]"
"loaded module [x-pack-sql]"
"loaded module [x-pack-stack]"
"loaded module [x-pack-text-structure]"
"loaded module [x-pack-voting-only-node]"
"loaded module [x-pack-watcher]"
"no plugins loaded"
"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/md2)]], net usable_space [200.4gb], net total_space [461gb], types [ext4]"
"heap size [31gb], compressed ordinary object pointers [true]"
"node name [elasticsearch-59588dcd4b-68ml8], node ID [jSFuEwKUR9q65j4Ta4E-cQ], cluster name [docker-cluster], roles [transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot]"
"Security is disabled"
"creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]"
"using discovery type [single-node] and seed hosts providers [settings]"
"initialized"
"starting ..."
"persistent cache index loaded"
"deprecation component started"
"publish_address {10.1.51.217:9300}, bound_addresses {[::]:9300}"
"Skipping unknown custom object with type system_index_migration"
"uncaught exception in thread [main]"

Has anyone else encountered this? Let me know how to provide more info if that's needed. I lost few days of data because of this. Not a big deal but a solution that can repair the index would be nice.

ilkkao · February 21, 2022, 8:59am

In case it matters, I saw this error while upgrading [Upgrade] migration system indices fails for tasks: mapper_parsing_exception · Issue #83779 · elastic/elasticsearch · GitHub After seeing it, I just deleted the task index and proceeded.

ilkkao · February 21, 2022, 3:43pm

I couldn't simply downgrade to es7 as it says that the db files have been touched by es8. From the backups I got working es7 setup. Now I can see the following warning

"Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead."

This probably caused it? And my mistake was not to check the logs, just rely on upgrade assistant?

DavidTurner · February 21, 2022, 6:20pm

That does seem like the explanation, but I think Elasticsearch should at least not leave you in a state where you can't start it up but nor can you downgrade either. Could you share the complete log output please?

ilkkao · February 21, 2022, 6:47pm

Here is a log when I try to downgrade:

$ kubectl -n mas logs elasticsearch-75d4bf46bb-h6xcp
{"type": "server", "timestamp": "2022-02-21T18:45:46,138Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "version[7.17.0], pid[7], build[default/docker/bee86328705acaa9a6daede7140defd4d9ec56bd/2022-01-28T08:36:04.875279988Z], OS[Linux/5.4.0-91-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]" }
{"type": "server", "timestamp": "2022-02-21T18:45:46,140Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2022-02-21T18:45:46,141Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-5817322145606499066, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [legacy-geo]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [vector-tile]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,484Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,484Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2022-02-21T18:45:47,495Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "elasticsearch-75d4bf46bb-h6xcp", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.0, perhaps it has already been upgraded to a later version",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.17.0.jar:7.17.0]",

ilkkao · February 21, 2022, 6:51pm

And when I try to use 8.0.0

kubectl -n mas logs elasticsearch-84b66fb459-gjpfc
{"@timestamp":"2022-02-21T18:48:43.416Z", "log.level": "INFO", "message":"version[8.0.0], pid[7], build[default/docker/1b6a7ece17463df5ff54a3e1302d825889aa1161/2022-02-03T16:47:57.507843096Z], OS[Linux/5.4.0-91-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:43.418Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:43.418Z", "log.level": "INFO", "message":"JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-18423929716915370894, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:44.369Z", "log.level": "WARN", "message":"SLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\".", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:44.370Z", "log.level": "WARN", "message":"SLF4J: Defaulting to no-operation (NOP) logger implementation", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:44.370Z", "log.level": "WARN", "message":"SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.", "ecs.version": 
[some module loaded lines deleted, this is standard docker image]
"1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.019Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.019Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-encrypted]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.020Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.021Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [vectors]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.022Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.023Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.024Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.025Z", "log.level": "INFO", "message":"no plugins loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.045Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/md2)]], net usable_space [188.9gb], net total_space [461gb], types [ext4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.045Z", "log.level": "INFO", "message":"heap size [31gb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:45.097Z", "log.level": "INFO", "message":"node name [elasticsearch-84b66fb459-gjpfc], node ID [jSFuEwKUR9q65j4Ta4E-cQ], cluster name [docker-cluster], roles [data_content, transform, data_hot, ml, data_frozen, ingest, data_cold, data, remote_cluster_client, master, data_warm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:48.398Z", "log.level": "INFO", "message":"Security is disabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:48.728Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:48.763Z", "log.level": "INFO", "message":"using discovery type [single-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.150Z", "log.level": "INFO", "message":"initialized", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.150Z", "log.level": "INFO", "message":"starting ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.161Z", "log.level": "INFO", "message":"persistent cache index loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.161Z", "log.level": "INFO", "message":"deprecation component started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.212Z", "log.level": "INFO", "message":"publish_address {10.1.51.245:9300}, bound_addresses {[::]:9300}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.TransportService","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.286Z", "log.level": "WARN", "message":"Skipping unknown custom object with type system_index_migration", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.metadata.Metadata","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.364Z", "log.level":"ERROR", "message":"uncaught exception in thread [main]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.bootstrap.StartupException","error.message":"java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/x7yO_28nSVaRv08z3qnr1A]]","error.stack_trace":"java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/x7yO_28nSVaRv08z3qnr1A]]\nLikely root cause: java.lang.IllegalArgumentException: Unknown pattern letter: t\n\tat java.base/java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1882)\n\tat java.base/java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1772)\n\tat org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2135)\n\tat org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116)\n\tat org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:277)\n\tat org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:331)\n\tat org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:220)\n\tat org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:100)\n\tat org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:101)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:149)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94)\n\tat org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:359)\n\tat org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:338)\n\tat org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:324)\n\tat org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:195)\n\tat org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:91)\n\tat org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:226)\n\tat org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:213)\n\tat org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:127)\n\tat org.elasticsearch.node.Node.start(Node.java:1124)\n\tat org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:272)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:367)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)\n\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)\n\tat org.elasticsearch.cli.Command.main(Command.java:77)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)\nFor complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log\n"}
uncaught exception in thread [main]
java.lang.IllegalStateException: Failed to parse mappings for index [[reindexed-v7-messages1-2.4.0/x7yO_28nSVaRv08z3qnr1A]]
Likely root cause: java.lang.IllegalArgumentException: Unknown pattern letter: t
	at java.base/java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1882)
	at java.base/java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1772)
	at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2135)
	at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116)
	at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:277)
	at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:331)
	at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:220)
	at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:100)
	at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:101)
	at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:149)
	at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94)
	at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:359)
	at org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:338)
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:324)
	at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:195)
	at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:91)
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:226)
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:213)
	at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:127)
	at org.elasticsearch.node.Node.start(Node.java:1124)
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:272)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:367)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log
{"@timestamp":"2022-02-21T18:48:49.366Z", "log.level": "INFO", "message":"stopping ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.392Z", "log.level": "INFO", "message":"stopped", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.392Z", "log.level": "INFO", "message":"closing ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-02-21T18:48:49.397Z", "log.level": "INFO", "message":"closed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"Thread-2","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elasticsearch-84b66fb459-gjpfc","elasticsearch.cluster.name":"docker-cluster"}

DavidTurner · February 21, 2022, 7:47pm

Thanks, I opened a bug report at Deprecated feature use in 7.x index fails upgrade to 8.x but cannot be downgraded. · Issue #84199 · elastic/elasticsearch · GitHub.

system · March 21, 2022, 7:48pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch won't start after upgrade to 8.1.0 Elasticsearch	6	1280	April 12, 2022
Mapper parsing exception - Am I looking in the right place or is ES fooling me!? Elasticsearch docker	5	411	July 25, 2022
Mapper Parsing Exception after Upgrade to 5.1.1 Elasticsearch	8	2700	March 6, 2017
Elasticsearch won’t start after upgrade to 8.4.1 Elasticsearch	5	1069	October 15, 2022
MapperParsingException after migrating Java client from ES 2.3.3 to 5.5.0 Elasticsearch	9	816	August 9, 2017

Crash after upgrading to es 8.0.0

Related topics