My Elasticsearch wont start after java update

Elastic Stack Elasticsearch
1

getting the below error and not sure where to start... can anyone assist?

[root@bh3elklpv01 jli]# sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─elasticsearch.conf
Active: failed (Result: exit-code) since Thu 2022-09-29 08:55:25 EDT; 3min 32s ago
Docs: https://www.elastic.co
Process: 1028 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 1028 (code=exited, status=1/FAILURE)

Sep 29 08:54:43 bh3elklpv01.prcins.net systemd[1]: Starting Elasticsearch...
Sep 29 08:55:23 bh3elklpv01.prcins.net systemd-entrypoint[1028]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/ossec.log
Sep 29 08:55:25 bh3elklpv01.prcins.net systemd-entrypoint[1028]: ERROR: Elasticsearch exited unexpectedly
Sep 29 08:55:25 bh3elklpv01.prcins.net systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Sep 29 08:55:25 bh3elklpv01.prcins.net systemd[1]: Failed to start Elasticsearch.
Sep 29 08:55:25 bh3elklpv01.prcins.net systemd[1]: Unit elasticsearch.service entered failed state.
Sep 29 08:55:25 bh3elklpv01.prcins.net systemd[1]: elasticsearch.service failed.

2

You will want to look at what's in /var/log/elasticsearch/ossec.log.

3

[2022-09-29T08:32:55,747][INFO ][o.e.i.r.RecoverySettings ] [ossec_node1] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-09-29T08:32:55,785][INFO ][o.e.d.DiscoveryModule ] [ossec_node1] using discovery type [multi-node] and seed hosts providers [settings]
[2022-09-29T08:32:56,870][INFO ][o.e.n.Node ] [ossec_node1] initialized
[2022-09-29T08:32:56,871][INFO ][o.e.n.Node ] [ossec_node1] starting ...
[2022-09-29T08:32:56,963][INFO ][o.e.x.s.c.f.PersistentCache] [ossec_node1] persistent cache index loaded
[2022-09-29T08:32:56,964][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [ossec_node1] deprecation component started
[2022-09-29T08:32:57,087][INFO ][o.e.t.TransportService ] [ossec_node1] publish_address {172.21.1.93:9300}, bound_addresses {172.21.1.93:9300}
[2022-09-29T08:32:57,919][ERROR][o.e.b.Elasticsearch ] [ossec_node1] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: Failed to parse mappings for index [[wazuh-monitoring-2022.27w/3ay_VAqNR4yo_AQDEFC8iQ]]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:201) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:93) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:225) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:212) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:126) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.node.Node.start(Node.java:1247) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:436) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:229) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.1.jar:?]
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: Error parsing [format] on field [timestamp]: Invalid format: [dateOptionalTime]: Unknown pattern letter: t
at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:372) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]
... 8 more
Caused by: java.lang.IllegalArgumentException: Error parsing [format] on field [timestamp]: Invalid format: [dateOptionalTime]: Unknown pattern letter: t
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:292) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:346) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:222) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:149) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]
... 8 more
Caused by: java.lang.IllegalArgumentException: Invalid format: [dateOptionalTime]: Unknown pattern letter: t
at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2129) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:286) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:346) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:222) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:149) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]
... 8 more
Caused by: java.lang.IllegalArgumentException: Unknown pattern letter: t
at java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1891) ~[?:?]
at java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1781) ~[?:?]
at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2126) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:286) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:346) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:222) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:149) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]

4

I posted the part that seems to be the issue as log was too large to paste in a reply.

here is java version
[root@bh3elklpv01 jdk]# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

5

Do you see the same errors in the log I notice ?
Caused by: java.lang.IllegalArgumentException: Unknown pattern letter: t
at java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1891) ~[?:?]
I also see this : java.lang.IllegalStateException: Failed to parse mappings for index [[wazuh-monitoring-2022.27w/3ay_VAqNR4yo_AQDEFC8iQ]]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:201) ~[elasticsearch-8.4.1.jar:?]
at

Maybe you want to look at how that wazuh-monitoring-2022.27w index is set up ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.