Elasticsearch 2.4.6 starts with warnings

Elastic Stack Elasticsearch
1

Hello, I am having a problem with the elasticsearch service (2.4.6) that I am running on a centos server, currently the service goes down every so often (1-2 days) and when I start the service it gives me this warning, I have also I have been reviewing the logs but there are thousands of lines and I mostly find problems regarding the queries but I don't think it should be taken for this case, could you tell me how I could solve this incident

note: the application that uses elasticsearch is on symfony 3.2 (I am maintaining this system)

[root@b9107e89 elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-04-03 13:14:40 UTC; 1h 7min ago
     Docs: http://www.elastic.co
  Process: 6067 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 6069 (java)
   CGroup: /system.slice/elasticsearch.service
           └─6069 java -Xms8g -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOcc...

Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
Apr 03 13:37:11 b9107e89.online-server.cloud elasticsearch[6069]: at java.lang.Thread.run(Thread.java:748)
2

I think you'd need to look for the actual logged error in its entirety.

With that said, Elasticsearch 2.4 has been EOL for several years - I would strongly suggest you consider upgrading to a newer version of Elasticsearch.

3

This would be a small fragment of the 2000 lines that elastic generated, but they refer to a query that has failed

[2024-04-03 03:17:10,476][DEBUG][action.search            ] [master] [vivados][0], node[mjqanen9R9mxE2vudVggwg], [P], v[48], s[STARTED], a[id=JCkMlh4dRCGUbrwtcYwSbQ]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7e39366e] lastShard [true]
RemoteTransportException[[master][82.223.35.218:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"query":{"bool":{"must":[{"regexp":{"referencia":{"value":".*409-409-Terreno junto a las Tas (A.*","boost":1}}},{"match":{"software":"inmobigrama"}},{"query_string":{"query":"1","default_field":"estado"}},{"nested":{"path":"agency","query":{"query_string":{"query":"5c9919dc2045815cff4ff803","default_field":"agency.id"}}}},{"nested":{"path":"data","query":{"bool":{}}}}],"minimum_number_should_match":1}},"sort":[{"createdat":{"order":"desc"}}]}]]; nested: IllegalArgumentException[expected ')' at position 38];
Caused by: SearchParseException[failed to parse search source [{"query":{"bool":{"must":[{"regexp":{"referencia":{"value":".*409-409-Terreno junto a las Tas (A.*","boost":1}}},{"match":{"software":"inmobigrama"}},{"query_string":{"query":"1","default_field":"estado"}},{"nested":{"path":"agency","query":{"query_string":{"query":"5c9919dc2045815cff4ff803","default_field":"agency.id"}}}},{"nested":{"path":"data","query":{"bool":{}}}}],"minimum_number_should_match":1}},"sort":[{"createdat":{"order":"desc"}}]}]]; nested: IllegalArgumentException[expected ')' at position 38];
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:873)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:667)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: expected ')' at position 38
        at org.apache.lucene.util.automaton.RegExp.parseSimpleExp(RegExp.java:1120)
        at org.apache.lucene.util.automaton.RegExp.parseCharClassExp(RegExp.java:1090)
        at org.apache.lucene.util.automaton.RegExp.parseComplExp(RegExp.java:1078)
        at org.apache.lucene.util.automaton.RegExp.parseRepeatExp(RegExp.java:1047)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1040)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseInterExp(RegExp.java:1033)
        at org.apache.lucene.util.automaton.RegExp.parseUnionExp(RegExp.java:1027)
        at org.apache.lucene.util.automaton.RegExp.<init>(RegExp.java:405)
        at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:108)
        at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:93)
        at org.elasticsearch.index.mapper.MappedFieldType.regexpQuery(MappedFieldType.java:516)
        at org.elasticsearch.index.query.RegexpQueryParser.parse(RegexpQueryParser.java:120)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:250)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:114)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:250)
        at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:324)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:224)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:219)
        at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:856)
        ... 12 more
[2024-04-03 03:17:10,476][DEBUG][action.search            ] [master] [vivados][4], node[mjqanen9R9mxE2vudVggwg], [P], v[48], s[STARTED], a[id=veRbm28uS8CpxSYiblNj6w]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7e39366e]
RemoteTransportException[[master][82.223.35.218:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"query":{"bool":{"must":[{"regexp":{"referencia":{"value":".*409-409-Terreno junto a las Tas (A.*","boost":1}}},{"match":{"software":"inmobigrama"}},{"query_string":{"query":"1","default_field":"estado"}},{"nested":{"path":"agency","query":{"query_string":{"query":"5c9919dc2045815cff4ff803","default_field":"agency.id"}}}},{"nested":{"path":"data","query":{"bool":{}}}}],"minimum_number_should_match":1}},"sort":[{"createdat":{"order":"desc"}}]}]]; nested: IllegalArgumentException[expected ')' at position 38];
Caused by: SearchParseException[failed to parse search source [{"query":{"bool":{"must":[{"regexp":{"referencia":{"value":".*409-409-Terreno junto a las Tas (A.*","boost":1}}},{"match":{"software":"inmobigrama"}},{"query_string":{"query":"1","default_field":"estado"}},{"nested":{"path":"agency","query":{"query_string":{"query":"5c9919dc2045815cff4ff803","default_field":"agency.id"}}}},{"nested":{"path":"data","query":{"bool":{}}}}],"minimum_number_should_match":1}},"sort":[{"createdat":{"order":"desc"}}]}]]; nested: IllegalArgumentException[expected ')' at position 38];
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:873)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:667)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: expected ')' at position 38
        at org.apache.lucene.util.automaton.RegExp.parseSimpleExp(RegExp.java:1120)
        at org.apache.lucene.util.automaton.RegExp.parseCharClassExp(RegExp.java:1090)
        at org.apache.lucene.util.automaton.RegExp.parseComplExp(RegExp.java:1078)
        at org.apache.lucene.util.automaton.RegExp.parseRepeatExp(RegExp.java:1047)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1040)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseConcatExp(RegExp.java:1041)
        at org.apache.lucene.util.automaton.RegExp.parseInterExp(RegExp.java:1033)
        at org.apache.lucene.util.automaton.RegExp.parseUnionExp(RegExp.java:1027)
        at org.apache.lucene.util.automaton.RegExp.<init>(RegExp.java:405)
        at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:108)
        at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:93)
        at org.elasticsearch.index.mapper.MappedFieldType.regexpQuery(MappedFieldType.java:516)
        at org.elasticsearch.index.query.RegexpQueryParser.parse(RegexpQueryParser.java:120)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:250)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:114)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:250)
        at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:324)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:224)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:219)
        at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:856)
        ... 12 more
4

This is not a correct regular expression:

".*409-409-Terreno junto a las Tas (A.*"

You need to escape the parenthesis I think.

But it does not seem related to the initial problem.

5

I am checking that the logs are created:
elasticsearch.log
elasticsearch.log.yyyy-mm-dd

In the first log fragment I passed it corresponds to elasticsearch.log but now reviewing the last lines of elasticsearch.log.yyyy-mm-dd I find this, since I only have one node I had defined the number of replicas to 0, maybe Why can't the shard error be one of those states mentioned and the memory collapse?

[2024-04-02 15:40:51,510][INFO ][cluster.metadata         ] [master] updating number_of_replicas to [0] for indices [vivados]
[2024-04-02 15:48:25,413][INFO ][node                     ] [master] stopping ...
[2024-04-02 15:48:26,643][INFO ][node                     ] [master] stopped
[2024-04-02 15:48:26,643][INFO ][node                     ] [master] closing ...
[2024-04-02 15:48:26,647][INFO ][node                     ] [master] closed
[2024-04-02 15:48:27,253][INFO ][node                     ] [master] version[2.4.6], pid[29401], build[5376dca/2017-07-18T12:17:44Z]
[2024-04-02 15:48:27,254][INFO ][node                     ] [master] initializing ...
[2024-04-02 15:48:27,569][INFO ][plugins                  ] [master] modules [lang-groovy, reindex, lang-expression], plugins [kopf], sites [kopf]
[2024-04-02 15:48:27,597][INFO ][env                      ] [master] using [1] data paths, mounts [[/var (/dev/mapper/vg00-var)]], net usable_space [103.5gb], net total_space [206.6gb], spins? [possibly], types [ext4]
[2024-04-02 15:48:27,597][INFO ][env                      ] [master] heap size [7.9gb], compressed ordinary object pointers [true]
[2024-04-02 15:48:28,546][INFO ][node                     ] [master] initialized
[2024-04-02 15:48:28,546][INFO ][node                     ] [master] starting ...
[2024-04-02 15:48:28,624][INFO ][transport                ] [master] publish_address {82.223.35.218:9300}, bound_addresses {82.223.35.218:9300}
[2024-04-02 15:48:28,627][INFO ][discovery                ] [master] vivados/mjqanen9R9mxE2vudVggwg
[2024-04-02 15:48:31,642][INFO ][cluster.service          ] [master] new_master {master}{mjqanen9R9mxE2vudVggwg}{82.223.35.218}{82.223.35.218:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2024-04-02 15:48:31,657][INFO ][http                     ] [master] publish_address {82.223.35.218:9200}, bound_addresses {82.223.35.218:9200}
[2024-04-02 15:48:31,657][INFO ][node                     ] [master] started
[2024-04-02 15:48:31,705][INFO ][gateway                  ] [master] recovered [1] indices into cluster_state
[2024-04-02 15:48:31,724][DEBUG][action.search            ] [master] All shards failed for phase: [query]
RemoteTransportException[[master][82.223.35.218:9300][indices:data/read/search[phase/query]]]; nested: IndexNotFoundException[no such index];
Caused by: [vivados] IndexNotFoundException[no such index]
        at org.elasticsearch.indices.IndicesService.indexServiceSafe(IndicesService.java:310)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:651)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
[2024-04-02 15:48:31,727][WARN ][rest.suppressed          ] path: /vivados/city,distrito,barrio/_search, params: {index=vivados, type=city,distrito,barrio}
Failed to execute phase [query], all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:874)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:852)
        at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:389)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
[2024-04-02 15:48:32,144][DEBUG][action.search            ] [master] All shards failed for phase: [query]
RemoteTransportException[[master][82.223.35.218:9300][indices:data/read/search[phase/query]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
Caused by: [vivados][[vivados][4]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
        at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1038)
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:852)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:656)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
[2024-04-02 15:48:32,144][WARN ][rest.suppressed          ] path: /vivados/city,distrito,barrio/_search, params: {index=vivados, type=city,distrito,barrio}
Failed to execute phase [query], all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:874)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:852)
        at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:389)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
[2024-04-02 15:48:32,175][DEBUG][action.search            ] [master] All shards failed for phase: [query]
RemoteTransportException[[master][82.223.35.218:9300][indices:data/read/search[phase/query]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
Caused by: [vivados][[vivados][4]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
        at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1038)
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:852)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:656)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
[2024-04-02 15:48:32,175][WARN ][rest.suppressed          ] path: /vivados/city,distrito,barrio/_search, params: {index=vivados, type=city,distrito,barrio}
Failed to execute phase [query], all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:874)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:852)
        at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:389)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)
[2024-04-02 15:48:32,543][INFO ][cluster.routing.allocation] [master] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[vivados][1]] ...]).
6

What is the full output of the cluster stats API?

7 
{
  "timestamp" : 1712162450361,
  "cluster_name" : "vivados",
  "status" : "yellow",
  "indices" : {
    "count" : 1,
    "shards" : {
      "total" : 5,
      "primaries" : 5,
      "replication" : 0.0,
      "index" : {
        "shards" : {
          "min" : 5,
          "max" : 5,
          "avg" : 5.0
        },
        "primaries" : {
          "min" : 5,
          "max" : 5,
          "avg" : 5.0
        },
        "replication" : {
          "min" : 0.0,
          "max" : 0.0,
          "avg" : 0.0
        }
      }
    },
    "docs" : {
      "count" : 534592,
      "deleted" : 811
    },
    "store" : {
      "size" : "723.6mb",
      "size_in_bytes" : 758760248,
      "throttle_time" : "0s",
      "throttle_time_in_millis" : 0
    },
    "fielddata" : {
      "memory_size" : "0b",
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "29.1kb",
      "memory_size_in_bytes" : 29832,
      "total_count" : 912,
      "hit_count" : 714,
      "miss_count" : 198,
      "cache_size" : 5,
      "cache_count" : 5,
      "evictions" : 0
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 77,
      "memory" : "5.1mb",
      "memory_in_bytes" : 5427229,
      "terms_memory" : "4.6mb",
      "terms_memory_in_bytes" : 4838257,
      "stored_fields_memory" : "210.8kb",
      "stored_fields_memory_in_bytes" : 215896,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "155.2kb",
      "norms_memory_in_bytes" : 158976,
      "doc_values_memory" : "209kb",
      "doc_values_memory_in_bytes" : 214100,
      "index_writer_memory" : "0b",
      "index_writer_memory_in_bytes" : 0,
      "index_writer_max_memory" : "810.8mb",
      "index_writer_max_memory_in_bytes" : 850270615,
      "version_map_memory" : "0b",
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set" : "0b",
      "fixed_bit_set_memory_in_bytes" : 0
    },
    "percolate" : {
      "total" : 0,
      "time" : "0s",
      "time_in_millis" : 0,
      "current" : 0,
      "memory_size_in_bytes" : -1,
      "memory_size" : "-1b",
      "queries" : 0
    }
  },
  "nodes" : {
    "count" : {
      "total" : 1,
      "master_only" : 0,
      "data_only" : 0,
      "master_data" : 1,
      "client" : 0
    },
    "versions" : [ "2.4.6" ],
    "os" : {
      "available_processors" : 12,
      "allocated_processors" : 12,
      "mem" : {
        "total" : "14.8gb",
        "total_in_bytes" : 15965536256
      },
      "names" : [ {
        "name" : "Linux",
        "count" : 1
      } ]
    },
    "process" : {
      "cpu" : {
        "percent" : 0
      },
      "open_file_descriptors" : {
        "min" : 413,
        "max" : 413,
        "avg" : 413
      }
    },
    "jvm" : {
      "max_uptime" : "3.6m",
      "max_uptime_in_millis" : 220489,
      "versions" : [ {
        "version" : "1.7.0_261",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "24.261-b02",
        "vm_vendor" : "Oracle Corporation",
        "count" : 1
      } ],
      "mem" : {
        "heap_used" : "464.2mb",
        "heap_used_in_bytes" : 486809184,
        "heap_max" : "7.9gb",
        "heap_max_in_bytes" : 8502706176
      },
      "threads" : 127
    },
    "fs" : {
      "total" : "206.6gb",
      "total_in_bytes" : 221879975936,
      "free" : "112.7gb",
      "free_in_bytes" : 121107435520,
      "available" : "104.2gb",
      "available_in_bytes" : 111942590464,
      "spins" : "true"
    },
    "plugins" : [ {
      "name" : "kopf",
      "version" : "2.0.1",
      "description" : "kopf - simple web administration tool for Elasticsearch",
      "url" : "/_plugin/kopf/",
      "jvm" : false,
      "site" : true
    } ]
  }
}
8

Right now I restarted elastic and the number of replicas was set to 1, but since I am on a single node there are unassigned shards and the cluster is yellow

9

That looks fine. I can't see anything there that would make the node fall over the way you described.

I have not used Elasticsearch 2.4 in a long, long time so do not tink I will be able to help much.

10

It looks like this error is due to a particular query that's being sent. I would be surprised if something like that made the node go red - but I can't rule it out entirely as 2.4 is so old and out of date and a lot of bugs have been fixed/changes have been made since that version.

Good luck troubleshooting, and I'll echo my suggestion to upgrade. Since 2.4 is so out of date it's a heavy lift, but it'll be much easier to help with this type of issue if you're running on supported versions.

2 Likes
11

The entire environment in which the application runs is very old :,) I don't think they want to pay to migrate all that, anyway, thank you very much for your answers

12

That's fair. You can start by diagnosing what's creating the bad query and see if that's actually causing the cluster to topple over. I suspect there might be something else going on, but it could be something weird happening. Good luck!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.