Enterprise Search failed to start

Elastic Search
1

Fresh install of Enterprise search after not touching if for several months. Different machine different cluster.

Java 11.

Very basic setup. Kibana and Elasticsearch info only. No other settings changed even set as HTTP just to get is started. Error message below is well helpless to me at least..

[2022-09-21T22:47:01.855+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0002727 seconds, Stopping threads took: 0.0000074 seconds
[2022-09-21T22:47:01.902+0000][135672][safepoint ] Application time: 0.0473156 seconds
[2022-09-21T22:47:01.902+0000][135672][safepoint ] Entering safepoint region: Deoptimize
[2022-09-21T22:47:01.902+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:01.902+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0002042 seconds, Stopping threads took: 0.0000100 seconds
[2022-09-21T22:47:01.979+0000][135672][safepoint ] Application time: 0.0762276 seconds
[2022-09-21T22:47:01.979+0000][135672][safepoint ] Entering safepoint region: RevokeBias
[2022-09-21T22:47:01.979+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:01.979+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0000919 seconds, Stopping threads took: 0.0000068 seconds
[2022-09-21T22:47:02.326+0000][135672][safepoint ] Application time: 0.3470732 seconds
[2022-09-21T22:47:02.326+0000][135672][safepoint ] Entering safepoint region: CollectForMetadataAllocation
[2022-09-21T22:47:02.326+0000][135672][gc,start ] GC(0) Pause Young (Concurrent Start) (Metadata GC Threshold)
[2022-09-21T22:47:02.327+0000][135672][gc,task ] GC(0) Using 8 workers of 8 for evacuation
[2022-09-21T22:47:02.327+0000][135672][gc,age ] GC(0) Desired survivor size 6815744 bytes, new threshold 15 (max threshold 15)
[2022-09-21T22:47:02.343+0000][135672][gc,age ] GC(0) Age table with threshold 15 (max threshold 15)
[2022-09-21T22:47:02.343+0000][135672][gc,age ] GC(0) - age 1: 5065304 bytes, 5065304 total
[2022-09-21T22:47:02.344+0000][135672][gc,phases ] GC(0) Pre Evacuate Collection Set: 0.1ms
[2022-09-21T22:47:02.344+0000][135672][gc,phases ] GC(0) Evacuate Collection Set: 15.6ms
[2022-09-21T22:47:02.344+0000][135672][gc,phases ] GC(0) Post Evacuate Collection Set: 1.0ms
[2022-09-21T22:47:02.344+0000][135672][gc,phases ] GC(0) Other: 1.2ms
[2022-09-21T22:47:02.344+0000][135672][gc,heap ] GC(0) Eden regions: 54->0(97)
[2022-09-21T22:47:02.344+0000][135672][gc,heap ] GC(0) Survivor regions: 0->5(13)
[2022-09-21T22:47:02.344+0000][135672][gc,heap ] GC(0) Old regions: 2->2
[2022-09-21T22:47:02.344+0000][135672][gc,heap ] GC(0) Humongous regions: 1->1
[2022-09-21T22:47:02.344+0000][135672][gc,metaspace ] GC(0) Metaspace: 19350K->19350K(1067008K)
[2022-09-21T22:47:02.344+0000][135672][gc ] GC(0) Pause Young (Concurrent Start) (Metadata GC Threshold) 55M->6M(2048M) 17.964ms
[2022-09-21T22:47:02.344+0000][135672][gc,cpu ] GC(0) User=0.06s Sys=0.01s Real=0.02s
[2022-09-21T22:47:02.344+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:02.344+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0181751 seconds, Stopping threads took: 0.0000072 seconds
[2022-09-21T22:47:02.344+0000][135672][gc ] GC(1) Concurrent Cycle
[2022-09-21T22:47:02.344+0000][135672][gc,marking ] GC(1) Concurrent Clear Claimed Marks
[2022-09-21T22:47:02.344+0000][135672][gc,marking ] GC(1) Concurrent Clear Claimed Marks 0.032ms
[2022-09-21T22:47:02.344+0000][135672][gc,marking ] GC(1) Concurrent Scan Root Regions
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Application time: 0.0005155 seconds
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Entering safepoint region: RevokeBias
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0001223 seconds, Stopping threads took: 0.0000677 seconds
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Application time: 0.0006466 seconds
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Entering safepoint region: RevokeBias
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:02.345+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0000729 seconds, Stopping threads took: 0.0000149 seconds
[2022-09-21T22:47:02.348+0000][135672][gc,marking ] GC(1) Concurrent Scan Root Regions 3.659ms
[2022-09-21T22:47:02.348+0000][135672][gc,marking ] GC(1) Concurrent Mark (0.971s)
[2022-09-21T22:47:02.348+0000][135672][gc,marking ] GC(1) Concurrent Mark From Roots
[2022-09-21T22:47:02.348+0000][135672][gc,task ] GC(1) Using 2 workers of 2 for marking
[2022-09-21T22:47:02.350+0000][135672][gc,marking ] GC(1) Concurrent Mark From Roots 1.932ms
[2022-09-21T22:47:02.350+0000][135672][gc,marking ] GC(1) Concurrent Preclean
[2022-09-21T22:47:02.350+0000][135672][gc,marking ] GC(1) Concurrent Preclean 0.059ms
[2022-09-21T22:47:02.350+0000][135672][gc,marking ] GC(1) Concurrent Mark (0.971s, 0.973s) 2.023ms
[2022-09-21T22:47:02.350+0000][135672][safepoint ] Application time: 0.0044055 seconds
[2022-09-21T22:47:02.350+0000][135672][safepoint ] Entering safepoint region: CGC_Operation
[2022-09-21T22:47:02.350+0000][135672][gc,start ] GC(1) Pause Remark
[2022-09-21T22:47:02.364+0000][135672][gc,stringtable] GC(1) Cleaned string and symbol table, strings: 9249 processed, 0 removed, symbols: 42311 processed, 9 removed
[2022-09-21T22:47:02.365+0000][135672][gc ] GC(1) Pause Remark 7M->7M(2048M) 14.834ms
[2022-09-21T22:47:02.365+0000][135672][gc,cpu ] GC(1) User=0.07s Sys=0.00s Real=0.01s
[2022-09-21T22:47:02.365+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:02.365+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0149645 seconds, Stopping threads took: 0.0000294 seconds
[2022-09-21T22:47:02.365+0000][135672][gc,marking ] GC(1) Concurrent Rebuild Remembered Sets
[2022-09-21T22:47:02.366+0000][135672][gc,marking ] GC(1) Concurrent Rebuild Remembered Sets 0.742ms
[2022-09-21T22:47:02.366+0000][135672][safepoint ] Application time: 0.0007760 seconds
[2022-09-21T22:47:02.366+0000][135672][safepoint ] Entering safepoint region: CGC_Operation
[2022-09-21T22:47:02.366+0000][135672][gc,start ] GC(1) Pause Cleanup
[2022-09-21T22:47:02.366+0000][135672][gc ] GC(1) Pause Cleanup 7M->7M(2048M) 0.379ms
[2022-09-21T22:47:02.366+0000][135672][gc,cpu ] GC(1) User=0.00s Sys=0.00s Real=0.00s
[2022-09-21T22:47:02.366+0000][135672][safepoint ] Leaving safepoint region
[2022-09-21T22:47:02.366+0000][135672][safepoint ] Total time for which application threads were stopped: 0.0004997 seconds, Stopping threads took: 0.0000626 seconds
[2022-09-21T22:47:02.366+0000][135672][gc,marking ] GC(1) Concurrent Cleanup for Next Mark
[2022-09-21T22:47:02.377+0000][135672][gc,marking ] GC(1) Concurrent Cleanup for Next Mark 11.072ms
[2022-09-21T22:47:02.377+0000][135672][gc ] GC(1) Concurrent Cycle 33.152ms
[2022-09-21T22:47:02.390+0000][135672][safepoint ] Application time: 0.0237725 seconds
[2022-09-21T22:47:02.390+0000][135672][safepoint ] Entering safepoint region: Deoptimize
[2022-09-21T22:47:02.390+0000][135672][safepoint ] Leaving safepoint region

2

From your question it sounds like you're having trouble getting Enterprise Search running, but those logs look like they're JVM or GC logs. I'd expect there are application logs from Enterprise Search that should provide more useful debugging info. Here are some general docs on how Enterprise Search logs output in various environments: Manage your logs | Elastic Enterprise Search documentation [8.4] | Elastic.

I hope that helps. Let us know if you're able to find any more info in your logs and we can try to help debug.

3

It's the only log being created in the default path /var/log/enterprise-search/. Debug is set in the enterprisesearch conf file yet it never seems to get past the errors above.

4

What version of Enterprise Search are you trying to run, and how are you running it?

5

8.4.2. Locally in a 3 node cluster. It's running on one of the Elasticsearch nodes 16vcpu with 32gb ram as it's a none production enterprise search instance mostly for my own tinkering as we're 90% on prim and not a dev shop so limited use. Kibana is running on a different node.

Elasticsearch is capped at 16Gb ram so the host isn't resources constrained by any means.

6

Can you try running Enterprise Search in the foreground? I'm confident we're simply not seeing the main application logs emitted by Enterprise Search that would indicate the real error.

Also, I realize I never asked: what symptom makes you think Enterprise Search isn't starting?

7

Found the cause for the initial fail.
Secret_Session_Key was causing the startup to fail with invalid yml.

Now the fail.
//
We need to perform 1/21 migrations before the service can be started.
Migrations pending: 20220616181308

Proceeding with migrations while indices are allowing writes can have unintended consequences.
Please enable read-only mode before proceeding:
Read-only mode | Elastic Enterprise Search documentation [8.4] | Elastic
\

I could care less about keeping anything. Would rather delete and start fresh as the last time it was tested was back in 7.15.

8

Well this looks like a pretty good reason to fail. Not a clue on how to fix that part.

[2022-09-22T20:13:54.167+00:00][226300][4004][es][DEBUG]: {
"request": {
"url": "https://somethingsomethingspaceshipname.ai:9200/_security/api_key",
"method": "delete",
"headers": {
"Authorization": "[FILTERED]",
"Content-Type": "application/json",
"x-elastic-product-origin": "enterprise-search",
"User-Agent": "Faraday v1.8.0"
},
"params": null,
"body": "{"ids":["bogus"]}"
},
"response": {
"status": 200,
"headers": {
"x-elastic-product": "Elasticsearch",
"content-type": "application/json",
"content-length": "80"
},
"body": "{"invalidated_api_keys":,"previously_invalidated_api_keys":,"error_count":0}"
},
"duration": 10.3,
"stack": [
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:575:in `block in delete_raw'",
"/usr/share/enterprise-search/lib/war/lib/apm_helpers.class:41:in `es_action_instrument'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:633:in `instrument'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:574:in `delete_raw'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:436:in `invalidate_api_key'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:196:in `api_key_service_enabled?'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:90:in `check_api_key_service!'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:23:in `block in run!'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:18:in `run!'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:14:in `run!'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:286:in `configure_elasticsearch!'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:265:in `configure!'",
"/usr/share/enterprise-search/lib/war/config/application.class:20:in `'",
"/usr/share/enterprise-search/lib/war/config/application.rb:1:in `'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:36:in `initialize'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
"/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'",
"bin/enterprise-search-internal:15:in `'"
]
}

9

I'm not seeing an actual error message in what you've most recently pasted. Let me know if you're still dealing with read-only mode issues as referenced in the earlier post.

I agree that if you don't care about any existing data, it may be easiest to start fresh.

10

Got it to run but it will only run in cli "interactive" mode not as a service.

After looking at it for a few min's again I think I'll be abandoning enterprise search for now as it's still very limited usefulness in my use cases.

I'll check it out mid next year and see if it has changed enough to make it beneficial in our environment.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.