Internal Server Error when viewing a search index page

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I have two search indexes that are automatically generated when creating two web crawlers for our internal sites. Elastic has been working great for a few weeks and suddenly today when I tried to restart Enterprise Search to apply new configurations, one of the search indexes started showing internal server error.

It looks like Elastic couldn't access this endpoint and got 502 Gateway:

image
image1662×257 21 KB

For our other index, the endpoint successfully returns a response so I doubt if it had anything to do with Gateway errors.

Inside the Enterprise Search pod, there's an error logged:

[2024-11-19T20:42:32.258+00:00][7][7292][app-server][INFO]: Started GET "/api/ent/v1/internal/indices/search-dts/crawler2" for 10.0.1.47 at 2024-11-19 20:42:32 +0000
[2024-11-19T20:42:32.265+00:00][7][7292][action_controller][INFO]: Processing by Api::SharedTogo::V1::Internal::Crawler2::OverviewController#index as JSON
[2024-11-19T20:42:32.382+00:00][7][7292][app-server][ERROR]: Exception: Unexpected internal error while processing a request: NoMethodError: undefined method `symbolize_keys' for #<String:0x1f9368dc>
/usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/crawl_request.class:283:in `crawl_config_with_configuration': undefined method `symbolize_keys' for #<String:0x1f9368dc> (NoMethodError)
        from /usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/concerns/formatting/crawler_concern.class:112:in `block in format_crawler_events_for_configuration'
        from org/jruby/RubyArray.java:1865:in `each'
        from /usr/share/enterprise-search/lib/war/actastic/lib/actastic/relation.class:325:in `each'
        from /usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/concerns/formatting/crawler_concern.class:108:in `format_crawler_events_for_configuration'
        from /usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/concerns/formatting/crawler_concern.class:90:in `format_crawler_for_configuration'
        from /usr/share/enterprise-search/lib/war/shared_togo/app/controllers/api/shared_togo/v1/internal/crawler2/overview_controller.class:5:in `index'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/base.rb:228:in `process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
        from /usr/share/enterprise-search/lib/war/shared_togo/app/controllers/api/shared_togo/v1/internal/base_controller.class:77:in `rescue_with_logging'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:137:in `run_callbacks'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:203:in `block in instrument'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:203:in `instrument'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/base.rb:165:in `process'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionview-6.1.7.7/lib/action_view/rendering.rb:39:in `process'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal.rb:190:in `dispatch'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal.rb:254:in `dispatch'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:50:in `block in serve'
        from org/jruby/RubyArray.java:1865:in `each'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:32:in `serve'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:842:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-attack-6.6.0/lib/rack/attack.rb:103:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/json_parser_error_middleware.class:9:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/rewrite_deprecated_routes_middleware.class:24:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/system_logging_middleware.class:37:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-attack-6.6.0/lib/rack/attack.rb:127:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/eweb_access_middleware.class:26:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
        from org/jruby/RubyKernel.java:1238:in `catch'
        from /usr/share/enterprise-search/lib/war/gems/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/tempfile_reaper.rb:15:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/etag.rb:27:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/conditional_get.rb:27:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/head.rb:12:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/content_type_middleware.class:21:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/content_security_policy_middleware.class:13:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/session/abstract/id.rb:266:in `context'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/session/abstract/id.rb:260:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/cookies.rb:697:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/cookie_strip_middleware.class:12:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:98:in `run_callbacks'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/actionable_exceptions.rb:22:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:37:in `call_app'
        from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:26:in `block in call'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/tagged_logging.rb:99:in `block in tagged'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/tagged_logging.rb:37:in `tagged'
        from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/tagged_logging.rb:99:in `tagged'
        from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:26:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/silencer_middleware.class:9:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/request_id_middleware.class:11:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/method_override.rb:24:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/runtime.rb:22:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/sendfile.rb:110:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/ssl.rb:77:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/stats_middleware.class:10:in `call'
        from /usr/share/enterprise-search/lib/war/lib/middleware/external_host_middleware.class:26:in `call'
        from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/engine.rb:539:in `call'
        from /usr/share/enterprise-search/lib/war/vendor/fishwife-servlet/lib/fishwife/rack_servlet.rb:74:in `service'

[2024-11-19T20:42:32.382+00:00][7][7292][app-server][ERROR]: Internal server error. Please check your application server logs for more details and contact Elastic support if the problem persists
[2024-11-19T20:42:32.386+00:00][7][7292][action_controller][INFO]: Completed 500 Internal Server Error in 121ms (Views: 1.2ms | Allocations: 0)

I'd appreciate if anyone could give me insights on how to resolve this error.

2

For anyone who stumbles upon this, remember to configure secret_management.encryption_keys in your Enterprise Search config as documented here

When restarting Enterprise Search, I didn't configure one so it seems Elastic Operator automatically generated a new one for me, which couldn't be used to decrypt old crawler information as that information was encrypted with a different key. Configuring a persistent encryption key fix the above error after restarting Enterprise Search.