I have two search indexes that are automatically generated when creating two web crawlers for our internal sites. Elastic has been working great for a few weeks and suddenly today when I tried to restart Enterprise Search to apply new configurations, one of the search indexes started showing internal server error.
It looks like Elastic couldn't access this endpoint and got 502 Gateway:
For our other index, the endpoint successfully returns a response so I doubt if it had anything to do with Gateway errors.
Inside the Enterprise Search pod, there's an error logged:
[2024-11-19T20:42:32.258+00:00][7][7292][app-server][INFO]: Started GET "/api/ent/v1/internal/indices/search-dts/crawler2" for 10.0.1.47 at 2024-11-19 20:42:32 +0000
[2024-11-19T20:42:32.265+00:00][7][7292][action_controller][INFO]: Processing by Api::SharedTogo::V1::Internal::Crawler2::OverviewController#index as JSON
[2024-11-19T20:42:32.382+00:00][7][7292][app-server][ERROR]: Exception: Unexpected internal error while processing a request: NoMethodError: undefined method `symbolize_keys' for #<String:0x1f9368dc>
/usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/crawl_request.class:283:in `crawl_config_with_configuration': undefined method `symbolize_keys' for #<String:0x1f9368dc> (NoMethodError)
from /usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/concerns/formatting/crawler_concern.class:112:in `block in format_crawler_events_for_configuration'
from org/jruby/RubyArray.java:1865:in `each'
from /usr/share/enterprise-search/lib/war/actastic/lib/actastic/relation.class:325:in `each'
from /usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/concerns/formatting/crawler_concern.class:108:in `format_crawler_events_for_configuration'
from /usr/share/enterprise-search/lib/war/shared_togo/app/models/shared_togo/crawler2/concerns/formatting/crawler_concern.class:90:in `format_crawler_for_configuration'
from /usr/share/enterprise-search/lib/war/shared_togo/app/controllers/api/shared_togo/v1/internal/crawler2/overview_controller.class:5:in `index'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/base.rb:228:in `process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
from /usr/share/enterprise-search/lib/war/shared_togo/app/controllers/api/shared_togo/v1/internal/base_controller.class:77:in `rescue_with_logging'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:137:in `run_callbacks'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/callbacks.rb:41:in `process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:203:in `block in instrument'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:203:in `instrument'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/rescue.rb:22:in `process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/abstract_controller/base.rb:165:in `process'
from /usr/share/enterprise-search/lib/war/gems/gems/actionview-6.1.7.7/lib/action_view/rendering.rb:39:in `process'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal.rb:190:in `dispatch'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_controller/metal.rb:254:in `dispatch'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:33:in `serve'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:50:in `block in serve'
from org/jruby/RubyArray.java:1865:in `each'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/journey/router.rb:32:in `serve'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/routing/route_set.rb:842:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-attack-6.6.0/lib/rack/attack.rb:103:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/json_parser_error_middleware.class:9:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/rewrite_deprecated_routes_middleware.class:24:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/system_logging_middleware.class:37:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-attack-6.6.0/lib/rack/attack.rb:127:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/eweb_access_middleware.class:26:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
from org/jruby/RubyKernel.java:1238:in `catch'
from /usr/share/enterprise-search/lib/war/gems/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/tempfile_reaper.rb:15:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/etag.rb:27:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/conditional_get.rb:27:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/head.rb:12:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/content_type_middleware.class:21:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/content_security_policy_middleware.class:13:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/session/abstract/id.rb:266:in `context'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/session/abstract/id.rb:260:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/cookies.rb:697:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/cookie_strip_middleware.class:12:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:98:in `run_callbacks'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/actionable_exceptions.rb:22:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:37:in `call_app'
from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:26:in `block in call'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/tagged_logging.rb:99:in `block in tagged'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/tagged_logging.rb:37:in `tagged'
from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-6.1.7.7/lib/active_support/tagged_logging.rb:99:in `tagged'
from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/rack/logger.rb:26:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/silencer_middleware.class:9:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/request_id_middleware.class:11:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/method_override.rb:24:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/runtime.rb:22:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/rack-2.2.8.1/lib/rack/sendfile.rb:110:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/ssl.rb:77:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-6.1.7.7/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/stats_middleware.class:10:in `call'
from /usr/share/enterprise-search/lib/war/lib/middleware/external_host_middleware.class:26:in `call'
from /usr/share/enterprise-search/lib/war/gems/gems/railties-6.1.7.7/lib/rails/engine.rb:539:in `call'
from /usr/share/enterprise-search/lib/war/vendor/fishwife-servlet/lib/fishwife/rack_servlet.rb:74:in `service'
[2024-11-19T20:42:32.382+00:00][7][7292][app-server][ERROR]: Internal server error. Please check your application server logs for more details and contact Elastic support if the problem persists
[2024-11-19T20:42:32.386+00:00][7][7292][action_controller][INFO]: Completed 500 Internal Server Error in 121ms (Views: 1.2ms | Allocations: 0)
I'd appreciate if anyone could give me insights on how to resolve this error.