Intermittent Error Which breaks Entreprise Search UI

Philippe_Trounev · December 11, 2020, 1:51am

So we are facing an error that breaks entire app-search UI. We've been facing it for 6 month and the only way to fix it is to delete all the indexes and rebuild everything from scratch. Which is not ideal and causes loss of a lot of analytics data etc.

This error is reproduced in the actual UI on the list engines call. This is the stack trace (it returns error 500)

Is this is a known issue?

[2020-12-11T01:41:25.196+00:00][1][2508][action_controller][INFO]: [d8696bd5-db4d-42ff-8200-cec6bc288414] Completed 500 Internal Server Error in 549ms
[2020-12-11T01:41:25.214+00:00][1][2508][app-server][ERROR]: [d8696bd5-db4d-42ff-8200-cec6bc288414] Exception: Engine::MissingDocumentTypeException: loco moco document type does not exist for engine: 5fd2c565edbbf1cb24614c7b
/usr/share/enterprise-search/lib/war/app/models/engine.class:578:in `loco_moco_document_type': loco moco document type does not exist for engine: 5fd2c565edbbf1cb24614c7b (Engine::MissingDocumentTypeException)
	from /usr/share/enterprise-search/lib/war/loco_moco/app/controllers/loco_moco/dashboard/concerns/formatting_concern.class:180:in `format_engine_details'
	from /usr/share/enterprise-search/lib/war/loco_moco/app/controllers/loco_moco/dashboard/concerns/formatting_concern.class:146:in `block in format_engines_with_pagination'
	from org/jruby/RubyArray.java:1814:in `each'
	from /usr/share/enterprise-search/lib/war/lib/actastic/relation.class:298:in `each'
	from org/jruby/RubyEnumerable.java:891:in `map'
	from /usr/share/enterprise-search/lib/war/loco_moco/app/controllers/loco_moco/dashboard/concerns/formatting_concern.class:145:in `format_engines_with_pagination'
	from /usr/share/enterprise-search/lib/war/loco_moco/app/controllers/loco_moco/dashboard/engines_controller.class:31:in `block in index'
	from /usr/share/enterprise-search/lib/war/app/controllers/concerns/pagination.class:39:in `respond_with_pagination'
	from /usr/share/enterprise-search/lib/war/loco_moco/app/controllers/loco_moco/dashboard/engines_controller.class:30:in `index'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/abstract_controller/base.rb:198:in `process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/rendering.rb:10:in `process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:117:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:555:in `block in compile'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:505:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/abstract_controller/callbacks.rb:19:in `process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/rescue.rb:29:in `process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/instrumentation.rb:36:in `block in process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/notifications.rb:164:in `block in instrument'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/notifications.rb:164:in `instrument'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/abstract_controller/base.rb:137:in `process'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionview-4.2.11.3/lib/action_view/rendering.rb:30:in `process'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal.rb:196:in `dispatch'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_controller/metal.rb:237:in `block in action'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/routing/route_set.rb:43:in `serve'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/journey/router.rb:43:in `block in serve'
	from org/jruby/RubyArray.java:1814:in `each'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/journey/router.rb:30:in `serve'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/routing/route_set.rb:817:in `call'
	from /usr/share/enterprise-search/lib/war/app/middleware/system_logging_middleware.class:36:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-attack-5.0.1/lib/rack/attack.rb:147:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
	from org/jruby/RubyKernel.java:1193:in `catch'
	from /usr/share/enterprise-search/lib/war/gems/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/etag.rb:24:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/breach-mitigation-rails-0.2.2/lib/breach_mitigation/length_hiding.rb:10:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/conditionalget.rb:25:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/head.rb:13:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
	from /usr/share/enterprise-search/lib/war/app/middleware/content_security_policy_middleware.class:11:in `call'
	from /usr/share/enterprise-search/lib/war/app/middleware/content_type_middleware.class:20:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/flash.rb:260:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/session/abstract/id.rb:252:in `context'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/session/abstract/id.rb:247:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/cookies.rb:560:in `call'
	from /usr/share/enterprise-search/lib/war/app/middleware/cookie_strip_middleware.class:12:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
	from /usr/share/enterprise-search/lib/war/shared_togo/config/initializers/actionpack.class:23:in `call_app'
	from /usr/share/enterprise-search/lib/war/gems/gems/railties-4.2.11.3/lib/rails/rack/logger.rb:20:in `block in call'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/tagged_logging.rb:68:in `block in tagged'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/tagged_logging.rb:26:in `tagged'
	from /usr/share/enterprise-search/lib/war/gems/gems/activesupport-4.2.11.3/lib/active_support/tagged_logging.rb:68:in `tagged'
	from /usr/share/enterprise-search/lib/war/gems/gems/railties-4.2.11.3/lib/rails/rack/logger.rb:20:in `call'
	from /usr/share/enterprise-search/lib/war/app/middleware/silencer_middleware.class:9:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/request_id.rb:21:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/methodoverride.rb:22:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/runtime.rb:18:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/actionpack-4.2.11.3/lib/action_dispatch/middleware/static.rb:120:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/rack-1.6.13/lib/rack/sendfile.rb:113:in `call'
	from /usr/share/enterprise-search/lib/war/app/middleware/stats_middleware.class:10:in `call'
	from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/external_host_middleware.class:26:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/railties-4.2.11.3/lib/rails/engine.rb:518:in `call'
	from /usr/share/enterprise-search/lib/war/gems/gems/railties-4.2.11.3/lib/rails/application.rb:165:in `call'
	from uri:classloader:/rack/handler/servlet.rb:22:in `call'

Sean_Story · December 11, 2020, 4:09pm

Hi @Philippe_Trounev,

I'm sorry to hear that you've been fighting this issue for quite a while. 6 months is a long time, and we've had a number of releases during that time period. What version of Enterprise Search are you using? Have you upgraded at any point? When you say "rebuild everything from scratch", can you tell me what exactly you've done?

Have you noticed any pattern of behavior that causes this? Does the engine work for a while and then something causes it to start giving this error? Or is it immediately broken, even right after "rebuilding everything from scratch?"

Have you looked in your Elasticsearch logs to see if there are any errors there? Is your Elasticsearch cluster healthy?

Looking through our records, we've had one report of this issue, which was resolved by re-creating the engine. The issue was seen on an older version of the product, and we haven't seen it reported on any of the newer versions. That's why I asked above what version you're running on, and if you've upgraded.

system · January 8, 2021, 4:09pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.