Hello All, I'm having an interesting issue with enterprise search I thought I'd share. Whenever I enable this setting:
ent_search.ssl.redirect_http_from_port: 80
I get these exceptions related to Jruby:
[2023-01-19T15:05:55.598+00:00][5557][4004][app-server][ERROR]: Exception: Unexpected exception while running Enterprise Search: NameError: uninitialized constant SharedTogo::JettyServer::SecuredRedirectHandler
org/jruby/RubyModule.java:3948:in `const_missing': uninitialized constant SharedTogo::JettyServer::SecuredRedirectHandler (NameError)
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:133:in `configure_handlers'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:109:in `initialize_jetty_server'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:62:in `initialize'
from org/jruby/RubyClass.java:886:in `new'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:69:in `start_application_server'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:26:in `run'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'
from bin/enterprise-search-internal:15:in `<main>'
Unexpected exception while running Enterprise Search:
Error: uninitialized constant SharedTogo::JettyServer::SecuredRedirectHandler at org/jruby/RubyModule.java:3948:in `const_missing'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:133:in `configure_handlers'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:109:in `initialize_jetty_server'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:62:in `initialize'
org/jruby/RubyClass.java:886:in `new'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:69:in `start_application_server'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:26:in `run'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'
bin/enterprise-search-internal:15:in `<main>'
I'm wondering if this is a bug or an error because I'm not setting something correctly? I'll be more than happy to update with a sanitized version of my config file if that will help.
Here's the log:
Found java executable in PATH
Java version detected: 11.0.17 (major version: 11)
Enterprise Search is starting...
[2023-01-19T15:05:41.839+00:00][5557][4004][app-server][INFO]: Elastic Enterprise Search version=8.6.0, JRuby version=9.3.3.0, Ruby version=2.6.8, Rails version=5.2.8.1
[2023-01-19T15:05:43.008+00:00][5557][4004][app-server][INFO]: Performing pre-flight checks for Elasticsearch running on https://elastic.ornl.fne:9200...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.javasupport.binding.ConstantField (file:/usr/share/enterprise-search/lib/war/lib/jruby-core-9.3.3.0-complete.jar) to field sun.security.x509.X509CertImpl.SIG
WARNING: Please consider reporting this to the maintainers of org.jruby.javasupport.binding.ConstantField
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2023-01-19T15:05:43.407+00:00][5557][4004][app-server][INFO]: [pre-flight] Elasticsearch cluster is ready
[2023-01-19T15:05:43.409+00:00][5557][4004][app-server][INFO]: [pre-flight] Successfully connected to Elasticsearch
[2023-01-19T15:05:43.439+00:00][5557][4004][app-server][INFO]: [pre-flight] Successfully loaded Elasticsearch plugin information for all nodes
[2023-01-19T15:05:43.444+00:00][5557][4004][app-server][INFO]: [pre-flight] Elasticsearch running with an active basic license
[2023-01-19T15:05:43.456+00:00][5557][4004][app-server][INFO]: [pre-flight] Elasticsearch API key service is enabled
[2023-01-19T15:05:43.460+00:00][5557][4004][app-server][INFO]: [pre-flight] Elasticsearch will be used for authentication
[2023-01-19T15:05:43.461+00:00][5557][4004][app-server][INFO]: Elasticsearch looks healthy and configured correctly to run Enterprise Search
[2023-01-19T15:05:43.462+00:00][5557][4004][app-server][INFO]: Performing pre-flight checks for Kibana running on https://kibana.ornl.fne...
[2023-01-19T15:05:43.585+00:00][5557][4004][app-server][INFO]: [pre-flight] Successfully connected to Kibana
[2023-01-19T15:05:44.362+00:00][5557][4004][app-server][INFO]: Kibana looks healthy and configured correctly to run Enterprise Search
[2023-01-19T15:05:46.813+00:00][5557][4004][app-server][INFO]: Elastic APM agent is disabled
[2023-01-19T15:05:53.037+00:00][5557][4004][app-server][WARN]: Index .elastic-connectors-v1 already exists!
[2023-01-19T15:05:53.049+00:00][5557][4004][app-server][WARN]: Going to re-create the existing empty index .elastic-connectors-v1
[2023-01-19T15:05:53.154+00:00][5557][4004][app-server][WARN]: Index .elastic-connectors-sync-jobs-v1 already exists!
[2023-01-19T15:05:53.164+00:00][5557][4004][app-server][WARN]: Going to re-create the existing empty index .elastic-connectors-sync-jobs-v1
[2023-01-19T15:05:53.262+00:00][5557][4004][app-server][INFO]: [db_lock] [installation] Status: [Starting] Ensuring migrations tracking index exists
[2023-01-19T15:05:53.282+00:00][5557][4004][app-server][INFO]: [db_lock] [installation] Status: [Finished] Ensuring migrations tracking index exists
[2023-01-19T15:05:53.495+00:00][5557][4004][app-server][INFO]: Found no migrations to run
[2023-01-19T15:05:53.495+00:00][5557][4004][app-server][INFO]: Enterprise Search indices are ready
[2023-01-19T15:05:53.557+00:00][5557][4004][app-server][INFO]: Elasticsearch ingest pipeline app_search_crawler (version 1) already exists
[2023-01-19T15:05:53.561+00:00][5557][4004][app-server][INFO]: Elasticsearch ingest pipeline ent_search_crawler (version 2) already exists
[2023-01-19T15:05:53.565+00:00][5557][4004][app-server][INFO]: Elasticsearch ingest pipeline elastic_analytics_events (version 1) already exists
[2023-01-19T15:05:53.571+00:00][5557][4004][app-server][INFO]: Elasticsearch ingest pipeline ent-search-generic-ingestion (version 1) already exists
[2023-01-19T15:05:53.577+00:00][5557][4004][app-server][INFO]: Elasticsearch ingest pipeline ent_search_connector (version 2) already exists
[2023-01-19T15:05:55.045+00:00][5557][4004][app-server][INFO]: Starting background workers...
[2023-01-19T15:05:55.048+00:00][5557][4004][app-server][INFO]: [workers] Configuring periodic background jobs...
[2023-01-19T15:05:55.049+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job CleanupHeartbeatIndex: {:execution_interval=>300, :timeout_interval=>30, :run_now=>false}
[2023-01-19T15:05:55.050+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job RefreshElasticsearchLicense: {:execution_interval=>60, :timeout_interval=>30, :run_now=>false}
[2023-01-19T15:05:55.051+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job RefreshFritoPieContentSources: {:execution_interval=>360, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.052+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job UpdateCustomSourcesConfig: {:execution_interval=>360, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.053+00:00][5557][4380][app-server][INFO]: Running task: RefreshFritoPieContentSources
[2023-01-19T15:05:55.053+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job DeleteOldConnectorsJobs: {:execution_interval=>21600, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.054+00:00][5557][4388][app-server][INFO]: Running task: UpdateCustomSourcesConfig
[2023-01-19T15:05:55.056+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job DeleteOldDoorkeeperRecords: {:execution_interval=>21600, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.056+00:00][5557][4400][app-server][INFO]: Running task: DeleteOldConnectorsJobs
[2023-01-19T15:05:55.058+00:00][5557][4380][cron-Work::Cron::RefreshFritoPieContentSources][INFO]: Performing task: RefreshFritoPieContentSources
[2023-01-19T15:05:55.058+00:00][5557][4388][cron-Work::Cron::UpdateCustomSourcesConfig][INFO]: Performing task: UpdateCustomSourcesConfig
[2023-01-19T15:05:55.060+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job KeepSidecarsAlive: {:execution_interval=>300, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.060+00:00][5557][4400][cron-Work::Cron::DeleteOldConnectorsJobs][INFO]: Performing task: DeleteOldConnectorsJobs
[2023-01-19T15:05:55.077+00:00][5557][4440][app-server][INFO]: Running task: DeleteOldDoorkeeperRecords
[2023-01-19T15:05:55.078+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job ScheduleCrawls: {:execution_interval=>3600, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.082+00:00][5557][4448][app-server][INFO]: Running task: KeepSidecarsAlive
[2023-01-19T15:05:55.084+00:00][5557][4448][cron-Work::Cron::KeepSidecarsAlive][INFO]: Performing task: KeepSidecarsAlive
[2023-01-19T15:05:55.084+00:00][5557][4440][cron-Work::Cron::DeleteOldDoorkeeperRecords][INFO]: Performing task: DeleteOldDoorkeeperRecords
[2023-01-19T15:05:55.085+00:00][5557][4448][app-server][INFO]: [sidecar] [filebeat] Starting the child process...
[2023-01-19T15:05:55.090+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job ScheduleCrawls2: {:execution_interval=>3600, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.092+00:00][5557][4480][app-server][INFO]: Running task: ScheduleCrawls
[2023-01-19T15:05:55.094+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job SendTelemetry: {:execution_interval=>21600, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.094+00:00][5557][4480][cron-Work::Cron::ScheduleCrawls][INFO]: Performing task: ScheduleCrawls
[2023-01-19T15:05:55.100+00:00][5557][4004][app-server][INFO]: [workers] Configuring background job UpdateSearchRelevanceSuggestions: {:execution_interval=>60, :timeout_interval=>30, :run_now=>true}
[2023-01-19T15:05:55.102+00:00][5557][4004][app-server][INFO]: [workers] Configuring background jobs for Workplace Search Connectors...
[2023-01-19T15:05:55.103+00:00][5557][4004][connectors][INFO]: Starting to process jobs.
[2023-01-19T15:05:55.111+00:00][5557][4504][app-server][INFO]: Running task: ScheduleCrawls2
[2023-01-19T15:05:55.119+00:00][5557][4512][app-server][INFO]: Running task: SendTelemetry
[2023-01-19T15:05:55.137+00:00][5557][4540][app-server][INFO]: Running task: UpdateSearchRelevanceSuggestions
[2023-01-19T15:05:55.139+00:00][5557][4004][app-server][INFO]: [workers] Starting a background process for the Crawl Queue Manager...
[2023-01-19T15:05:55.145+00:00][5557][4504][cron-Work::Cron::ScheduleCrawls2][INFO]: Performing task: ScheduleCrawls2
[2023-01-19T15:05:55.145+00:00][5557][4512][cron-Work::Cron::SendTelemetry][INFO]: Performing task: SendTelemetry
[2023-01-19T15:05:55.147+00:00][5557][4540][cron-Work::Cron::UpdateSearchRelevanceSuggestions][INFO]: Performing task: UpdateSearchRelevanceSuggestions
[2023-01-19T15:05:55.156+00:00][5557][4004][app-server][INFO]: [workers] Starting a background process for the Crawler2 Queue Manager...
[2023-01-19T15:05:55.159+00:00][5557][4004][app-server][INFO]: [workers] Starting crawler watchers
[2023-01-19T15:05:55.160+00:00][5557][4004][app-server][INFO]: Starting asynchronous processing workers...
[2023-01-19T15:05:55.188+00:00][5557][4512][app-server][INFO]: [db_lock] [singleton-Work::Cron::SendTelemetry] Waiting on another instance to release the "singleton-Work::Cron::SendTelemetry" lock: {"product_version"=>"8.6.0", "locked_at"=>"2023-01-19T15:04:25+00:00", "expires_at"=>"2023-01-19T15:06:25+00:00", "last_heartbeat_at"=>"2023-01-19T15:04:25+00:00", "status_update_at"=>"2023-01-19T15:04:25+00:00", "last_status"=>"Acquiring the lock", "node_name"=>"kmssearchserver", "pid"=>5363, "tid"=>4456}
[2023-01-19T15:05:55.191+00:00][5557][4380][cron-Work::Cron::RefreshFritoPieContentSources][INFO]: Done performing task: RefreshFritoPieContentSources
[2023-01-19T15:05:55.194+00:00][5557][4448][app-server][INFO]: [sidecar] [filebeat] Writing config: /usr/share/enterprise-search/tmp/ent-search-filebeat.yml
[2023-01-19T15:05:55.195+00:00][5557][4380][app-server][INFO]: Done running task: RefreshFritoPieContentSources
[2023-01-19T15:05:55.197+00:00][5557][4592][app-server][INFO]: [workers] Starting EsqueuesMe background jobs...
[2023-01-19T15:05:55.210+00:00][5557][4628][crawler][INFO]: [cqm] Starting CQM monitoring loop (with support for up to 16 parallel crawls)...
[2023-01-19T15:05:55.211+00:00][5557][4592][worker][INFO]: Ensuring esqueues me queues exist
[2023-01-19T15:05:55.215+00:00][5557][4936][crawler2][INFO]: [cqm] Starting CQM monitoring loop (with support for up to 16 parallel crawls)...
[2023-01-19T15:05:55.266+00:00][5557][4440][cron-Work::Cron::DeleteOldDoorkeeperRecords][INFO]: Done performing task: DeleteOldDoorkeeperRecords
[2023-01-19T15:05:55.267+00:00][5557][4440][app-server][INFO]: Done running task: DeleteOldDoorkeeperRecords
[2023-01-19T15:05:55.280+00:00][5557][4540][cron-Work::Cron::UpdateSearchRelevanceSuggestions][INFO]: Done performing task: UpdateSearchRelevanceSuggestions
[2023-01-19T15:05:55.280+00:00][5557][4388][cron-Work::Cron::UpdateCustomSourcesConfig][INFO]: Done performing task: UpdateCustomSourcesConfig
[2023-01-19T15:05:55.281+00:00][5557][4388][app-server][INFO]: Done running task: UpdateCustomSourcesConfig
[2023-01-19T15:05:55.281+00:00][5557][4540][app-server][INFO]: Done running task: UpdateSearchRelevanceSuggestions
[2023-01-19T15:05:55.282+00:00][5557][4480][cron-Work::Cron::ScheduleCrawls][INFO]: Done performing task: ScheduleCrawls
[2023-01-19T15:05:55.283+00:00][5557][4480][app-server][INFO]: Done running task: ScheduleCrawls
2023-01-19 10:05:55.289:INFO::main: Logging initialized @24606ms to org.eclipse.jetty.util.log.StdErrLog
[2023-01-19T15:05:55.301+00:00][5557][4400][cron-Work::Cron::DeleteOldConnectorsJobs][INFO]: Done performing task: DeleteOldConnectorsJobs
[2023-01-19T15:05:55.301+00:00][5557][4400][app-server][INFO]: Done running task: DeleteOldConnectorsJobs
[2023-01-19T15:05:55.308+00:00][5557][4504][cron-Work::Cron::ScheduleCrawls2][INFO]: Done performing task: ScheduleCrawls2
[2023-01-19T15:05:55.308+00:00][5557][4504][app-server][INFO]: Done running task: ScheduleCrawls2
[2023-01-19T15:05:55.383+00:00][5557][4592][worker][INFO]: Ensuring heartbeat index exists
[2023-01-19T15:05:55.394+00:00][5557][4448][app-server][INFO]: [sidecar] [filebeat] Started a child process with PID #5656
[2023-01-19T15:05:55.398+00:00][5557][4448][app-server][INFO]: [sidecar] [metricbeat] Writing config: /usr/share/enterprise-search/tmp/ent-search-metricbeat.yml
[2023-01-19T15:05:55.427+00:00][5557][4448][app-server][INFO]: [sidecar] [metricbeat] Sidecar process disabled
[2023-01-19T15:05:55.428+00:00][5557][4448][cron-Work::Cron::KeepSidecarsAlive][INFO]: Done performing task: KeepSidecarsAlive
[2023-01-19T15:05:55.428+00:00][5557][4448][app-server][INFO]: Done running task: KeepSidecarsAlive
[2023-01-19T15:05:55.436+00:00][5557][4592][worker][INFO]: esqueues me while I process jobs on engine_destroyer, process_crawl, mailer, process_crawl2
[2023-01-19T15:05:55.598+00:00][5557][4004][app-server][ERROR]: Exception: Unexpected exception while running Enterprise Search: NameError: uninitialized constant SharedTogo::JettyServer::SecuredRedirectHandler
org/jruby/RubyModule.java:3948:in `const_missing': uninitialized constant SharedTogo::JettyServer::SecuredRedirectHandler (NameError)
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:133:in `configure_handlers'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:109:in `initialize_jetty_server'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:62:in `initialize'
from org/jruby/RubyClass.java:886:in `new'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:69:in `start_application_server'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:26:in `run'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'
from /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'
from bin/enterprise-search-internal:15:in `<main>'
Unexpected exception while running Enterprise Search:
Error: uninitialized constant SharedTogo::JettyServer::SecuredRedirectHandler at org/jruby/RubyModule.java:3948:in `const_missing'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:133:in `configure_handlers'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:109:in `initialize_jetty_server'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/jetty_server.class:62:in `initialize'
org/jruby/RubyClass.java:886:in `new'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:69:in `start_application_server'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/app_server_command.class:26:in `run'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'
/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'
bin/enterprise-search-internal:15:in `<main>'
[2023-01-19T15:05:55.599+00:00][5557][4004][worker][INFO]: Received shutdown signal, stopping.
[2023-01-19T15:05:55.601+00:00][5557][4004][app-server][INFO]: [sidecar] [filebeat] Stopping the child process with PID 5656 as parent process is exiting
[2023-01-19T15:05:55.615+00:00][5557][4004][app-server][INFO]: [sidecar] [filebeat] The child process has been shut down
[2023-01-19T15:05:55.616+00:00][5557][4004][app-server][INFO]: [workers] Waiting for all background workers to shut down...
[2023-01-19T15:05:55.617+00:00][5557][4004][crawler][INFO]: CQM shutdown starting...
[2023-01-19T15:05:55.617+00:00][5557][4004][crawler][INFO]: [cqm] Received a shutdown request (process shutdown), starting the crawler shutdown procedure...
[2023-01-19T15:05:55.618+00:00][5557][4004][crawler][INFO]: CQM is still running, waiting for a second before checking back...
[2023-01-19T15:05:55.761+00:00][5557][4628][crawler][INFO]: [cqm] Shutting down the crawl queue monitoring loop...
[2023-01-19T15:05:55.762+00:00][5557][4628][crawler][INFO]: [cqm] All crawler workers have been successfully shut down
[2023-01-19T15:05:55.762+00:00][5557][4628][crawler][INFO]: [cqm] Crawler shutdown complete!
[2023-01-19T15:05:55.951+00:00][5557][4592][worker][INFO]: EsqueuesMe worker has stopped
[2023-01-19T15:05:56.458+00:00][5557][4512][app-server][INFO]: [db_lock] [singleton-Work::Cron::SendTelemetry] Waiting on another instance to release the "singleton-Work::Cron::SendTelemetry" lock: {"product_version"=>"8.6.0", "locked_at"=>"2023-01-19T15:04:25+00:00", "expires_at"=>"2023-01-19T15:06:25+00:00", "last_heartbeat_at"=>"2023-01-19T15:04:25+00:00", "status_update_at"=>"2023-01-19T15:04:25+00:00", "last_status"=>"Acquiring the lock", "node_name"=>"kmssearchserver", "pid"=>5363, "tid"=>4456}
[2023-01-19T15:05:56.619+00:00][5557][4004][crawler][INFO]: CQM is down
[2023-01-19T15:05:56.619+00:00][5557][4004][app-server][INFO]: Background workers shutdown complete