Stop monkey patch `Kernel#require`

[v-kolesnikov (Vasily Kolesnikov)]

APM Agent language and version: Ruby gem elastic-apm 3.6.0

Steps to reproduce:

# frozen_string_literal: true

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'rack', '~> 2.0', require: false
  gem 'elastic-apm', '3.6.0', require: false
  gem 'sidekiq', '6.0.7', require: false
  gem 'rollbar', '2.25.0', require: false
end

require 'rack'
require 'rack/lobster'

require 'elastic_apm'
ElasticAPM.start(
  service_name: 'Lobster',
  logger: Logger.new(STDOUT),
  framework_name: 'Rack',
  framework_version: Rack::VERSION.join('.')
)

require 'rollbar'
require 'rollbar/middleware/rack'
Rollbar.configure {}

app = Rack::Builder.new do
  use ElasticAPM::Middleware
  use Rollbar::Middleware::Rack
  run Rack::Lobster.new
end

at_exit do
  ElasticAPM.stop
end

Rack::Server.start(app: app)

This simple script (running as ruby server.rb) raises an error:

Failed hooking into 'sidekiq'. Please report this at github.com/elastic/apm-agent-ruby
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies/sidekiq.rb:52:in `install_processor'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies/sidekiq.rb:78:in `install'
~/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/forwardable.rb:235:in `install'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:53:in `hook_into'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:78:in `require'
~/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.0.7/lib/sidekiq/scheduled.rb:3:in `<top (required)>'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:75:in `require'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:75:in `require'
~/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:3:in `<top (required)>'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:75:in `require'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:75:in `require'
~/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:6:in `<top (required)>'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:75:in `require'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:75:in `require'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies/sidekiq.rb:50:in `install_processor'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies/sidekiq.rb:78:in `install'
~/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/forwardable.rb:235:in `install'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:53:in `hook_into'
~/.rvm/gems/ruby-2.7.1/gems/elastic-apm-3.6.0/lib/elastic_apm/spies.rb:78:in `require'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugin.rb:98:in `block in require_dependency'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugin.rb:115:in `all?'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugin.rb:115:in `dependencies_satisfy?'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugin.rb:107:in `load?'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugin.rb:54:in `load!'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugins.rb:33:in `block in load!'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugins.rb:32:in `each'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar/plugins.rb:32:in `load!'
~/.rvm/gems/ruby-2.7.1/gems/rollbar-2.25.0/lib/rollbar.rb:68:in `configure'
server.rb:26:in `<main>'

This happens by coincidence, each of which, incidentally, refers to a violation of good manners when writing general-purpose libraries.

1. Patching Kernel#require in ElasticAPM::Spies:

elastic/apm-agent-ruby/blob/ce3504ea079e0862588ba29795b1f103fc3e6179/lib/elastic_apm/spies.rb#L84-L102

2. Unconditional and uncontrolled load 3-rd party dependencies by Rollbar gem.

3. Sidekiq. It is probably the worst code over all Ruby gems ever. The particular issue in that case is requiring sidekiq inside one of Sidekiq's module:

mperham/sidekiq/blob/455e9d56f46f0299eaf3b761596207e15f906a39/lib/sidekiq/exception_handler.rb#L3

The last two are not issues that we could resolve here, but I just listed these to complete the picture.

So, what happened in this script? Let's examine:

1. require 'elastic_apm' patches Kernel#require

2. Rollbar.configure requires 'sidekiq' that calls ElasticAPM::Spies.hook_into('sidekiq') in patched Kernel#require.

3. ElasticAPM::Spies::SidekiqSpy#install_processor calls require 'sidekiq/processor' that calls require 'sidekiq' in lib/sidekiq/exception_handler.rb

It gives rise a circular dependency that could be illustrated like that:

# sidekiq.rb
module Sidekiq
end

# sidekiq/processor.rb
require 'sidekiq'

module Sidekiq
  module Processor
  end
end

# server.rb
require 'rollbar'
  require 'sidekiq' # in patched Kernel#require
    require 'sidekiq' => true  # in Kernel#require_without_apm
    require 'sidekiq/processor' # in ElasticAPM::Splies::SidekiqSpy#install_processor
      require 'sidekiq' => false # in Kernel#require_without_apm
        require 'sidekiq/processor' => false # in Kernel#require_without_apm
        calls Sidekiq::Processor # raises uninitialized constant Sidekiq::Processor
        # ...
        defines Sidekiq::Processor

Sidekiq::Processor called before than it is defined.

So, I understand that it is a complex issue related to all the three gems at the same time. I'll report to other gems as well. But please, consider to give a developers who use your library a chance to manage their application code: don't redefine Kernel module. Please.