Tracer is null

Elastic Observability APM
1

Based upon 1.27.0.0 of Elastic.Apm.NetCoreAll.

During initialization, Tracer is set to null on agentComponents after execution of own code called from Startup of website method ConfigureServices(IServiceCollection services):

AgentComponents agentComponents = new AgentComponents(null, elasticApmConfig, null);

where elasticApmConfig is of a class implementing IConfigurationReader, with Enabled = true.

According to Elastic.Apm.Agent.IsConfigured being true, the configuration is correct.

Reflection

The code of AgentComponents was checked. The constructor with 3 arguments seems to be forwarded to an internal constructor. After execution also other properties are null, such as PayloadSender, despite being initialized in the internal constructor as:

PayloadSender = payloadSender
				?? new PayloadSenderV2(Logger, ConfigurationStore.CurrentSnapshot, Service, system,					 ApmServerInfo, isEnabled: Configuration.Enabled, serverInfoCallback: ServerInfoCallback);

Question

How can Tracer be initialized correctly?

2

Problem strongly resembles Object Reference after updating 1.8.1 -> 1.9 (Elastic.Apm.Agent.Tracer is null) · Issue #1254 · elastic/apm-agent-dotnet · GitHub.

AgentComponents has:

catch (Exception e)
{
   Logger.Error()?.LogException(e, "Failed initializing agent.");
}

which can be a No-Operation. The exception is for reasons described in the issue not re-raised, and with logger missing leads to an ignored exception.

In 1.9 it seems to have been logger?.Error()?....

I will try to use a logger.

3

With logger attached, the problem becomes clear, an unhandled NullReferenceException on constructor of PayloadSenderV2:

   at Elastic.Apm.Report.PayloadSenderV2..ctor(IApmLogger logger, IConfiguration configuration, Service service, System system, IApmServerInfo apmServerInfo, HttpMessageHandler httpMessageHandler, String dbgName, Boolean isEnabled, IEnvironmentVariables environmentVariables, Action`2 serverInfoCallback)
   at Elastic.Apm.AgentComponents..ctor(IApmLogger logger, IConfigurationReader configurationReader, IPayloadSender payloadSender, IMetricsCollector metricsCollector, ICurrentExecutionSegmentsContainer currentExecutionSegmentsContainer, ICentralConfigurationFetcher centralConfigurationFetcher, IApmServerInfo apmServerInfo, BreakdownMetricsProvider breakdownMetricsProvider)

The root cause is the following code:

var process = ProcessInformation.Create();
_metadata = new Metadata { Service = service, System = System, Process = process };
foreach (var globalLabelKeyValue in configuration.GlobalLabels)
	_metadata.Labels.Add(globalLabelKeyValue.Key, globalLabelKeyValue.Value);
_cachedActivationMethod = _metadata.Service?.Agent.ActivationMethod;
ResetActivationMethodIfKnownBrokenApmServer();

GlobalLabels is an undocumented entry in IConfigurationReader, and there is no clue whether it is nullable or not (the Elastic APM package does not yet use the new checks to avoid NullReferenceExceptions such as by

After changing the custom class implementing IConfigurationReader to use:

        [NSJ.JsonProperty("GlobalLabels")]
        [STJ.JsonPropertyName("GlobalLabels")]
        [ConfigurationKeyName("GlobalLabels")]
        public IReadOnlyDictionary<string, string> GlobalLabels { get; set; } = new Dictionary<string, string>();

the specific exception does not occur.

Take aways

  • Always add a logger, possibly with a crash on an error. The initialization of AgentComponents violates the Fail-Fast principle behind Ada.
  • I was unable to find a sample implementation of IConfigurationReader and IApmLogger. Take your time to craft one nonetheless.

The maintainer might want to consider adding an if to the foreach, or using the new language features to avoid this type of error.