.NET Elastic APM Agent Transactions Missing

[fritziyeke]

Kibana version: 8.16.3

Elasticsearch version: 8.16.3

APM Server version: 8.16.3

APM Agent language and version: .NET 1.31.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: nuget

Fresh install or upgraded from other version? Fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

If I send multiple transactions only one of them will actually appear sampled in the Kibana portal. Its usually the first transaction to be sent that will retain metadata and labels but the rest never appear. I have tried setting the trace sample rate to "1.0" but this isn't working.

Environment.SetEnvironmentVariable("ELASTIC_APM_SERVICE_NAME", "xxx_Charlie5");
Environment.SetEnvironmentVariable("ELASTIC_APM_TRANSACTION_SAMPLE_RATE", "1.0");

Tracer = Agent.Tracer;

Tracer.CaptureTransaction("TestCapture", "TestCaptureType", transaction1 =>
{
    transaction1.SetLabel("Testing more", "Very interested");
});

Tracer.CaptureTransaction("TestCapture2", "TestCaptureType2", transaction2 =>
{
    transaction2.SetLabel("Testing more", "Very interested");
});

Would add screenshots, but I'm getting an error whenever I attempt to.

[stevejgordon (Steve Gordon)]

Hi, @fritziyeke.

Thanks for the question. I think you hit the same confusion I did when I attempted to reproduce this. By default, the UI allows you to filter the views by the transaction type. If you send two different types through, only one transaction will show for each type.

Do you see this dropdown in the UI to control which transactions you wish to view?

I have started discussing this on the Kibana repo, as it was confusing to me, too.

[fritziyeke]

@stevejgordon Thank you for your quick response. I am aware of the different views for transaction types.

The issue I'm referring to it with the sampling of my transactions. I have confirmed my sample rate is 1, but in spite of this whenever I send multiple transactions I can only view one transaction with its full metadata a labels.

Whether the transactions are of the same type or different this issue persists. I can see that all transactions were sent to Kibana but the bottom of the page will simply say:

"The selected trace cannot be found"

Instead of allowing me to view the transactions metadata. Again I wish I could provide screenshots. I think my org is preventing me

[fritziyeke]

@stevejgordon To provide more detail:

using Elastic.Apm;
using Microsoft.Extensions.DependencyInjection;
using System.Diagnostics;
using System.Windows;
using WHfB.viewmodels;

namespace WHfB;

/// <summary>
/// Interaction logic for App.xaml
/// </summary>
public partial class App : Application
{
    private IServiceProvider serviceProvider;
    //private ILoggerFactory factory;
    

    public App()
    {
        // APM Configuration
        Environment.SetEnvironmentVariable("ELASTIC_APM_RECORDING", "true");
        Environment.SetEnvironmentVariable("ELASTIC_APM_SERVICE_NAME", "xxx");
        Environment.SetEnvironmentVariable("ELASTIC_APM_SERVER_URL", "https://apm-xxx");
        Environment.SetEnvironmentVariable("ELASTIC_APM_LOG_LEVEL", "Trace");
        Environment.SetEnvironmentVariable("ELASTIC_APM_VERIFY_SERVER_CERT", "false");
        Environment.SetEnvironmentVariable("ELASTIC_APM_FLUSH_INTERVAL", "300");
        Environment.SetEnvironmentVariable("ELASTIC_APM_MAX_BATCH_EVENT_COUNT", "20");

        Agent.Setup(new AgentComponents());

        Task.Delay(7000).Wait();

        ServiceCollection services = new ServiceCollection();
        Debug.WriteLine("App.ConfigureServices:services");
        ConfigureServices(services);
        Debug.WriteLine("App.services.BuildServiceProvider");
        serviceProvider = services.BuildServiceProvider();

        
    }

    private void ConfigureServices(ServiceCollection services)
    {
        
        //services

        Agent.Tracer.CaptureTransaction("ConfigureServices", "Entry", (x) =>
        {
            x.CaptureSpan("ConfigureServices", "Initiate", (s) => {
                Debug.WriteLine($"Apm.Agent.IsConfigured: {Elastic.Apm.Agent.IsConfigured}");
                services.AddSingleton<MainWindow>();
                services.AddSingleton<MainWindowViewModel>();
                services.AddLogging();

            });
        });
    }

    /// <summary>
    /// Main entry point for application
    /// </summary>
    /// <param name="e"></param>
    protected override void OnStartup(StartupEventArgs e)
    {

        Agent.Tracer.CaptureTransaction("Debug", "Entry", (x) =>
        {
            x.CaptureSpan("Debug", "Initiate", (s) =>
            {
                Debug.WriteLine("App.OnStartup");
                Debug.WriteLine("App.OnStartup");
                Debug.WriteLine("App.OnStartup");
            });
        });

        Agent.Tracer.CaptureTransaction("OnStartup", "Entry", (x) =>
        {
            x.CaptureSpan("OnStartUp", "Initiate", (s) =>
            {
                base.OnStartup(e);
                var mainWindow = serviceProvider.GetService<MainWindow>();
                Debug.WriteLine($"Apm.Agent.IsConfigured: {Elastic.Apm.Agent.IsConfigured}");
                Debug.WriteLine("App.OnStartup.mainWindow.Show");

                Debug.WriteLine($"Config: {Agent.Config}");
                mainWindow?.Show();
            });
        });

        
    }
}

For example with the code above. The transactions "ConfigureServices" and "Debug" both get sent to Kibana, but when I click on them to view the metadata, I simply see the "The selected trace cannot be found".

In comparison though, the "OnStartup" transaction can be fully viewed. I have tried a few things to solve this including increasing the flush interval and max batch event count. And sometimes I do randomly get one of these transactions to send fully sampled, but it is never consistent. Even if I then run the exact same code again I rarely get the same results twice in a row. The only transaction that consistently sends is the "OnStartup"

[fritziyeke]

@stevejgordon
For further detail:

I've ran a test one more time with the settings above to give an idea of the issue I'm facing, I will include the results for each transaction.

OnStartup:

As with all missing transactions I've been getting the logs aren't showing an error with sampling:

[2025-04-10 12:30:01.310 +01:00][1][Trace] - {Transaction} New Transaction instance created: Transaction{Id: 3400e7ece2c12728, TraceId: 49e02298b41b7996888edd80f1e6a7f4, ParentId: null, Name: OnStartup, Type: Entry, Outcome: Unknown, IsSampled: True}. IsSampled (True) is based on the given sampler (Sampler{ constant: True }). Start time: 2025-04-10 11:30:01.3084490 UTC (as timestamp: 1744284601308449)

[2025-04-10 12:30:01.877 +01:00][1][Trace] - {Transaction} Ended Transaction{Id: 3400e7ece2c12728, TraceId: 49e02298b41b7996888edd80f1e6a7f4, ParentId: null, Name: OnStartup, Type: Entry, Outcome: Success, IsSampled: True}. Start time: 2025-04-10 11:30:01.3084490 UTC (as timestamp: 1744284601308449), End time: 2025-04-10 11:30:01.8778850 UTC (as timestamp: 1744284601877885), Duration: 569.436ms

[2025-04-10 12:30:01.884 +01:00][16][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 6. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 3400e7ece2c12728, TraceId: 49e02298b41b7996888edd80f1e6a7f4, ParentId: null, Name: OnStartup, Type: Entry, Outcome: Success, IsSampled: True}.

ConfigureServices:

This is now the transaction that I get back fully traced, but I've included the related logs anyway:

[2025-04-10 12:30:01.219 +01:00][1][Trace] - {Transaction} New Transaction instance created: Transaction{Id: 0f3160bc3aac18df, TraceId: 782ef887664f92d9ca9e2c3f64f6a220, ParentId: null, Name: ConfigureServices, Type: Entry, Outcome: Unknown, IsSampled: True}. IsSampled (True) is based on the given sampler (Sampler{ constant: True }). Start time: 2025-04-10 11:30:01.2151770 UTC (as timestamp: 1744284601215177)

[2025-04-10 12:30:01.266 +01:00][1][Trace] - {Transaction} Ended Transaction{Id: 0f3160bc3aac18df, TraceId: 782ef887664f92d9ca9e2c3f64f6a220, ParentId: null, Name: ConfigureServices, Type: Entry, Outcome: Success, IsSampled: True}. Start time: 2025-04-10 11:30:01.2151770 UTC (as timestamp: 1744284601215177), End time: 2025-04-10 11:30:01.2666810 UTC (as timestamp: 1744284601266681), Duration: 51.504ms

[2025-04-10 12:30:01.275 +01:00][16][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 2. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 0f3160bc3aac18df, TraceId: 782ef887664f92d9ca9e2c3f64f6a220, ParentId: null, Name: ConfigureServices, Type: Entry, Outcome: Success, IsSampled: True}.

Debug:

And as I've come to expect since I usually only get one transaction sampled, this trace is also missing. Logs below:

[2025-04-10 12:30:01.290 +01:00][1][Trace] - {Transaction} New Transaction instance created: Transaction{Id: 32a12deaddb05a08, TraceId: 10e682540588146e884a8cfef3a32d2e, ParentId: null, Name: Debug, Type: Entry, Outcome: Unknown, IsSampled: True}. IsSampled (True) is based on the given sampler (Sampler{ constant: True }). Start time: 2025-04-10 11:30:01.2896590 UTC (as timestamp: 1744284601289659)

[2025-04-10 12:30:01.305 +01:00][1][Trace] - {Transaction} Ended Transaction{Id: 32a12deaddb05a08, TraceId: 10e682540588146e884a8cfef3a32d2e, ParentId: null, Name: Debug, Type: Entry, Outcome: Success, IsSampled: True}. Start time: 2025-04-10 11:30:01.2896590 UTC (as timestamp: 1744284601289659), End time: 2025-04-10 11:30:01.3046980 UTC (as timestamp: 1744284601304698), Duration: 15.039ms

[2025-04-10 12:30:01.308 +01:00][16][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 4. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 32a12deaddb05a08, TraceId: 10e682540588146e884a8cfef3a32d2e, ParentId: null, Name: Debug, Type: Entry, Outcome: Success, IsSampled: True}.

I'd really appreciate some assistance with this issue, myself and different colleagues have tried a few different approaches but nothing is giving consistent results. I'm also quite concerned to have found several posts mentioning an issue with transactions showing the message:

"The selected trace cannot be found"

As most of these posts have been closed with no clear resolution.

Thank you