Unwanted metrics

[rchmiel_fp]

Hello,

I have a working .NET based system utilizing APM 7.17.9 and data streams. Lastly we implemented filter to remove logging transactions for [OPTION] requests which works but we still get metrics logged for those requests. Is there any option to remove those metrics too? They make up total ~20% of all metrics we gather for requests and since we don't need to track them and want to save some space we wonder if we can somehow filter them out too.

Our system is .net based application, filter looks like below:

public static ITransaction FilterOutPreflight(ITransaction transaction)
        {
            if (transaction.Type?.ToUpperInvariant() == "REQUEST")
            {
                if (!string.IsNullOrEmpty(transaction.Name)
                    && transaction.Name.StartsWith("OPTIONS", StringComparison.InvariantCultureIgnoreCase))
                {
                    return null;
                }

                if (transaction.Context?.Request != null
                    && transaction.Context.Request.Method.Equals("OPTIONS", StringComparison.InvariantCultureIgnoreCase))
                {
                    return null;
                }
            }

            return transaction;
        }

And is registered using method from Elastic.Apm.Agent class

public static bool AddFilter(Func<ITransaction, ITransaction> filter);

That alone works but we still have tons of metric we don't really need nor want. Data is stored in .ds-traces-apm-default index and, as shown on screen there is quite a lot of it

What can we do about it?

[carly.richmond]

Hi @rchmiel_fp,

Welcome to the community! Thanks for sharing your .NET filter example. I can think of a couple of options for you to try:

1. Similar to what you've specified and the remove example here in the documentation, I wonder if you could return a modified transaction with only the fields you want by modifying the above code.
2. You could also try creating and registering a custom ingest pipeline containing remove processors to drop known fields you don't want, or conversely use the keep option on the remove processor to only retain the fields you want. If there are additional events you want to drop you can also use the drop processor. I would recommend having a look at this tutorial and this other tutorial for a rough idea of the create and registration steps.

Hope that helps! Do let us know if you have any questions on the above.

[carly.richmond]

You can also look at using the disable_metrics option too. Thanks to @GregKalapos for pointing this out!

[rchmiel_fp]

To clarify what I want to achieve. I don't want at all to collect metrics for requests of method type OPTIONS. Nothing more, nothing less. Is that achievable?

[carly.richmond]

It looks like you can by specifying a wildcard string with disable_metrics setting as per the docs.

[GregKalapos]

One thing to note here is that metrics and traces are orthogonal in general. The only metrics that are tied to transactions (and spans) are the breakdown metrics (those are the ones that power the "time spent by span type" charts) - everything else (CPU, Memory, GC usage, etc.) is just periodically collected - regardless of the transaction type. If there is no transaction, they are still collected. So from this reason, there isn't really any way to drop metrics based on transaction type - most of the metrics aren't associated to or triggered by any transaction. We don't have a filters API for metrics, but even if we would have it, you could still not query transactions in metrics filters from reasons described before.

disable_metrics would stop the collection altogether and I feel that's the easiest and best you can do to save storage.

If you want to really have some advanced logic to check transactions and their type and drop metrics based on that, then you can look into ingest pipelines suggested by @carly.richmond, but that'd be a bit more work.