How to reduce APM transactions and metrics?

[EZprogramming (Kourosh Azizi)]

Kibana version:

• 7.3.2

Elasticsearch version:

• 7.3.2

APM Server version:

• APM 7.3.2

APM Agent language and version:

• Python Flask 4.2.2

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

• Provided by Elastic Cloud subscription

Fresh install or upgraded from another version?

• Upgraded from 7.1.1

Additional information:

• We are using the Elastic Cloud services, so we are not running our stack locally.

Problem:

1. I receive a lot of transactions that check for /ping or /healthcheck of a URL that doesn't exist anymore. I receive a lot of these every day and I just want to find a way to get rid of them.

2. Another question that I have is what if I reduce the transaction sample rate, do APM store transactions somewhere and then update them all together or do I lose access to those transactions? Currently, we are monitoring our website using APM so we can better understand customer's behaviors/issues, and my concern is that I might lose access to this information.

Objective:

• Our plan is to reduce the data that we receive from APM, and only gather the data that is useful to us. Using this approach can help us reduce the amount of data which ultimately reduces our shards and can contribute to the maintenance of our cluster.

I appreciate any feedback or resources that you might have, feel free to ask for more information if you think there is missing detail in this post.

[basepi (Colton Myers)]

1. The transaction_ignore_patterns config should suit your needs nicely. Add the transactions in question and they will be dropped silently.

2. Transaction sampling decisions happen at the agent level, and changes to this config do not affect data already in elasticsearch. You won't lose any historical data, but future data will be sampled at the lower rate.

Hope that helps!

[EZprogramming (Kourosh Azizi)]

Hi Colton, thanks for your suggestion/clarification.

@basepi

Add the transactions in question and they will be dropped silently

Could you please elaborate on what you mean by adding the transactions to the questions? Do you mean the ignore filter?

For transaction_ignore_patterns, I believe I have add it to the Flask APM configuration in my Python code, right? I don't think it is for the apm-server.yml.

@basepi

You won't lose any historical data, but future data will be sampled at the lower rate.

My question is if for one of my services, if I lower the transaction.sample.rate from 1.0 to 0.5, does that mean I lose half of my future data (not the data already in Elasticsearch) or does it queue it and send it to me in the future at a slower rate?

[basepi (Colton Myers)]

@EZprogramming

For transaction_ignore_patterns, I believe I have add it to the Flask APM configuration in my Python code, right?

Yes, here is the flask-specific documentation for ignoring the routes: Flask support | APM Python Agent Reference [6.x] | Elastic

@EZprogramming

My question is if for one of my services, if I lower the transaction.sample.rate from 1.0 to 0.5, does that mean I lose half of my future data (not the data already in Elasticsearch) or does it queue it and send it to me in the future at a slower rate?

You will lose half of future data. Note that even unsampled transactions are recorded, but they only have the name of the transaction, the overall transaction time, and the result. They won't have any spans or additional data.

Performance tuning | APM Python Agent Reference [6.x] | Elastic

[EZprogramming (Kourosh Azizi)]

@basepi

Yes, here is the flask-specific documentation for ignoring the routes: Flask support | APM Python Agent Reference [6.x] | Elastic

It worked!

@basepi

You will lose half of future data. Note that even unsampled transactions are recorded, but they only have the name of the transaction, the overall transaction time, and the result. They won't have any spans or additional data.

Makes sense. Is there any way to drop transaction samples that are set to false? When I set transaction.sample.count to 0, transaction.sample got set to false instead and I don't want to log these transactions anymore. I followed these steps (link) and added the lines in my apm-server.yml in Elastic Cloud Deployment configuration, but once I save, my cluster doesn't accept those changes.

I also figured out that it is a good idea to disable APM metrics for CPU and Memory since we don't monitor these metrics, but it didn't work, here is the documentation I used:

Configuration | APM Python Agent Reference [master] | Elastic

Here is is my APM configuration in Python code:

from elasticapm.contrib.flask import ElasticAPM

app = Flask(__name__)
app.config['ELASTIC_APM'] = {
  'SERVICE_NAME': 'FLASK-APP-PROJECT',
  'SECRET_TOKEN': '***',
  'SERVER_URL': '***',
  'DISABLE_METRICS': ["*.cpu.*", "system.memory.*", "system.process.memory.*"]
}

apm = ElasticAPM(app)

Do you know what might be the issue?

[basepi (Colton Myers)]

@EZprogramming

but once I save, my cluster doesn't accept those changes.

Hmm, you might have to show your apm-server.yml, perhaps you have a typo?

@EZprogramming

I also figured out that it is a good idea to disable APM metrics for CPU and Memory since we don't monitor these metrics, but it didn't work

For the DISABLE_METRICS, the config needs to be a comma separated string, not a list:

app.config['ELASTIC_APM'] = {
  'SERVICE_NAME': 'FLASK-APP-PROJECT',
  'SECRET_TOKEN': '***',
  'SERVER_URL': '***',
  'DISABLE_METRICS': '*.cpu.*,system.memory.*,system.process.memory.*'
}

I think that should fix it.

[EZprogramming (Kourosh Azizi)]

@basepi

Hmm, you might have to show your apm-server.yml , perhaps you have a typo?

I think I'll post a new question for this.

@basepi

For the DISABLE_METRICS , the config needs to be a comma separated string, not a list:

app.config['ELASTIC_APM'] = {
  'SERVICE_NAME': 'FLASK-APP-PROJECT',
  'SECRET_TOKEN': '***',
  'SERVER_URL': '***',
  'DISABLE_METRICS': '*.cpu.*,system.memory.*,system.process.memory.*'
}

Yes I tried this, even with comma separated string, and it still sends CPU metrics. I also, tried metrics_interval set to 0 as well to completely stop the metrics from sending to Elasticsearch from APM side, and it still sent data.

I also saw some git issues opened for disabling APM metrics and I also saw this post, APM Disabling Metrics Issue. I'm not sure to what extent this issue is resolved, but for Python Flask web apps it doesn't work. Could you please test it manually as well just to confirm the issue?

[basepi (Colton Myers)]

Thanks for your patience, it does sound like something we need to check.

Can you please open an issue so that we can better track this?

[EZprogramming (Kourosh Azizi)]

No worries, glad I'm helping you guys. Eventually I need to sovle this issue as well.

I opened a Git ticket for this and tagged you, https://github.com/elastic/apm-agent-python/issues/696#issue-553080916

Let me know if you think something is missing.

[EZprogramming (Kourosh Azizi)]

Hey @basepi , any feedback/report so far? Let me know if you need additional details for your investigation.

[beniwohli (Benjamin Wohlwend @ APM)]

Hi @EZprogramming

sorry for the delay here, we haven't been able to reproduce this properly yet. As a workaround until we have a real solution, you could remove the CPU metrics from the METRICS_SETS setting. This setting is undocumented, as it can be a bit finikey. You can set it to

app.config['ELASTIC_APM']['METRICS_SETS'] = ["elasticapm.metrics.sets.transactions.TransactionsMetricSet",]

The TransactionsMetricSet is required to get the overall metrics how much time your app spends in database queries, http requests etc, so I recommend to leave it in.

Cheers
Beni

[EZprogramming (Kourosh Azizi)]

Yes that was the solution @beniwohli!

I really appreciate your answer, I tested it locally and it worked.

app.config['ELASTIC_APM'] = {
  'SERVICE_NAME': 'FLASK-APP-PROJECT',
  'SECRET_TOKEN': '...',
  'SERVER_URL': '...',
  'METRICS_SETS': "elasticapm.metrics.sets.transactions.TransactionsMetricSet"
}