# APM dropping metrics

**URL:** https://discuss.elastic.co/t/apm-dropping-metrics/182384
**Category:** APM
**Tags:** python
**Created:** [May 23, 2019, 9:18am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384 "2019-05-23T09:18:16Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![borko](https://avatars.discourse-cdn.com/v4/letter/b/58956e/32.png) [@borko](https://discuss.elastic.co/u/borko)
#### Post date: [May 23, 2019, 9:18am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/1 "2019-05-23T09:18:16Z")

</div>

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂

**Kibana version** : 6.7.0

**Elasticsearch version** : 6.7.0

**APM Server version** : 6.7.0

**APM Agent language and version** : Python - elastic-apm==4.2.1

**Browser version** :

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: YUM

**Fresh install or upgraded from other version?** Fresh

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

ec2 instance running ElasticSearch  
ec2 instance running both APM server and Kibana  
2 ec2 instances running the project with agent installed

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

We noticed that there are flat lines in our metrics as of recently, nothing changed except updating the agent from 3.0.2 to 4.2.1. But the updated agent reported properly. All I can offer as of now is:  
 ![20](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d96914a3ece9be5b9d4e58575e6a6cca02314a48.png)  
Here you can see a wider period of time like a month (update happened 2 months ago)  
 ![39](https://us1.discourse-cdn.com/elastic/original/3X/2/1/215a511bc3b6d77bdac1a2c4f6f32acf0c460813.png)  
Closer look  
 ![52](https://us1.discourse-cdn.com/elastic/original/3X/d/6/d67180c533a7454ba31c123b47cf38e13ae298d0.png)  
Here you can see the gaps that are going on, almost like if the TPM doesn't exceed a limit it stops reporting.

Another thing I notices is that we have N/A requests reported and for the life of me I cannot see what those are:

 ![36](https://us1.discourse-cdn.com/elastic/original/3X/4/5/45c165008689b0441eda9c3580cddb20684041f1.png)

I'm looking for advice as to what to look at as there are multiple components that could be an issue. I'm not sure what could have changed as all the components worked and we did not change the configuration or other parts of the system that worked a month without any issue.

**Steps to reproduce** :  
1.  
2.  
3.

**Errors in browser console (if relevant)**:

**Provide logs and/or server output (if relevant)**:

Not sure what is relevant.

---

<div class="post-metadata">

### Author: ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)
#### Post date: [May 23, 2019, 9:45am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/2 "2019-05-23T09:45:26Z")

</div>



---

<div class="post-metadata">

### Author: ![wa7son](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wa7son/32/21259_2.png) [@wa7son](https://discuss.elastic.co/u/wa7son)
#### Post date: [May 23, 2019, 9:45am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/3 "2019-05-23T09:45:58Z")

</div>



---

<div class="post-metadata">

### Author: ![jalvz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jalvz/32/45613_2.png) [@jalvz](https://discuss.elastic.co/u/jalvz)
#### Post date: [May 23, 2019, 10:14am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/4 "2019-05-23T10:14:02Z")

</div>

Hello,

Sorry that you are experiencing this. In order to confirm our rule out an APM Server issue, can you provide server logs?  
Do you see 4XX or 5XX HTTP errors (eg: queue is full)?

Juan

---

<div class="post-metadata">

### Author: ![beniwohli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beniwohli/32/21256_2.png) [@beniwohli](https://discuss.elastic.co/u/beniwohli)
#### Post date: [May 23, 2019, 12:37pm UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/5 "2019-05-23T12:37:01Z")

</div>

Hi @borko

in addition to server logs, it would be super helpful to know if the agent logs any problems. If you haven't set up logging for your Django app yet, you can add something like this to your `settings.py` file:

```
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(name)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'),
        },
        'elasticapm': {
            'handlers': ['console'],
            'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'),
        },
    },
}

```

You should start seeing log messages on your console now (or wherever you configured STDOUT to be directed to).

---

<div class="post-metadata">

### Author: ![borko](https://avatars.discourse-cdn.com/v4/letter/b/58956e/32.png) [@borko](https://discuss.elastic.co/u/borko)
#### Post date: [June 3, 2019, 11:45am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/6 "2019-06-03T11:45:56Z")

</div>

Hi guys,

@jalvz and @beniwohli

Thanks for the help, I manage to see that the disk filled up and landed on expanding the disk space.

The thing that I did not realize is that I had to run this:

curl -XPUT -H "Content-Type: application/json" [http://localhost:9200/\_all/\_settings](http://localhost:9200/_all/_settings) -d '{"index.blocks.read\_only\_allow\_delete": null}'

In order to reset the indexes not to be read-only.

I think this topic can be closed and I really want to thank you for steering me in the right direction.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 24, 2019, 7:45am UTC](https://discuss.elastic.co/t/apm-dropping-metrics/182384/7 "2019-06-24T07:45:59Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
