# Update to apm-server 7.9 increased my APM doc count (a Lot!) **URL:** \ **Category:** APM\ **Tags:** python, server\ **Created:** [September 7, 2020, 4:53pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791 "2020-09-07T16:53:55Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 7, 2020, 4:53pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/1 "2020-09-07T16:53:56Z")
Hi!, I've been using APM with elastic for some months now. Everything's is fine so far but about 2 weeks ago I updated from 7.5 to 7.9. I noticed since then that the amount of documents in my apm\* indices has grown, a lot (about 10x) I've trying to see what's the source of this since there seems to be no noticeable change in the amount of transactions per unit of time. So to be clear, the increment is in the spans indices. Elaticstack version at 7.9 (Elasitc, Kibana and apmserver) I'm using python apm agent 5.8 Any ideas why with the same "input" i'm getting that many more docs ? Thanks Rodrigo. ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 8, 2020, 2:50am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/2 "2020-09-08T02:50:33Z")
If the rate of transaction docs has remained stable, but the rate of span docs has increased, that could be due to a change in sampling rate. You could check this by comparing the number of transaction docs with `transaction.sampled: true` vs `transaction.sampled: false`, using Discover. If it is the case that the sampling rate has changed: as far as I know just upgrading the server from 7.5 to 7.9 wouldn't cause that. Did you (or someone else) also change agent sampling rate configuration around the same time? ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 8, 2020, 9:03am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/3 "2020-09-08T09:03:29Z")
Hi! thanks for you reply! I just checked and no, my `apm-7.5.1-transaction-000004` index has all docs with sampled in true; `apm-7.9.0-transaction-000001` is showing the same sample value. I also checked the agent's config and it's set to `transaction_sample_rate`: 1 One thing that I had to "fix" was the ingest pipeline for apm: [https://www.elastic.co/guide/en/apm/server/current/upgrading-to-77.html](https://www.elastic.co/guide/en/apm/server/current/upgrading-to-77.html). I did that yesterday but I see no effects on my docs yet (I understand that this should impact docs size and not doc count, but I thought it's interesting..) Thanks! ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [September 8, 2020, 9:30am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/4 "2020-09-08T09:30:35Z")
Have you also recently updated the python agent or only the APM Server? ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 8, 2020, 1:32pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/5 "2020-09-08T13:32:28Z")
No I did not , we're running our django code with this `elastic-apm==5.3.2` package ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 10, 2020, 1:28am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/6 "2020-09-10T01:28:47Z")
@Rodrigo_Jimenez the only thing that we (Silvia) have thought of as a possible reason is changes in the server's "self-instrumentation" (`apm-server.instrumentation.*` config). I don't suppose you have enabled that? If that is the case, then the spike in docs should all have `service.name: "apm-server"`. Otherwise, I'm really not sure what could have caused a spike in docs. If you can find anything common to the additional docs, then that might help us narrow it down. ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 10, 2020, 7:24pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/7 "2020-09-10T19:24:24Z")
Ok, so, first of all, thanks for taking the time (you and Silvia) to help out with this, Regarding apm-server config, no , the only special config is: ` setup.template.enabled: true` ` setup.template.overwrite: true` ` setup.template.append_fields:` ` - name: http.request.headers.Origin` ` type: keyword` (format looks wrong here, but the config syntax is ok) The idea was to be able to aggregate by this field (I shows as indexable now in kibana , but filtering does not work. I always shows no documents ☹ ). It's important for me since our React frontend will send the Origin field and that way I would be able to see which front end instances are hitting my API. Back to the issue of increased volume, trying to find some field of type of new type of doc is the first thing I tried; basically, I grabbed the top 5 menu in Discover, but since it takes the last 500 docs, I decided to create some ad-hoc visualizations and aggregate by terms (span type, subtype, host.name). All I could find is that only one of the agents shows a spike. ![dev](https://us1.discourse-cdn.com/elastic/original/3X/9/b/9b5f6b85fb9d516571703fcd6c6d6295ce1548d2.png) ![staging](https://us1.discourse-cdn.com/elastic/original/3X/e/4/e4039370c980a815c7f16fbf65d9a08c65bdf48e.png) ![prod](https://us1.discourse-cdn.com/elastic/original/3X/2/4/2439b13c6ff782f185c859000ab3c5a9a8381b4d.png) You could see that the "api-prod" increased in volume around the 25th (when I updated Elastic, Kibana and apm-server , all from 7.5 to 7.9) ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 10, 2020, 7:41pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/8 "2020-09-10T19:41:51Z")
I don't know if this helps but: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/6/369a154c551bef0b358696a6da7cdf271d56bf9b.png) this is the time histogram with span.type, span.subtype buckets. DB spans go crazy ☹ Also, this is the agent config for that app: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4cc12262bc4fb18d4ccb42eae901e4a3081e100c.png) Thanks! ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 14, 2020, 9:05am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/9 "2020-09-14T09:05:06Z")
No smoking gun I'm afraid. It's interesting that `transaction_max_spans` has a non-default value. That makes me wonder if there's a problem with agent central config, preventing the agent from fetching the config and falling back to the default of 500. Does the "Agent remote configuration" UI indicate that the config has been applied? ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dc60d5007bf8146f95b9842c8b8a5bb5c12b5faf.png) ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 14, 2020, 3:38pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/10 "2020-09-14T15:38:16Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b4fcd9f2c9c04918055326e33cf4c4deca83072d.png) It does ☹ ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 15, 2020, 1:37am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/11 "2020-09-15T01:37:00Z")
Unless there is some coincidental change (which I suppose you have ruled out), the only other thing I can think of is that the agent has been dropping events because the server couldn't keep up. We did implement some performance optimisations between 7.5 and 7.9, so it's conceivable that the server is able to keep up with a higher load now. This seems unlikely to explain your situation though, as it should manifest in transactions being dropped as well as spans. Anyway, you could try comparing `transaction.span_count.dropped` between 7.5 and 7.9 to see if there's a change in number of dropped spans per transaction. ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 18, 2020, 6:55pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/12 "2020-09-18T18:55:01Z")
You may be getting closer with that. About 10 days ago I **changed the agent for that "api-prod" max span setting to 100** (it was taking too much space and messing up with my retention) But I checked what you suggested. So first, this is the volume of transactions during the last 30 days: ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b7038265fd938a2c262b5d7cd42efeca5196602e.png) Constant all the way.. This is Spans, same period: ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b94bc4e3459980ea3d0f16e1f6709b1a650effbc.png) Now check this out, this shows, on avg the amount of dropped spans during those 30 days ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e241b71b8b10472f7757916ddad98d9f5c619152.png) Finally, this is the amount of transactions which lost some span. ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/20c00e294115ff881b9aae86e18a8ec388fb59a6.png) This one shows that, some spans were dropped after I changed the max span setting 10 days ago. So, with all this, some things might have been happening at the same time. First, I'm not sure that the 500 max spans was true while working on Stack 7.5 Second, after upgrade to 7.9 amount of span increased but dropped spans decreased so.. performance upgrade??? Third, the 100 max span is being honoured, i'm dropping some but the amount of spans are not similar to those while on Stack 7.5 What do you think? Thanks!!! ---
**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\ **Post date:** [September 21, 2020, 10:23am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/13 "2020-09-21T10:23:37Z")
> [@Rodrigo\_Jimenez](#): > > Second, after upgrade to 7.9 amount of span increased but dropped spans decreased so.. performance upgrade??? Sorry, I think I confused things earlier by mentioning server performance and dropped spans together; there should be no relationship between server performance and the span\_count.dropped. The count is increased whenever the _agent_ decides to drop a span, due to the configured max spans being reached. If the server is unable to process events quickly enough, its queue fills up and it will stop processing new events from the agent and return a "503 queue full" error. When this occurs, some events may be lost, but `span_count.dropped` will not capture this. If the performance increased in 7.9 sufficiently, then this may be happening less often, with fewer events being lost. However, looking again at the differences you're seeing I would be surprised if increased performance alone were enough to answer for them. Do you have Stack Monitoring enabled? If so, have a look at the "Response Errors Intake API" graph and see if there was a significant change around the time you upgraded. ---
**Author:** ![Rodrigo\_Jimenez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rodrigo_jimenez/32/34722_2.png) [@Rodrigo\_Jimenez](https://discuss.elastic.co/u/Rodrigo_Jimenez)\ **Post date:** [September 21, 2020, 1:28pm UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/14 "2020-09-21T13:28:23Z")
I do have that enabled but it seems that only 7-day retention for this so, no clue about errors intake ☹ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 12, 2020, 9:28am UTC](https://discuss.elastic.co/t/update-to-apm-server-7-9-increased-my-apm-doc-count-a-lot/247791/15 "2020-10-12T09:28:33Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.