How to use less storage space for APM index?

[Jitesh (jitesh Babu)]

APM Server Version - 6.6.2:

**Google Chrome Version 75.0.3770.100 (Official Build) (64-bit)*:

Downloaded through yum install:

Fresh install

Currently we are using APM agents in around 50+ servers and all of them send the data to our APM index, which is having around 500GB of space, as the data is more its occupying around 10GB/day appx 400GB in 2 months span.
We need to store a year data, as we require them to generate graphs and metrics. Is there any alternate option to achieve them by using less storage space? Also at present itself kibana is loading slow with the amount of daya its holding in th index
if it holds a year data, what will be the performance?? Any help?

[Christian_Dahlqvist]

Given the amount of data you are ingesting and how long you want to keep it I would say you will need to scale out your cluster. If there was a way to dramatically improve storage efficiency that would most likely have been made the default.

[roncohen]

Hey @Jitesh

You can delete the span documents when you no longer need them. As long as you retain transaction documents, you will still be able to see graphs and metrics. By default, these documents go into separate indices, so you can set up separate ILM policies for each.

You can also reduce the sampling rate to collect fewer span documents in the first place. This is can be controlled by using the ELASTIC_APM_TRANSACTION_SAMPLE_RATE environment variable. It needs to be set on the APM agent side.

Hope that helps. Keep us posted.

/Ron

[Jitesh (jitesh Babu)]

Thanks so much Ron @roncohen Let me check that and revert back.

[Jitesh (jitesh Babu)]

@roncohen: We have tried this, how to validate it has gained? is there any specific place to validate?

Here is the setenv.sh

CATALINA_OPTS="${CATALINA_OPTS} -javaagent:/usr/local/apm-agent/elastic-apm-agent.jar"

CATALINA_OPTS="${CATALINA_OPTS} -Delastic.apm.service_name=XXXXX"

CATALINA_OPTS="${CATALINA_OPTS} -Delastic.apm.application_packages=XXXXX"

CATALINA_OPTS="${CATALINA_OPTS} -Delastic.apm.server_urls=XXXXXX"

CATALINA_OPTS="${CATALINA_OPTS} -Delastic.apm.transaction_sample_rate=0.2"