We have a large amount of transactional data coming in. Each transactions has a transaction identified (i.e. ti) that is unique to each transaction. This looks like this:
ti : "144e647b-3b72-4206-9d72-3fa18ea4896d"
we have to convert this to a not_analyzed string due to ES stripping out the "-", and mapped this to a new field, ti.raw. (we keep the original ti, just in case).
We can now perform a "distinct count" based on the ti.raw (wasn't possible on the ti itself).
However, ES is now loading the ti.raw into the fielddata, using a ton of heap memory. We are essentially maxed out at 75% memory usage all day long.
If we were to set the ti.raw to doc_value in the mappings, I presume this would have a few effects:
- Disk size would bloom significantly (every single entry in our logs has a ti.raw)
- Memory usage would drop significantly
- Disk I/O would rise up significantly (but we are using SSD's, so I don't think it will be a huge issue)
- Searches using ti.raw for a distinct count would take slightly longer
The current entry looks like this:
and the proposed change would add:
We utilize ti.raw significantly, so am wondering if:
a) This is the proper way to add this
b) How much storage this would utilize? (we are logging around 60m transactions a day)
c) Would the speed be significantly worse?
d) This would actually reduce our memory usage
-6x nodes (4x CPUs, 61gb RAM, 512gb SSD drives)
-3x master nodes
-running on AWS
-Current CPU utilization varies between 15-80% (depending on what searches are being run).
-Memory is always near 60-78%
-Read IOPS is between 0 to 10, spiking to 300 IOPS when a heavy search is run.
-Write IOPs is between 0-125, spiking to 125 when large data is gathered and pushed in bulk to the cluster. Normally not over 50 IOPS
-Overall search performance is acceptable at this time unless dashboards and scheduled searches are run at the same time (causes too much CPU usage and too many search threads queued up - something we are aware of, as the dashboards are running many, many searches all at once).
Thanks in advance!!!