Docker Hard Disk Image File Being Too Large

[Ethan777100 (Ethan Chia)]

Its now killing my computer space. I'm in the midst of ingesting 537 csv files (total 10.7GB) into elasticsearch.

C:\Users\ethan\AppData\Local\Docker\wsl\data

I'm not sure why it became so big.

[Ethan777100 (Ethan Chia)]

TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 3 2 3.057GB 766.5MB (25%)
Containers 3 2 1.32GB 0B (0%)
Local Volumes 3 3 2.02GB 0B (0%)
Build Cache 0 0 0B 0B

My "C:\Users\ethan\AppData\Local\Docker\wsl\data\ext4.vhdx" being 25.6GB huge when I'm merely ingesting 492MB worth of csv files?

I checked my configs in logstash.yml and logstash.conf.

I bind mounted these files to logstash to be piped in.

I have tweaked things. Now I am only ingested 1 month worth of csv files to investigate.

I did docker system df and i see that my Local Volumes are growing in sizze. Nothing can be relcaimed in space.

Within logstash, I can confirm bind mounts to my csv files.

However, I searched within the ES file directory. Can't find anything big where my datalake resides.

So from 8.5GB -> 14.6GB and counting.

========================================

I have tried shutting down docker and killing off vmmem.wsl so that I can use diskpart to compact vdisk. I could cut the file size fo 8.5GB.

However when docker restarts, the file grows again.

THis is not good. Im not sure what else to troubleshoot. Pls help me.

[Ethan777100 (Ethan Chia)]

At 1 million rows ingested, the disk increased a little slower from 10.5 GB -> 13.5 GB

I did some improvements to streamline how the mapping is done on the columns.

Disk is growing still...but it does feel growing a little slower.

Not sure how much more can I improve on things to make the Disk grow minimally.

Now I am only ingesting 30 days of csv file. Each day about 137000 rows of data.

Eventually I want to ingest 18 months worth of csv files of 137000 rows of Data.

I only have about 200GB of free space to work with. I dont want to burst my file space budget.

I might even, have more csv files to ingest in future - of other data sets.

PS: My files ingestion stopped midway. But the disk is still growing now at 25 GB! This is astonishing!