Hi
I am trying to move the SQL database into elasticsearch. Successfully flattened the data from SQL into elasticsearch.
These is the details of the index which was created from SQL to elastic Doc'sCount : 49201833, Storage Size : 13.7gb.
This will be increasing. Our backend is in python. Whenever we try to pull the data through scroll API all my machine RAM is getting used and our flask is getting restarted. Below is the machine which we are using. What should be our ideal machine configuration for that data size?
H/W path Device Class Description
system t3a.2xlarge
/0 bus Motherboard
/0/0 memory 64KiB BIOS
/0/4 processor AMD EPYC 7571
/0/1 memory 31GiB System memory
/0/100 bridge 440FX - 82441FX PMC [Natoma]
/0/100/1 bridge 82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.3 generic 82371AB/EB/MB PIIX4 ACPI
/0/100/3 display Amazon.com, Inc.
/0/100/4 storage Amazon.com, Inc.
/0/100/4/0 /dev/nvme0 storage Amazon Elastic Block Store
/0/100/4/0/1 /dev/nvme0n1 disk 429GB NVMe namespace
/0/100/4/0/1/1 /dev/nvme0n1p1 volume 399GiB EXT4 volume
/0/100/5 eth0 network Elastic Network Adapter (ENA)