Hello,
I am new to being a sysadmin for ES and in this case, it is backing a Magento2 installation running on Ubuntu 18.04 LTS
Problem:
Occasionally, and I'm not sure why, ES performance begins to degrade and then crashes entirely. I've noticed GC seems to take much longer than it should in this scenario (almost a minute). Ultimately this leads to ES crash due to insufficient memory.
I have configured JVM with 24GB heap size (Xms/Xmx) which I have progressively increased as it seems to help, though my gut tells me this is NOT the solution I really need and is acting like "bandaid"
The largest index holds about 623K documents and is under 100mb in size. There are (3) indicies of a similar size. Given how small these are and how much memory I've given to JVM, I am struggling to understand why ES continues to crash. I have noticed this is occuring at higher traffic periods (e.g., multiple crawlers) and the store is currently running a sale, which also increases traffic.
I would sincerely appreciate advice on what I can change on ES or JVM to help solve this problem "the right way"
I've included what I think are the most helpful logs and outputs from ES, and obfuscated some names for privacy sake. In case anything else would be helpful, please do not hesitate to ask and I'll retrieve it right away.
From /var/log/syslog:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 21474836480 bytes for committing reserved memory.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000002c0000000, 21474836480, 0) failed; error='Not enough space' (errno=12)
The ES server is version 7.8.1 and currently holds (23) indicies with (1) node. The server has 64GB of memory and 16 cores.
Here is the output from _nodes
{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"elasticsearch","nodes":{"3WXMfDd3QAasPCKAVUtnhQ":{"name":"production.x.com","transport_address":"127.0.0.1:9300","host":"127.0.0.1","ip":"127.0.0.1","version":"7.8.1","build_flavor":"default","build_type":"deb","build_hash":"b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89","total_indexing_buffer":2576980377,"roles":["data","ingest","master","ml","remote_cluster_client","transform"],"attributes":{"ml.machine_memory":"67193634816","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"20"},"settings":{"client":{"type":"node"},"cluster":{"name":"elasticsearch","election":{"strategy":"supports_voting_only"}},"http":{"type":"security4","type.default":"netty4"},"node":{"attr":{"transform":{"node":"true"},"xpack":{"installed":"true"},"ml":{"machine_memory":"67193634816","max_open_jobs":"20"}},"name":"production.x.com","pidfile":"/var/run/elasticsearch/elasticsearch.pid"},"path":{"data":["/var/lib/elasticsearch"],"logs":"/var/log/elasticsearch","home":"/usr/share/elasticsearch"},"transport":{"type":"security4","features":{"x-pack":"true"},"type.default":"netty4"}},"os":{"refresh_interval_in_millis":1000,"name":"Linux","pretty_name":"Ubuntu 18.04.4 LTS","arch":"amd64","version":"4.15.0-109-generic","available_processors":16,"allocated_processors":16},"process":{"refresh_interval_in_millis":1000,"id":9354,"mlockall":false},"jvm":{"pid":9354,"version":"14.0.1","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"14.0.1+7","vm_vendor":"AdoptOpenJDK","bundled_jdk":true,"using_bundled_jdk":true,"start_time_in_millis":1699893581738,"mem":{"heap_init_in_bytes":25769803776,"heap_max_in_bytes":25769803776,"non_heap_init_in_bytes":7667712,"non_heap_max_in_bytes":0,"direct_max_in_bytes":0},"gc_collectors":["G1 Young Generation","G1 Old Generation"],"memory_pools":["CodeHeap 'non-nmethods'","Metaspace","CodeHeap 'profiled nmethods'","Compressed Class Space","G1 Eden Space","G1 Old Gen","G1 Survivor Space","CodeHeap 'non-profiled nmethods'"],"using_compressed_ordinary_object_pointers":"true","input_arguments":["-Xshare:auto","-Des.networkaddress.cache.ttl=60","-Des.networkaddress.cache.negative.ttl=10","-XX:+AlwaysPreTouch","-Xss1m","-Djava.awt.headless=true","-Dfile.encoding=UTF-8","-Djna.nosys=true","-XX:-OmitStackTraceInFastThrow","-XX:+ShowCodeDetailsInExceptionMessages","-Dio.netty.noUnsafe=true","-Dio.netty.noKeySetOptimization=true","-Dio.netty.recycler.maxCapacityPerThread=0","-Dio.netty.allocator.numDirectArenas=0","-Dlog4j.shutdownHookEnabled=false","-Dlog4j2.disable.jmx=true","-Djava.locale.providers=SPI,COMPAT","-Xms24g","-Xmx24g","-XX:+UseG1GC","-XX:G1ReservePercent=25","-XX:InitiatingHeapOccupancyPercent=30","-Djava.io.tmpdir=/tmp/elasticsearch-4127757315199113599","-XX:+HeapDumpOnOutOfMemoryError","-XX:HeapDumpPath=/var/lib/elasticsearch","-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log","-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m","-XX:MaxDirectMemorySize=12884901888","-Des.path.home=/usr/share/elasticsearch","-Des.path.conf=/etc/elasticsearch","-Des.distribution.flavor=default","-Des.distribution.type=deb","-Des.bundled_jdk=true"]},"thread_pool":{"watcher":{"type":"fixed","size":50,"queue_size":1000},"force_merge":{"type":"fixed","size":1,"queue_size":-1},"security-token-key":{"type":"fixed","size":1,"queue_size":1000},"ml_datafeed":{"type":"scaling","core":1,"max":512,"keep_alive":"1m","queue_size":-1},"fetch_shard_started":{"type":"scaling","core":1,"max":32,"keep_alive":"5m","queue_size":-1},"listener":{"type":"fixed","size":8,"queue_size":-1},"refresh":{"type":"scaling","core":1,"max":8,"keep_alive":"5m","queue_size":-1},"generic":{"type":"scaling","core":4,"max":128,"keep_alive":"30s","queue_size":-1},"rollup_indexing":{"type":"fixed","size":4,"queue_size":4},"warmer":{"type":"scaling","core":1,"max":5,"keep_alive":"5m","queue_size":-1},"search":{"type":"fixed_auto_queue_size","size":25,"queue_size":1000},"ccr":{"type":"fixed","size":32,"queue_size":100},"flush":{"type":"scaling","core":1,"max":5,"keep_alive":"5m","queue_size":-1},"fetch_shard_store":{"type":"scaling","core":1,"max":32,"keep_alive":"5m","queue_size":-1},"management":{"type":"scaling","core":1,"max":5,"keep_alive":"5m","queue_size":-1},"ml_utility":{"type":"scaling","core":1,"max":2048,"keep_alive":"10m","queue_size":-1},"get":{"type":"fixed","size":16,"queue_size":1000},"analyze":{"type":"fixed","size":1,"queue_size":16},"ml_job_comms":{"type":"scaling","core":4,"max":2048,"keep_alive":"1m","queue_size":-1},"transform_indexing":{"type":"fixed","size":4,"queue_size":4},"write":{"type":"fixed","size":16,"queue_size":200},"snapshot":{"type":"scaling","core":1,"max":5,"keep_alive":"5m","queue_size":-1},"search_throttled":{"type":"fixed_auto_queue_size","size":1,"queue_size":100}},"transport":{"bound_address":["127.0.0.1:9300"],"publish_address":"127.0.0.1:9300","profiles":{}},"http":{"bound_address":["127.0.0.1:9200"],"publish_address":"127.0.0.1:9200","max_content_length_in_bytes":104857600},"plugins":[{"name":"analysis-icu","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"The ICU Analysis plugin integrates the Lucene ICU module into Elasticsearch, adding ICU-related analysis components.","classname":"org.elasticsearch.plugin.analysis.icu.AnalysisICUPlugin","extended_plugins":[],"has_native_controller":false},{"name":"analysis-phonetic","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"The Phonetic Analysis plugin integrates phonetic token filter analysis with elasticsearch.","classname":"org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin","extended_plugins":[],"has_native_controller":false}],"modules":[{"name":"aggs-matrix-stats","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Adds aggregations whose input are a list of numeric fields and output includes a matrix.","classname":"org.elasticsearch.search.aggregations.matrix.MatrixAggregationPlugin","extended_plugins":[],"has_native_controller":false},{"name":"analysis-common","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Adds \"built in\" analyzers to Elasticsearch.","classname":"org.elasticsearch.analysis.common.CommonAnalysisPlugin","extended_plugins":["lang-painless"],"has_native_controller":false},{"name":"constant-keyword","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Module for the constant-keyword field type, which is a specialization of keyword for the case when all documents have the same value.","classname":"org.elasticsearch.xpack.constantkeyword.ConstantKeywordMapperPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"flattened","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Module for the flattened field type, which allows JSON objects to be flattened into a single field.","classname":"org.elasticsearch.xpack.flattened.FlattenedMapperPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"frozen-indices","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A plugin for the frozen indices functionality","classname":"org.elasticsearch.xpack.frozen.FrozenIndices","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"ingest-common","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Module for ingest processors that do not require additional security permissions or have large dependencies and resources","classname":"org.elasticsearch.ingest.common.IngestCommonPlugin","extended_plugins":["lang-painless"],"has_native_controller":false},{"name":"ingest-geoip","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database","classname":"org.elasticsearch.ingest.geoip.IngestGeoIpPlugin","extended_plugins":[],"has_native_controller":false},{"name":"ingest-user-agent","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Ingest processor that extracts information from a user agent","classname":"org.elasticsearch.ingest.useragent.IngestUserAgentPlugin","extended_plugins":[],"has_native_controller":false},{"name":"kibana","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Plugin exposing APIs for Kibana system indices","classname":"org.elasticsearch.kibana.KibanaPlugin","extended_plugins":[],"has_native_controller":false},{"name":"lang-expression","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Lucene expressions integration for Elasticsearch","classname":"org.elasticsearch.script.expression.ExpressionPlugin","extended_plugins":[],"has_native_controller":false},{"name":"lang-mustache","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Mustache scripting integration for Elasticsearch","classname":"org.elasticsearch.script.mustache.MustachePlugin","extended_plugins":[],"has_native_controller":false},{"name":"lang-painless","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"An easy, safe and fast scripting language for Elasticsearch","classname":"org.elasticsearch.painless.PainlessPlugin","extended_plugins":[],"has_native_controller":false},{"name":"mapper-extras","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Adds advanced field mappers","classname":"org.elasticsearch.index.mapper.MapperExtrasPlugin","extended_plugins":[],"has_native_controller":false},{"name":"parent-join","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"This module adds the support parent-child queries and aggregations","classname":"org.elasticsearch.join.ParentJoinPlugin","extended_plugins":[],"has_native_controller":false},{"name":"percolator","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Percolator module adds capability to index queries and query these queries by specifying documents","classname":"org.elasticsearch.percolator.PercolatorPlugin","extended_plugins":[],"has_native_controller":false},{"name":"rank-eval","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"The Rank Eval module adds APIs to evaluate ranking quality.","classname":"org.elasticsearch.index.rankeval.RankEvalPlugin","extended_plugins":[],"has_native_controller":false},{"name":"reindex","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"The Reindex module adds APIs to reindex from one index to another or update documents in place.","classname":"org.elasticsearch.index.reindex.ReindexPlugin","extended_plugins":[],"has_native_controller":false},{"name":"repository-url","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Module for URL repository","classname":"org.elasticsearch.plugin.repository.url.URLRepositoryPlugin","extended_plugins":[],"has_native_controller":false},{"name":"search-business-rules","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A plugin for applying business rules to search result rankings","classname":"org.elasticsearch.xpack.searchbusinessrules.SearchBusinessRules","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"searchable-snapshots","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A plugin for the searchable snapshots functionality","classname":"org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"spatial","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A plugin for Basic Spatial features","classname":"org.elasticsearch.xpack.spatial.SpatialPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"systemd","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Integrates Elasticsearch with systemd","classname":"org.elasticsearch.systemd.SystemdPlugin","extended_plugins":[],"has_native_controller":false},{"name":"tasks","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Supports the Tasks API","classname":"org.elasticsearch.tasksplugin.TasksPlugin","extended_plugins":[],"has_native_controller":false},{"name":"transform","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A plugin to transform data","classname":"org.elasticsearch.xpack.transform.Transform","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"transport-netty4","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Netty 4 based transport implementation","classname":"org.elasticsearch.transport.Netty4Plugin","extended_plugins":[],"has_native_controller":false},{"name":"vectors","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A plugin for working with vectors","classname":"org.elasticsearch.xpack.vectors.Vectors","extended_plugins":["x-pack-core","lang-painless"],"has_native_controller":false},{"name":"x-pack-analytics","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Analytics","classname":"org.elasticsearch.xpack.analytics.AnalyticsPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-async-search","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"A module which allows to track the progress of a search asynchronously.","classname":"org.elasticsearch.xpack.search.AsyncSearch","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-autoscaling","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Autoscaling","classname":"org.elasticsearch.xpack.autoscaling.Autoscaling","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-ccr","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - CCR","classname":"org.elasticsearch.xpack.ccr.Ccr","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-core","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Core","classname":"org.elasticsearch.xpack.core.XPackPlugin","extended_plugins":[],"has_native_controller":false},{"name":"x-pack-deprecation","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Deprecation","classname":"org.elasticsearch.xpack.deprecation.Deprecation","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-enrich","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Enrich","classname":"org.elasticsearch.xpack.enrich.EnrichPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-eql","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"The Elasticsearch plugin that powers EQL for Elasticsearch","classname":"org.elasticsearch.xpack.eql.plugin.EqlPlugin","extended_plugins":["x-pack-ql","lang-painless"],"has_native_controller":false},{"name":"x-pack-graph","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Graph","classname":"org.elasticsearch.xpack.graph.Graph","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-identity-provider","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Identity Provider","classname":"org.elasticsearch.xpack.idp.IdentityProviderPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-ilm","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Index Lifecycle Management","classname":"org.elasticsearch.xpack.ilm.IndexLifecycle","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-logstash","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Logstash","classname":"org.elasticsearch.xpack.logstash.Logstash","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-ml","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Machine Learning","classname":"org.elasticsearch.xpack.ml.MachineLearning","extended_plugins":["x-pack-core","lang-painless"],"has_native_controller":true},{"name":"x-pack-monitoring","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Monitoring","classname":"org.elasticsearch.xpack.monitoring.Monitoring","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-ql","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch infrastructure plugin for EQL and SQL for Elasticsearch","classname":"org.elasticsearch.xpack.ql.plugin.QlPlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-rollup","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Rollup","classname":"org.elasticsearch.xpack.rollup.Rollup","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-security","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Security","classname":"org.elasticsearch.xpack.security.Security","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-sql","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"The Elasticsearch plugin that powers SQL for Elasticsearch","classname":"org.elasticsearch.xpack.sql.plugin.SqlPlugin","extended_plugins":["x-pack-ql","lang-painless"],"has_native_controller":false},{"name":"x-pack-voting-only-node","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Voting-only node","classname":"org.elasticsearch.cluster.coordination.VotingOnlyNodePlugin","extended_plugins":["x-pack-core"],"has_native_controller":false},{"name":"x-pack-watcher","version":"7.8.1","elasticsearch_version":"7.8.1","java_version":"1.8","description":"Elasticsearch Expanded Pack Plugin - Watcher","classname":"org.elasticsearch.xpack.watcher.Watcher","extended_plugins":["x-pack-core"],"has_native_controller":false}],"ingest":{"processors":[{"type":"append"},{"type":"bytes"},{"type":"circle"},{"type":"convert"},{"type":"csv"},{"type":"date"},{"type":"date_index_name"},{"type":"dissect"},{"type":"dot_expander"},{"type":"drop"},{"type":"enrich"},{"type":"fail"},{"type":"foreach"},{"type":"geoip"},{"type":"grok"},{"type":"gsub"},{"type":"html_strip"},{"type":"inference"},{"type":"join"},{"type":"json"},{"type":"kv"},{"type":"lowercase"},{"type":"pipeline"},{"type":"remove"},{"type":"rename"},{"type":"script"},{"type":"set"},{"type":"set_security_user"},{"type":"sort"},{"type":"split"},{"type":"trim"},{"type":"uppercase"},{"type":"urldecode"},{"type":"user_agent"}]}}}}
And here is the output from _cat/indices
green open magento2_b2b_english_catalog_category_20231107_142556 Q88ZtUZ3QamTJ9dW3bMYZA 1 0 430 4 446.5kb 446.5kb
green open magento2_rs_english_catalog_category_20231107_142557 EUuBhbd0Rjef9as_jN2Kyw 1 0 434 4 447.3kb 447.3kb
green open magento2_cadretail_catalog_product_20231113_163843 4Pxm3E4GQ4ytWb0CFvoTDQ 1 0 623105 0 98.3mb 98.3mb
green open magento2_ll_english_catalog_product_20231113_163840 zKoXXg7LQG2J-wfBstGlaw 1 0 19271 0 2.9mb 2.9mb
green open magento2_cadls_catalog_category_20231107_142558 cvbrdWpQQ1iYQan3xBYChA 1 0 97 0 111.9kb 111.9kb
green open magento2_b2b_english_thesaurus_20231107_142559 Ez_2TEVUTFuHfuZ7BY3nOg 1 0 0 0 208b 208b
green open magento2_default_catalog_category_20201029_043149 X8JvGryGS6y08jddEPRqeg 1 0 1 0 7.5kb 7.5kb
green open magento2_default_thesaurus_20201029_043150 geRfxfvdSc20t4-AGbNvfQ 1 0 0 0 208b 208b
green open magento2_usdretail_english_thesaurus_20231107_142559 itkZZNPiTN2N4LBtH0M8mw 1 0 0 0 208b 208b
green open magento2_cadretail_thesaurus_20231107_142559 qjuqZFLwRJWupLLXm4s8XA 1 0 0 0 208b 208b
green open magento2_cadls_thesaurus_20231107_142600 zYfWvGQBQDWYHMTYLYefAQ 1 0 0 0 208b 208b
green open magento2_usdretail_english_catalog_product_20231113_163715 IjIzy_3ZQOKReChckic9hA 1 0 623094 0 97.5mb 97.5mb
green open magento2_cadretail_catalog_category_20231107_142557 -tCrpFQ8QD265FT41tvxqQ 1 0 434 4 442.7kb 442.7kb
green open magento2_ll_english_thesaurus_20231107_142559 CLpx7cp_RVewBvOieRqmrg 1 0 0 0 208b 208b
green open magento2_cadretail_catalog_product_20230123_182529 Mbv_HH5WS8GShnUdGGHpqg 1 0 470830 0 64.1mb 64.1mb
yellow open magento2_product_1_v1 GllIqz4wSpqvvo98AwV8TQ 1 1 0 0 3kb 3kb
green open magento2_usdretail_english_catalog_category_20231107_142555 W7eqp3lAQIKIDqAx2C3L5Q 1 0 433 4 445.8kb 445.8kb
green open magento2_ll_english_catalog_category_20231107_142556 gGcgC4hLRYCV4K82xrlbZg 1 0 87 0 103.9kb 103.9kb
green open magento2_rs_english_thesaurus_20231107_142559 GAp4jRwfSFG9eNigSbqA5g 1 0 0 0 208b 208b
green open magento2_cadls_catalog_product_20231113_163927 6g81X7MVRmuCCV93acBKgQ 1 0 19276 0 2.9mb 2.9mb
green open magento2_rs_english_catalog_product_20231113_163843 oi4bn9DUTaqcy4b5ZLRdHg 1 0 564 0 110.8kb 110.8kb
green open magento2_default_catalog_product_20200907_204623 RPaZWE39REarCfAvwTMQ0A 1 0 0 0 3kb 3kb
green open magento2_b2b_english_catalog_product_20231113_163755 DRbBRIi8RPGP0eX7dJEQxA 1 0 623707 0 104.1mb 104.1mb
And output from meminfo
cat /proc/meminfo
MemTotal: 65618784 kB
MemFree: 12215372 kB
MemAvailable: 31565068 kB
Buffers: 673724 kB
Cached: 18785772 kB
SwapCached: 44480 kB
Active: 37879052 kB
Inactive: 14158032 kB
Active(anon): 32447500 kB
Inactive(anon): 239580 kB
Active(file): 5431552 kB
Inactive(file): 13918452 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16001016 kB
SwapFree: 10752264 kB
Dirty: 32172 kB
Writeback: 0 kB
AnonPages: 32542120 kB
Mapped: 488612 kB
Shmem: 111980 kB
Slab: 952632 kB
SReclaimable: 721816 kB
SUnreclaim: 230816 kB
KernelStack: 17216 kB
PageTables: 126064 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 48810408 kB
Committed_AS: 35019144 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12085480 kB
DirectMap2M: 53620736 kB
DirectMap1G: 3145728 kB
A million "thank yous" to any kind soul that can help me solve this. To futher complicate the matter, my wife is now (3) days past due on our first child and I would really really really love to have confidence in this server as I step away for new parent leave 
Thank you ES community!