Strange High CPU usage related to licensing

Elastic Stack Elasticsearch
1

Encountered a strange Cluster behavior on a central logging cluster where about 50 filebeats are ingesting data.

It seems that most of the CPU is used for security operations.
Here is what GET _nodes/hot_threads is saying:

::: {xxxx}{MNIchlz2RlKrZLGI7VKt9Q}{RopZnvbASKi2kTTGLNhO9g}{10.244.16.4}{10.244.16.4:9300}{cdhilmrstw}{ml.machine_memory=10737418240, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}
   Hot threads at 2021-06-22T15:50:58.684Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   59.4% (297ms out of 500ms) cpu usage by thread 'elasticsearch[xxxxx][management][T#4]'
     4/10 snapshots sharing following 31 elements
       java.base@15.0.1/sun.security.provider.ByteArrayAccess.l2bBig(ByteArrayAccess.java:449)
       java.base@15.0.1/sun.security.provider.SHA5.implDigest(SHA5.java:131)
       java.base@15.0.1/sun.security.provider.DigestBase.engineDigest(DigestBase.java:210)
       java.base@15.0.1/sun.security.provider.DigestBase.engineDigest(DigestBase.java:189)
       java.base@15.0.1/java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:680)
       java.base@15.0.1/java.security.MessageDigest.digest(MessageDigest.java:391)
       java.base@15.0.1/com.sun.crypto.provider.HmacCore.engineDoFinal(HmacCore.java:233)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:580)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:623)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.deriveKey(PBKDF2KeyImpl.java:194)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.<init>(PBKDF2KeyImpl.java:118)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2Core.engineGenerateSecret(PBKDF2Core.java:69)
       java.base@15.0.1/javax.crypto.SecretKeyFactory.generateSecret(SecretKeyFactory.java:340)
       org.elasticsearch.license.CryptUtils.deriveSecretKey(CryptUtils.java:187)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:135)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:115)
       org.elasticsearch.license.SelfGeneratedLicense.verify(SelfGeneratedLicense.java:72)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:569)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:559)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:350)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:56)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:25)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:99)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:166)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$$Lambda$6117/0x00000008019cd488.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@15.0.1/java.lang.Thread.run(Thread.java:832)
     2/10 snapshots sharing following 24 elements
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:580)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:623)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.deriveKey(PBKDF2KeyImpl.java:194)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.<init>(PBKDF2KeyImpl.java:118)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2Core.engineGenerateSecret(PBKDF2Core.java:69)
       java.base@15.0.1/javax.crypto.SecretKeyFactory.generateSecret(SecretKeyFactory.java:340)
       org.elasticsearch.license.CryptUtils.deriveSecretKey(CryptUtils.java:187)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:135)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:115)
       org.elasticsearch.license.SelfGeneratedLicense.verify(SelfGeneratedLicense.java:72)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:569)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:559)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:350)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:56)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:25)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:99)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:166)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$$Lambda$6117/0x00000008019cd488.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@15.0.1/java.lang.Thread.run(Thread.java:832)
     4/10 snapshots sharing following 5 elements
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@15.0.1/java.lang.Thread.run(Thread.java:832)
   
   50.7% (253.5ms out of 500ms) cpu usage by thread 'elasticsearch[centrallogging-prod-001-dci-001-master-2][management][T#2]'
     2/10 snapshots sharing following 31 elements
       java.base@15.0.1/sun.security.provider.ByteArrayAccess.l2bBig(ByteArrayAccess.java:449)
       java.base@15.0.1/sun.security.provider.SHA5.implDigest(SHA5.java:131)
       java.base@15.0.1/sun.security.provider.DigestBase.engineDigest(DigestBase.java:210)
       java.base@15.0.1/sun.security.provider.DigestBase.engineDigest(DigestBase.java:189)
       java.base@15.0.1/java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:680)
       java.base@15.0.1/java.security.MessageDigest.digest(MessageDigest.java:391)
       java.base@15.0.1/com.sun.crypto.provider.HmacCore.engineDoFinal(HmacCore.java:233)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:580)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:623)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.deriveKey(PBKDF2KeyImpl.java:194)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.<init>(PBKDF2KeyImpl.java:118)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2Core.engineGenerateSecret(PBKDF2Core.java:69)
       java.base@15.0.1/javax.crypto.SecretKeyFactory.generateSecret(SecretKeyFactory.java:340)
       org.elasticsearch.license.CryptUtils.deriveSecretKey(CryptUtils.java:187)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:135)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:115)
       org.elasticsearch.license.SelfGeneratedLicense.verify(SelfGeneratedLicense.java:72)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:569)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:559)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:350)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:56)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:25)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:99)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:166)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$$Lambda$6117/0x00000008019cd488.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@15.0.1/java.lang.Thread.run(Thread.java:832)
     2/10 snapshots sharing following 30 elements
       java.base@15.0.1/sun.security.provider.ByteArrayAccess.l2bBig(ByteArrayAccess.java:449)
       java.base@15.0.1/sun.security.provider.SHA5.implDigest(SHA5.java:131)
       java.base@15.0.1/sun.security.provider.DigestBase.engineDigest(DigestBase.java:210)
       java.base@15.0.1/java.security.MessageDigest$Delegate.engineDigest(MessageDigest.java:686)
       java.base@15.0.1/java.security.MessageDigest.digest(MessageDigest.java:418)
       java.base@15.0.1/com.sun.crypto.provider.HmacCore.engineDoFinal(HmacCore.java:240)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:580)
       java.base@15.0.1/javax.crypto.Mac.doFinal(Mac.java:623)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.deriveKey(PBKDF2KeyImpl.java:194)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2KeyImpl.<init>(PBKDF2KeyImpl.java:118)
       java.base@15.0.1/com.sun.crypto.provider.PBKDF2Core.engineGenerateSecret(PBKDF2Core.java:69)
       java.base@15.0.1/javax.crypto.SecretKeyFactory.generateSecret(SecretKeyFactory.java:340)
       org.elasticsearch.license.CryptUtils.deriveSecretKey(CryptUtils.java:187)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:135)
       org.elasticsearch.license.CryptUtils.decrypt(CryptUtils.java:115)
       org.elasticsearch.license.SelfGeneratedLicense.verify(SelfGeneratedLicense.java:72)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:569)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:559)
       org.elasticsearch.license.LicenseService.getLicense(LicenseService.java:350)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:56)
       org.elasticsearch.license.TransportGetLicenseAction.masterOperation(TransportGetLicenseAction.java:25)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:99)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:166)
       app//org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$$Lambda$6117/0x00000008019cd488.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@15.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
2

What is the output from the _cluster/stats?pretty&human API?

3

Thanks for the fast response:

GET _cluster/stats?pretty&human

{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "xxxxxxx",
  "cluster_uuid" : "furiAI8PTR6eTevlLbmasg",
  "timestamp" : 1624435147576,
  "status" : "green",
  "indices" : {
    "count" : 117,
    "shards" : {
      "total" : 638,
      "primaries" : 319,
      "replication" : 1.0,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 6,
          "avg" : 5.452991452991453
        },
        "primaries" : {
          "min" : 1,
          "max" : 3,
          "avg" : 2.7264957264957266
        },
        "replication" : {
          "min" : 1.0,
          "max" : 1.0,
          "avg" : 1.0
        }
      }
    },
    "docs" : {
      "count" : 1434145496,
      "deleted" : 5580166
    },
    "store" : {
      "size" : "948.8gb",
      "size_in_bytes" : 1018872692885,
      "reserved" : "0b",
      "reserved_in_bytes" : 0
    },
    "fielddata" : {
      "memory_size" : "74.8kb",
      "memory_size_in_bytes" : 76624,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "803.1kb",
      "memory_size_in_bytes" : 822459,
      "total_count" : 111137673,
      "hit_count" : 83703448,
      "miss_count" : 27434225,
      "cache_size" : 856,
      "cache_count" : 829273,
      "evictions" : 828417
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 5079,
      "memory" : "72.3mb",
      "memory_in_bytes" : 75871612,
      "terms_memory" : "50.6mb",
      "terms_memory_in_bytes" : 53089120,
      "stored_fields_memory" : "3.3mb",
      "stored_fields_memory_in_bytes" : 3484456,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "62.9kb",
      "norms_memory_in_bytes" : 64448,
      "points_memory" : "0b",
      "points_memory_in_bytes" : 0,
      "doc_values_memory" : "18.3mb",
      "doc_values_memory_in_bytes" : 19233588,
      "index_writer_memory" : "278.7mb",
      "index_writer_memory_in_bytes" : 292298860,
      "version_map_memory" : "296.5kb",
      "version_map_memory_in_bytes" : 303626,
      "fixed_bit_set" : "300.2mb",
      "fixed_bit_set_memory_in_bytes" : 314877104,
      "max_unsafe_auto_id_timestamp" : 1624406410302,
      "file_sizes" : { }
    },
    "mappings" : {
      "field_types" : [
        {
          "name" : "alias",
          "count" : 103,
          "index_count" : 4
        },
        {
          "name" : "binary",
          "count" : 17,
          "index_count" : 4
        },
        {
          "name" : "boolean",
          "count" : 491,
          "index_count" : 43
        },
        {
          "name" : "date",
          "count" : 740,
          "index_count" : 92
        },
        {
          "name" : "date_nanos",
          "count" : 3,
          "index_count" : 3
        },
        {
          "name" : "double",
          "count" : 157,
          "index_count" : 11
        },
        {
          "name" : "flattened",
          "count" : 37,
          "index_count" : 5
        },
        {
          "name" : "float",
          "count" : 215,
          "index_count" : 25
        },
        {
          "name" : "geo_point",
          "count" : 25,
          "index_count" : 4
        },
        {
          "name" : "geo_shape",
          "count" : 1,
          "index_count" : 1
        },
        {
          "name" : "half_float",
          "count" : 56,
          "index_count" : 14
        },
        {
          "name" : "integer",
          "count" : 240,
          "index_count" : 39
        },
        {
          "name" : "ip",
          "count" : 377,
          "index_count" : 4
        },
        {
          "name" : "keyword",
          "count" : 15136,
          "index_count" : 116
        },
        {
          "name" : "long",
          "count" : 5557,
          "index_count" : 87
        },
        {
          "name" : "nested",
          "count" : 173,
          "index_count" : 46
        },
        {
          "name" : "object",
          "count" : 4406,
          "index_count" : 89
        },
        {
          "name" : "short",
          "count" : 303,
          "index_count" : 3
        },
        {
          "name" : "text",
          "count" : 1254,
          "index_count" : 52
        }
      ]
    },
    "analysis" : {
      "char_filter_types" : [ ],
      "tokenizer_types" : [ ],
      "filter_types" : [
        {
          "name" : "pattern_capture",
          "count" : 1,
          "index_count" : 1
        }
      ],
      "analyzer_types" : [
        {
          "name" : "custom",
          "count" : 1,
          "index_count" : 1
        }
      ],
      "built_in_char_filters" : [ ],
      "built_in_tokenizers" : [
        {
          "name" : "uax_url_email",
          "count" : 1,
          "index_count" : 1
        }
      ],
      "built_in_filters" : [
        {
          "name" : "lowercase",
          "count" : 1,
          "index_count" : 1
        },
        {
          "name" : "unique",
          "count" : 1,
          "index_count" : 1
        }
      ],
      "built_in_analyzers" : [ ]
    }
  },
  "nodes" : {
    "count" : {
      "total" : 3,
      "coordinating_only" : 0,
      "data" : 3,
      "data_cold" : 3,
      "data_content" : 3,
      "data_hot" : 3,
      "data_warm" : 3,
      "ingest" : 3,
      "master" : 3,
      "ml" : 3,
      "remote_cluster_client" : 3,
      "transform" : 3,
      "voting_only" : 0
    },
    "versions" : [
      "7.10.1"
    ],
    "os" : {
      "available_processors" : 12,
      "allocated_processors" : 12,
      "names" : [
        {
          "name" : "Linux",
          "count" : 3
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "CentOS Linux 8 (Core)",
          "count" : 3
        }
      ],
      "mem" : {
        "total" : "30gb",
        "total_in_bytes" : 32212254720,
        "free" : "17.4mb",
        "free_in_bytes" : 18264064,
        "used" : "29.9gb",
        "used_in_bytes" : 32193990656,
        "free_percent" : 0,
        "used_percent" : 100
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 138
      },
      "open_file_descriptors" : {
        "min" : 2435,
        "max" : 2469,
        "avg" : 2447
      }
    },
    "jvm" : {
      "max_uptime" : "42.1d",
      "max_uptime_in_millis" : 3639619368,
      "versions" : [
        {
          "version" : "15.0.1",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "15.0.1+9",
          "vm_vendor" : "AdoptOpenJDK",
          "bundled_jdk" : true,
          "using_bundled_jdk" : true,
          "count" : 3
        }
      ],
      "mem" : {
        "heap_used" : "7.8gb",
        "heap_used_in_bytes" : 8396711424,
        "heap_max" : "24gb",
        "heap_max_in_bytes" : 25769803776
      },
      "threads" : 350
    },
    "fs" : {
      "total" : "5.1tb",
      "total_in_bytes" : 5706825719808,
      "free" : "4.2tb",
      "free_in_bytes" : 4663991361536,
      "available" : "4.2tb",
      "available_in_bytes" : 4663941029888
    },
    "plugins" : [ ],
    "network_types" : {
      "transport_types" : {
        "security4" : 3
      },
      "http_types" : {
        "security4" : 3
      }
    },
    "discovery_types" : {
      "zen" : 3
    },
    "packaging_types" : [
      {
        "flavor" : "default",
        "type" : "docker",
        "count" : 3
      }
    ],
    "ingest" : {
      "number_of_pipelines" : 5,
      "processor_stats" : {
        "date" : {
          "count" : 3839045952,
          "failed" : 3730415108,
          "current" : 0,
          "time" : "1.1h",
          "time_in_millis" : 4122422
        },
        "gsub" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time" : "0s",
          "time_in_millis" : 0
        },
        "pipeline" : {
          "count" : 1919522976,
          "failed" : 0,
          "current" : 0,
          "time" : "1.4h",
          "time_in_millis" : 5280838
        },
        "script" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time" : "0s",
          "time_in_millis" : 0
        }
      }
    }
  }
}
4

Can you upgrade? 7.13 is latest.
It may not solve your issue, but it's always good to try that as a first step given we release fixes often.

5

Thanks. Upgrade to 7.13.2 shows the same issue. May this be related to xpack somehow?

GET _cluster/stats?pretty&human

{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "xxxxxx",
  "cluster_uuid" : "furiAI8PTR6eTevlLbmasg",
  "timestamp" : 1624449152595,
  "status" : "green",
  "indices" : {
    "count" : 117,
    "shards" : {
      "total" : 634,
      "primaries" : 317,
      "replication" : 1.0,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 6,
          "avg" : 5.418803418803419
        },
        "primaries" : {
          "min" : 1,
          "max" : 3,
          "avg" : 2.7094017094017095
        },
        "replication" : {
          "min" : 1.0,
          "max" : 1.0,
          "avg" : 1.0
        }
      }
    },
    "docs" : {
      "count" : 1138131334,
      "deleted" : 4579806
    },
    "store" : {
      "size" : "774.3gb",
      "size_in_bytes" : 831467048243,
      "total_data_set_size" : "774.3gb",
      "total_data_set_size_in_bytes" : 831467048243,
      "reserved" : "0b",
      "reserved_in_bytes" : 0
    },
    "fielddata" : {
      "memory_size" : "0b",
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "0b",
      "memory_size_in_bytes" : 0,
      "total_count" : 2080,
      "hit_count" : 454,
      "miss_count" : 1626,
      "cache_size" : 0,
      "cache_count" : 164,
      "evictions" : 164
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 5172,
      "memory" : "73.9mb",
      "memory_in_bytes" : 77520504,
      "terms_memory" : "52.6mb",
      "terms_memory_in_bytes" : 55166400,
      "stored_fields_memory" : "3.1mb",
      "stored_fields_memory_in_bytes" : 3318880,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "66.6kb",
      "norms_memory_in_bytes" : 68288,
      "points_memory" : "0b",
      "points_memory_in_bytes" : 0,
      "doc_values_memory" : "18mb",
      "doc_values_memory_in_bytes" : 18966936,
      "index_writer_memory" : "212.6mb",
      "index_writer_memory_in_bytes" : 222976080,
      "version_map_memory" : "1.3kb",
      "version_map_memory_in_bytes" : 1388,
      "fixed_bit_set" : "229.1mb",
      "fixed_bit_set_memory_in_bytes" : 240298320,
      "max_unsafe_auto_id_timestamp" : 1624448752877,
      "file_sizes" : { }
    },
    "mappings" : {
      "field_types" : [
        {
          "name" : "alias",
          "count" : 69,
          "index_count" : 3,
          "script_count" : 0
        },
        {
          "name" : "boolean",
          "count" : 318,
          "index_count" : 38,
          "script_count" : 0
        },
        {
          "name" : "date",
          "count" : 487,
          "index_count" : 84,
          "script_count" : 0
        },
        {
          "name" : "date_nanos",
          "count" : 2,
          "index_count" : 2,
          "script_count" : 0
        },
        {
          "name" : "double",
          "count" : 118,
          "index_count" : 10,
          "script_count" : 0
        },
        {
          "name" : "flattened",
          "count" : 18,
          "index_count" : 2,
          "script_count" : 0
        },
        {
          "name" : "float",
          "count" : 175,
          "index_count" : 22,
          "script_count" : 0
        },
        {
          "name" : "geo_point",
          "count" : 17,
          "index_count" : 3,
          "script_count" : 0
        },
        {
          "name" : "half_float",
          "count" : 56,
          "index_count" : 14,
          "script_count" : 0
        },
        {
          "name" : "integer",
          "count" : 182,
          "index_count" : 33,
          "script_count" : 0
        },
        {
          "name" : "ip",
          "count" : 252,
          "index_count" : 3,
          "script_count" : 0
        },
        {
          "name" : "keyword",
          "count" : 10876,
          "index_count" : 108,
          "script_count" : 0
        },
        {
          "name" : "long",
          "count" : 4453,
          "index_count" : 84,
          "script_count" : 0
        },
        {
          "name" : "nested",
          "count" : 155,
          "index_count" : 41,
          "script_count" : 0
        },
        {
          "name" : "object",
          "count" : 3293,
          "index_count" : 80,
          "script_count" : 0
        },
        {
          "name" : "short",
          "count" : 202,
          "index_count" : 2,
          "script_count" : 0
        },
        {
          "name" : "text",
          "count" : 910,
          "index_count" : 46,
          "script_count" : 0
        }
      ],
      "runtime_field_types" : [ ]
    },
    "analysis" : {
      "char_filter_types" : [ ],
      "tokenizer_types" : [ ],
      "filter_types" : [ ],
      "analyzer_types" : [ ],
      "built_in_char_filters" : [ ],
      "built_in_tokenizers" : [ ],
      "built_in_filters" : [ ],
      "built_in_analyzers" : [ ]
    },
    "versions" : [
      {
        "version" : "7.8.0",
        "index_count" : 7,
        "primary_shard_count" : 11,
        "total_primary_size" : "9.7mb",
        "total_primary_bytes" : 10176120
      },
      {
        "version" : "7.10.1",
        "index_count" : 109,
        "primary_shard_count" : 305,
        "total_primary_size" : "388.2gb",
        "total_primary_bytes" : 416838091582
      },
      {
        "version" : "7.13.2",
        "index_count" : 1,
        "primary_shard_count" : 1,
        "total_primary_size" : "19.1kb",
        "total_primary_bytes" : 19578
      }
    ]
  },
  "nodes" : {
    "count" : {
      "total" : 3,
      "coordinating_only" : 0,
      "data" : 3,
      "data_cold" : 3,
      "data_content" : 3,
      "data_frozen" : 3,
      "data_hot" : 3,
      "data_warm" : 3,
      "ingest" : 3,
      "master" : 3,
      "ml" : 3,
      "remote_cluster_client" : 3,
      "transform" : 3,
      "voting_only" : 0
    },
    "versions" : [
      "7.13.2"
    ],
    "os" : {
      "available_processors" : 12,
      "allocated_processors" : 12,
      "names" : [
        {
          "name" : "Linux",
          "count" : 3
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "CentOS Linux 8",
          "count" : 3
        }
      ],
      "architectures" : [
        {
          "arch" : "amd64",
          "count" : 3
        }
      ],
      "mem" : {
        "total" : "30gb",
        "total_in_bytes" : 32212254720,
        "free" : "21.1mb",
        "free_in_bytes" : 22224896,
        "used" : "29.9gb",
        "used_in_bytes" : 32190029824,
        "free_percent" : 0,
        "used_percent" : 100
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 136
      },
      "open_file_descriptors" : {
        "min" : 2280,
        "max" : 2520,
        "avg" : 2400
      }
    },
    "jvm" : {
      "max_uptime" : "22.2m",
      "max_uptime_in_millis" : 1334385,
      "versions" : [
        {
          "version" : "16",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "16+36",
          "vm_vendor" : "AdoptOpenJDK",
          "bundled_jdk" : true,
          "using_bundled_jdk" : true,
          "count" : 3
        }
      ],
      "mem" : {
        "heap_used" : "12.7gb",
        "heap_used_in_bytes" : 13739855352,
        "heap_max" : "24gb",
        "heap_max_in_bytes" : 25769803776
      },
      "threads" : 223
    },
    "fs" : {
      "total" : "5.1tb",
      "total_in_bytes" : 5706825719808,
      "free" : "4.4tb",
      "free_in_bytes" : 4871576879104,
      "available" : "4.4tb",
      "available_in_bytes" : 4871526547456
    },
    "plugins" : [ ],
    "network_types" : {
      "transport_types" : {
        "security4" : 3
      },
      "http_types" : {
        "security4" : 3
      }
    },
    "discovery_types" : {
      "zen" : 3
    },
    "packaging_types" : [
      {
        "flavor" : "default",
        "type" : "docker",
        "count" : 3
      }
    ],
    "ingest" : {
      "number_of_pipelines" : 5,
      "processor_stats" : {
        "date" : {
          "count" : 995550,
          "failed" : 969049,
          "current" : 0,
          "time" : "13.8s",
          "time_in_millis" : 13847
        },
        "gsub" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time" : "0s",
          "time_in_millis" : 0
        },
        "pipeline" : {
          "count" : 497775,
          "failed" : 0,
          "current" : 0,
          "time" : "17.3s",
          "time_in_millis" : 17376
        },
        "script" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time" : "0s",
          "time_in_millis" : 0
        }
      }
    }
  }
}
6

Thanks, there's nothing unusual I can see here, other than the hot threads.
I don't suppose you could post some of the logs from the current master?

7

Hi,

here the logs:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.