Fields count in elasticsearch indice

Elastic Stack Beats
1

Hi.
I have elasticsearch, Grafana and some beats deployed in k8s. Version is 7.6.1.
There are some fields from my configuration:

# env values
      - env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: discovery.seed_hosts
          value: es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch
        - name: cluster.initial_master_nodes
          value: es-cluster-0,es-cluster-1,es-cluster-2
        - name: ES_JAVA_OPTS
          value: -Xms7g -Xmx7g
        - name: ELASTIC_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: elasticsearch-password
        - name: http.max_header_size
          value: 20000kb
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.security.transport.ssl.enabled
          value: "true"
        - name: xpack.security.transport.ssl.verification_mode
          value: certificate
        - name: xpack.security.transport.ssl.keystore.path
          value: /usr/share/elasticsearch/config/elastic-certificates.p12
        - name: xpack.security.transport.ssl.truststore.path
          value: /usr/share/elasticsearch/config/elastic-certificates.p12
        - name: xpack.monitoring.collection.enabled
          value: "true"
        - name: xpack.monitoring.elasticsearch.collection.enabled
          value: "false"

# resources
        resources:
          limits:
            cpu: "3"
            memory: 14Gi
          requests:
            cpu: "3"
            memory: 14Gi

All works good. We have one index pattern (filebeat-*) in kibana with huge mount of fields (62000 fields)
Kibana works very slow when opening setting of this index pattern and when opening the dashboard that uses this index pattern.
So... Please, help me understand the issue. Is it "normal" to have so many fields in index pattern?

2

What version(s) of the stack are you using?

3

I wrote it in my message :slight_smile:
7.6.1

4

For the entire stack - Elasticsearch, Kibana and beats?

5

Sure

6

Ok, what modules are you loading in Beats?

7

For filebeat I am using two versions - 6.8.12 (deployed outside of k8s) and 7.6.1 (inside of k8s).
So much fields are generated in 6.8.12 version of filebeat. This filebeat uses log as input without modules.
The problem is not with beat. I am just asking about fields count.

8

Yeah but Filebeat is generating these fields, so it's a relevant area.

It's not really normal to have an index with that many fields. I don't know if this situation improved in later versions, but let me move this over to Beats and see if anyone can help.

9

Thanks! Will wait for answer

10

:metal:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.