Problem when changing resources

[ccaillet (Christophe Caillet)]

Hi all,

When I change resources like cpu and memory on limits ECK does nothing, the only way for me to have the sts to be update with my changes is to edit the sts manifest with kubectl edit for applying new values to the sts

Exemple

I want to add the following :

resources:
  requests:
    cpu: 2
    memory: 2Gi
  limits:
    cpu: 4
    memory: 4Gi

On the spec for podTemplate instead of

resources:
  limits:
    memory: 2Gi

Am I the only guy with this behaviour ?

Thanks by advance for your replies.

Regards
Christophe

[michael.morello]

Could you:

• Provide the full Elasticsearch manifest, before and after the change.
• Get the current Elasticsearch Status subresource value with kubectl get es/quickstart -o jsonpath='{ .status }' -n my-namespace
• Check if there is any error in the operator logs

Thanks

[ccaillet (Christophe Caillet)]

Hi Michael,

Here is the manifest used to create the cluster, you see here all the resources requested and limited but on the first try only memory limit have been set.

The result you wanted

kubectl get elasticsearch -A

NAMESPACE NAME HEALTH NODES VERSION PHASE AGE

cdn-bigdata sophie-int green 19 8.6.0 Ready 24h

cdn-bigdata sophie-mon green 3 8.6.0 Ready 24h

kubectl get es/sophie-int -o jsonpath='{ .status }' -n cdn-bigdata

{
	"availableNodes": 19,
	"conditions": [{
		"lastTransitionTime": "2023-01-18T08:45:43Z",
		"status": "True",
		"type": "ReconciliationComplete"
	}, {
		"lastTransitionTime": "2023-01-17T13:08:36Z",
		"message": "All nodes are running version 8.6.0",
		"status": "True",
		"type": "RunningDesiredVersion"
	}, {
		"lastTransitionTime": "2023-01-17T13:09:55Z",
		"message": "Service cdn-bigdata/sophie-int-es-internal-http has endpoints",
		"status": "True",
		"type": "ElasticsearchIsReachable"
	}, {
		"lastTransitionTime": "2023-01-17T13:08:38Z",
		"message": "Cannot get compute and storage resources from Elasticsearch resource generation 2: cannot compute resources for NodeSet \"master\": memory request and limit do not have the same value",
		"status": "False",
		"type": "ResourcesAwareManagement"
	}],
	"health": "green",
	"inProgressOperations": {
		"downscale": {
			"lastUpdatedTime": "2023-01-17T13:08:36Z"
		},
		"upgrade": {
			"lastUpdatedTime": "2023-01-18T08:45:03Z"
		},
		"upscale": {
			"lastUpdatedTime": "2023-01-17T13:08:36Z"
		}
	},
	"monitoringAssociationStatus": {
		"cdn-bigdata/sophie-mon": "Established"
	},
	"observedGeneration": 2,
	"phase": "Ready",
	"version": "8.6.0"
}

Correct me if I’m wrong but why requests and limits need to be the same ? for me it’s a non-sense because requests and limits doesn’t have the same function.

(Attachment sophie-int-es.yaml is missing)

[ccaillet (Christophe Caillet)]

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: sophie-int
  namespace: cdn-bigdata
spec:
  version: 8.6.0
  http:
    service:
      metadata:
        name: sophie-es-http
        namespace: cdn-bigdata
      spec:
        type: LoadBalancer
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: sophie-mon
        namespace: cdn-bigdata
    logs:
      elasticsearchRefs:
      - name: sophie-mon
        namespace: cdn-bigdata
  nodeSets:
  - name: master
    count: 3
    config:
      node.roles: ["master","remote_cluster_client"]
      xpack.ml.enabled: true
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 4Gi
        storageClassName: lyo0-oc9k-nfs
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              cpu: 2
              memory: 2Gi
            limits:
              cpu: 2
              memory: 4Gi
  - name: ingest
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 4Gi
        storageClassName: lyo0-oc9k-nfs
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
            - name: ES_JAVA_OPTS
              value: -Xms6g -Xmx6g
          resources:
            requests:
              cpu: 4
              memory: 6Gi
            limits:
              cpu: 4
              memory: 8Gi
    config:
      node.roles: ["ingest"]
  - name: data-hot
    count: 5
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 80Gi
        storageClassName: lyo0-oc5k-nfs
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
            - name: ES_JAVA_OPTS
              value: -Xms8g -Xmx8g
          resources:
            requests:
              cpu: 4
              memory: 4Gi
            limits:
              cpu: 12
              memory: 10Gi
    config:
      node.roles: ["data_hot","data_content","remote_cluster_client"]
  - name: data-warm
    count: 4
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 150Gi
        storageClassName: lyo0-oc5k-nfs
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
            - name: ES_JAVA_OPTS
              value: -Xms8g -Xmx8g
          resources:
            requests:
              cpu: 4
              memory: 4Gi
            limits:
              cpu: 12
              memory: 10Gi
    config:
      node.roles: ["data_warm","data_content","remote_cluster_client"]
  - name: data-cold
    count: 4
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 200Gi
        storageClassName: lyo0-oc9k-nfs
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
            - name: ES_JAVA_OPTS
              value: -Xms4g -Xmx4g
          resources:
            requests:
              cpu: 4
              memory: 4Gi
            limits:
              cpu: 6
              memory: 6Gi
    config:
      node.roles: ["data_cold","data_content","remote_cluster_client"]

Ok so email seems to be problematic ... here is the content

[michael.morello]

The status states that the resource is Reconciled. Either it is actually the case, or you applied a change but the operator is not (correctly) running. Could you please:

1. Provide the operator logs (see the command in my previous message)
2. Get a diff between the actual state and the expected state.

Also "applying new values to the sts" should not work as expected since there are controlled by the operator.

Regarding the memory requests and limits I would suggest to read this documentation.

[ccaillet (Christophe Caillet)]

HI Michael,

After a restart from scratch of the manifest all is working well and as expected. I think we can close this discussion

Regards