(Preface - I recognize this question leans more towards Terraform, but wanted to ask it here as it mainly involves ECK/Elasticsearch).
Hi All,
I was wondering if anyone has a working example of using the Terraform Kubernetes kubernetes_manifest, for deploying/updating an Elasticsearch cluster custom resource (CR).
I'm trying to deploy Elasticsearch clusters via Terraform, but have run into a bit of a snag and I'm unsure where/why the issue is occurring.
Issue:
I am able to successfully deploy the Elasticsearch CR successfully the first time, but then if I go to change any value within the CR and have Terraform update it, rather than Terraform performing an update on the CR it instead tries to do a replace, which deletes the old Elasticsearch cluster and builds a new one.
Setup:
Terraform locals:
locals {
ELASTICSEARCH_API_VERSION = "elasticsearch.k8s.elastic.co/v1"
ELASTICSEARCH_KIND = "Elasticsearch"
ELASTICSEARCH_FILE = "${path.module}/environments/${var.ENVIRONMENT_NAME}/elasticsearch.yml.tftpl"
ELASTICSEARCH_CONTENT = fileexists(local.ELASTICSEARCH_FILE) ? templatefile(local.ELASTICSEARCH_FILE, var.ELASTICSEARCH_TEMPLATE_ARGS) : "NoSettingsFileFound: true"
ELASTICSEARCH_CONFIG = yamldecode(local.ELASTICSEARCH_CONTENT)
}
Terraform Elasticsearch kubernetes_manifest:
resource "kubernetes_manifest" "elasticsearch" {
depends_on = [
...
]
computed_fields = []
manifest = {
apiVersion = local.ELASTICSEARCH_API_VERSION
kind = local.ELASTICSEARCH_KIND
metadata = {
name = var.ELASTICSEARCH_CLUSTER_NAME
namespace = var.ELASTICSEARCH_NAMESPACE
}
spec = {
version = var.ELASTICSEARCH_VERSION
updateStrategy = local.ELASTICSEARCH_CONFIG[var.ELASTICSEARCH_CLUSTER_NAME].updateStrategy
volumeClaimDeletePolicy = local.ELASTICSEARCH_CONFIG[var.ELASTICSEARCH_CLUSTER_NAME].volumeClaimDeletePolicy
secureSettings = local.ELASTICSEARCH_SECURE_SETTINGS
http = {
service = {
spec = {
selector = local.ELASTICSEARCH_CONFIG[var.ELASTICSEARCH_CLUSTER_NAME].http.service.spec.selector
}
}
tls = {
certificate = {
secretName = format("%s%s", "elasticsearch-cert", var.ENVIRONMENT_SUB_NAME != null ? "-${var.ENVIRONMENT_SUB_NAME}" : "")
}
}
}
podDisruptionBudget = {}
nodeSets = [
for node in local.ELASTICSEARCH_CONFIG[var.ELASTICSEARCH_CLUSTER_NAME].nodeSets : {
name = node.name
count = node.count
config = try(merge(node.config, local.ELASTICSEARCH_CONFIG[var.ELASTICSEARCH_CLUSTER_NAME].baseNodeConfig), local.ELASTICSEARCH_CONFIG[var.ELASTICSEARCH_CLUSTER_NAME].baseNodeConfig)
podTemplate = node.podTemplate
volumeClaimTemplates = try(node.volumeClaimTemplates, null)
}
]
}
}
field_manager {
name = "elastic-operator"
force_conflicts = true
}
}
ELASTICSEARCH_CONTENT
${CLUSTER_NAME}:
updateStrategy:
changeBudget:
maxSurge: 3
maxUnavailable: 1
volumeClaimDeletePolicy: DeleteOnScaledownOnly
http:
service:
spec:
selector:
elasticsearch.k8s.elastic.co/cluster-name: ${CLUSTER_NAME}
baseNodeConfig:
cluster:
routing:
allocation:
awareness:
attributes: k8s_node_name,zone
xpack:
security:
audit:
enabled: true
nodeSets:
- name: rack1
count: 1
podTemplate:
spec:
automountServiceAccountToken: true
containers:
- name: elasticsearch
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: "1"
memory: 1Gi
volumeMounts:
- mountPath: /usr/share/elasticsearch/config/certs
name: tls-ca
initContainers:
- command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
name: sysctl
securityContext:
privileged: true
volumes:
- name: tls-ca
secret:
secretName: %{ if ENVIRONMENT_SUB_NAME != null }tls-ca-${ENVIRONMENT_SUB_NAME}%{ else }tls-ca%{ endif }
- name: elasticsearch-data
emptyDir:
medium: ""
As mentioned this works fine for initial deployment, but when I try to make any changes (example: increase requests.cpu from 1 to 2), Terraform thinks it needs to replace rather than update.
Example:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# kubernetes_manifest.elasticsearch must be replaced
-/+ resource "kubernetes_manifest" "elasticsearch" {
~ manifest = {
~ spec = {
~ nodeSets = [
~ {
name = "rack1"
~ podTemplate = {
~ spec = {
~ containers = [
~ {
name = "elasticsearch"
~ resources = {
~ requests = {
~ cpu = "1" -> "2"
# (1 unchanged element hidden)
}
# (1 unchanged element hidden)
}
# (1 unchanged element hidden)
},
]
# (4 unchanged elements hidden)
}
}
# (3 unchanged elements hidden)
},
...
~ object = {
~ metadata = {
~ annotations = {
- "eck.k8s.elastic.co/orchestration-hints" = jsonencode(
{
- no_transient_settings = true
- service_accounts = true
}
)
- "elasticsearch.k8s.elastic.co/cluster-uuid" = "kq-OmZ83Qj259Q6sTjgA6A"
} -> (known after apply)
~ clusterName = null -> (known after apply)
~ creationTimestamp = null -> (known after apply)
~ deletionGracePeriodSeconds = null -> (known after apply)
~ deletionTimestamp = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ generateName = null -> (known after apply)
~ generation = null -> (known after apply)
~ labels = null -> (known after apply)
~ managedFields = null -> (known after apply)
name = "elasticsearch"
~ ownerReferences = null -> (known after apply)
~ resourceVersion = null -> (known after apply)
~ selfLink = null -> (known after apply)
~ uid = null -> (known after apply)
# (1 unchanged element hidden)
}
~ spec = {
~ auth = {
~ fileRealm = null -> (known after apply)
~ roles = null -> (known after apply)
}
~ http = {
~ service = {
~ metadata = {
~ annotations = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ labels = null -> (known after apply)
~ name = null -> (known after apply)
~ namespace = null -> (known after apply)
}
~ spec = {
~ allocateLoadBalancerNodePorts = null -> (known after apply)
~ clusterIP = null -> (known after apply)
~ clusterIPs = null -> (known after apply)
~ externalIPs = null -> (known after apply)
~ externalName = null -> (known after apply)
~ externalTrafficPolicy = null -> (known after apply)
~ healthCheckNodePort = null -> (known after apply)
~ internalTrafficPolicy = null -> (known after apply)
~ ipFamilies = null -> (known after apply)
~ ipFamilyPolicy = null -> (known after apply)
~ loadBalancerClass = null -> (known after apply)
~ loadBalancerIP = null -> (known after apply)
~ loadBalancerSourceRanges = null -> (known after apply)
~ ports = null -> (known after apply)
~ publishNotReadyAddresses = null -> (known after apply)
~ sessionAffinity = null -> (known after apply)
~ sessionAffinityConfig = {
~ clientIP = {
~ timeoutSeconds = null -> (known after apply)
}
}
~ type = null -> (known after apply)
# (1 unchanged element hidden)
}
}
~ tls = {
~ selfSignedCertificate = {
~ disabled = null -> (known after apply)
~ subjectAltNames = null -> (known after apply)
}
# (1 unchanged element hidden)
}
}
~ image = null -> (known after apply)
~ monitoring = {
~ logs = {
~ elasticsearchRefs = null -> (known after apply)
}
~ metrics = {
~ elasticsearchRefs = null -> (known after apply)
}
}
~ nodeSets = [
~ {
name = "rack1"
~ podTemplate = {
~ spec = {
~ containers = [
~ {
name = "elasticsearch"
~ resources = {
~ requests = {
~ cpu = "1" -> "2"
# (1 unchanged element hidden)
}
# (1 unchanged element hidden)
}
# (1 unchanged element hidden)
},
]
# (4 unchanged elements hidden)
}
}
~ volumeClaimTemplates = [
~ {
~ apiVersion = null -> (known after apply)
~ kind = null -> (known after apply)
~ metadata = {
~ annotations = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ labels = null -> (known after apply)
name = "elasticsearch-data"
~ namespace = null -> (known after apply)
}
~ spec = {
~ dataSource = {
~ apiGroup = null -> (known after apply)
~ kind = null -> (known after apply)
~ name = null -> (known after apply)
}
~ dataSourceRef = {
~ apiGroup = null -> (known after apply)
~ kind = null -> (known after apply)
~ name = null -> (known after apply)
}
~ resources = {
~ limits = null -> (known after apply)
# (1 unchanged element hidden)
}
~ selector = {
~ matchExpressions = null -> (known after apply)
~ matchLabels = null -> (known after apply)
}
~ volumeMode = null -> (known after apply)
~ volumeName = null -> (known after apply)
# (2 unchanged elements hidden)
}
~ status = {
~ accessModes = null -> (known after apply)
~ allocatedResources = null -> (known after apply)
~ capacity = null -> (known after apply)
~ conditions = null -> (known after apply)
~ phase = null -> (known after apply)
~ resizeStatus = null -> (known after apply)
}
},
]
# (2 unchanged elements hidden)
},
~ {
name = "rack2"
~ volumeClaimTemplates = [
~ {
~ apiVersion = null -> (known after apply)
~ kind = null -> (known after apply)
~ metadata = {
~ annotations = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ labels = null -> (known after apply)
name = "elasticsearch-data"
~ namespace = null -> (known after apply)
}
~ spec = {
~ dataSource = {
~ apiGroup = null -> (known after apply)
~ kind = null -> (known after apply)
~ name = null -> (known after apply)
}
~ dataSourceRef = {
~ apiGroup = null -> (known after apply)
~ kind = null -> (known after apply)
~ name = null -> (known after apply)
}
~ resources = {
~ limits = null -> (known after apply)
# (1 unchanged element hidden)
}
~ selector = {
~ matchExpressions = null -> (known after apply)
~ matchLabels = null -> (known after apply)
}
~ volumeMode = null -> (known after apply)
~ volumeName = null -> (known after apply)
# (2 unchanged elements hidden)
}
~ status = {
~ accessModes = null -> (known after apply)
~ allocatedResources = null -> (known after apply)
~ capacity = null -> (known after apply)
~ conditions = null -> (known after apply)
~ phase = null -> (known after apply)
~ resizeStatus = null -> (known after apply)
}
},
]
# (3 unchanged elements hidden)
},
~ {
name = "rack5"
~ volumeClaimTemplates = [
~ {
~ apiVersion = null -> (known after apply)
~ kind = null -> (known after apply)
~ metadata = {
~ annotations = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ labels = null -> (known after apply)
name = "elasticsearch-data"
~ namespace = null -> (known after apply)
}
~ spec = {
~ dataSource = {
~ apiGroup = null -> (known after apply)
~ kind = null -> (known after apply)
~ name = null -> (known after apply)
}
~ dataSourceRef = {
~ apiGroup = null -> (known after apply)
~ kind = null -> (known after apply)
~ name = null -> (known after apply)
}
~ resources = {
~ limits = null -> (known after apply)
# (1 unchanged element hidden)
}
~ selector = {
~ matchExpressions = null -> (known after apply)
~ matchLabels = null -> (known after apply)
}
~ volumeMode = null -> (known after apply)
~ volumeName = null -> (known after apply)
# (2 unchanged elements hidden)
}
~ status = {
~ accessModes = null -> (known after apply)
~ allocatedResources = null -> (known after apply)
~ capacity = null -> (known after apply)
~ conditions = null -> (known after apply)
~ phase = null -> (known after apply)
~ resizeStatus = null -> (known after apply)
}
},
]
# (3 unchanged elements hidden)
},
]
~ podDisruptionBudget = {
~ metadata = {
~ annotations = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ labels = null -> (known after apply)
~ name = null -> (known after apply)
~ namespace = null -> (known after apply)
}
~ spec = {
~ maxUnavailable = null -> (known after apply)
~ minAvailable = null -> (known after apply)
~ selector = {
~ matchExpressions = null -> (known after apply)
~ matchLabels = null -> (known after apply)
}
}
}
~ remoteClusters = null -> (known after apply)
~ revisionHistoryLimit = null -> (known after apply)
~ secureSettings = [
~ {
~ entries = null -> (known after apply)
# (1 unchanged element hidden)
},
~ {
~ entries = null -> (known after apply)
# (1 unchanged element hidden)
},
~ {
~ entries = null -> (known after apply)
# (1 unchanged element hidden)
},
]
~ serviceAccountName = null -> (known after apply)
~ transport = {
~ service = {
~ metadata = {
~ annotations = null -> (known after apply)
~ finalizers = null -> (known after apply)
~ labels = null -> (known after apply)
~ name = null -> (known after apply)
~ namespace = null -> (known after apply)
}
~ spec = {
~ allocateLoadBalancerNodePorts = null -> (known after apply)
~ clusterIP = null -> (known after apply)
~ clusterIPs = null -> (known after apply)
~ externalIPs = null -> (known after apply)
~ externalName = null -> (known after apply)
~ externalTrafficPolicy = null -> (known after apply)
~ healthCheckNodePort = null -> (known after apply)
~ internalTrafficPolicy = null -> (known after apply)
~ ipFamilies = null -> (known after apply)
~ ipFamilyPolicy = null -> (known after apply)
~ loadBalancerClass = null -> (known after apply)
~ loadBalancerIP = null -> (known after apply)
~ loadBalancerSourceRanges = null -> (known after apply)
~ ports = null -> (known after apply)
~ publishNotReadyAddresses = null -> (known after apply)
~ selector = null -> (known after apply)
~ sessionAffinity = null -> (known after apply)
~ sessionAffinityConfig = {
~ clientIP = {
~ timeoutSeconds = null -> (known after apply)
}
}
~ type = null -> (known after apply)
}
}
~ tls = {
~ certificate = {
~ secretName = null -> (known after apply)
}
~ otherNameSuffix = null -> (known after apply)
~ subjectAltNames = null -> (known after apply)
}
}
# (3 unchanged elements hidden)
}
# (2 unchanged elements hidden)
}
# (1 unchanged attribute hidden)
# (1 unchanged block hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.