I am running Elasticsearch 7.17.0 in EKS. Under /app/management/stack/upgrade_assistant/es_deprecations I see following
I updated config to
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: es-master
namespace: kube-logging
spec:
serviceName: elasticsearch
replicas: 3
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
restartPolicy: Always
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:8.0.1
resources:
limits:
cpu: 2000m
memory: 4G
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
readinessProbe:
httpGet:
path: /_cluster/health?local=true
port: 9200
initialDelaySeconds: 5
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
env:
- name: cluster.name
value: prod-eks-logs
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: node.name
value: "$(NODE_NAME).elasticsearch"
# - name: discovery.zen.ping.unicast.hosts
# value: "es-master-0.elasticsearch,es-master-1.elasticsearch,es-master-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "es-master-0.elasticsearch,es-master-1.elasticsearch,es-master-2.elasticsearch"
# - name: discovery.zen.minimum_master_nodes
# value: "2"
- name: xpack.security.enabled
value: "false"
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: aws-gp2
resources:
requests:
storage: 300Gi
With above changes I see following message in the Kibana UI
{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}
And following in the Elasticsearch container
{"@timestamp":"2022-03-07T17:31:49.385Z", "log.level": "WARN", "message":"path: /.monitoring-beats-7-2022.03.07/_bulk, params: {index=.monitoring-beats-7-2022.03.07}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][write][T#1]","log.logger":"rest.suppressed","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs","error.type":"org.elasticsearch.cluster.block.ClusterBlockException","error.message":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];","error.stack_trace":"org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];\n\tat org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:639)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:492)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: org.elasticsearch.discovery.MasterNotDiscoveredException\n\t\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:303)\n\t\tat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\t\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\t\tat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)\n\t\t... 4 more\n"}
{"@timestamp":"2022-03-07T17:31:49.541Z", "log.level": "WARN", "message":"path: /.monitoring-beats-7-2022.03.07/_bulk, params: {index=.monitoring-beats-7-2022.03.07}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][write][T#2]","log.logger":"rest.suppressed","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs","error.type":"org.elasticsearch.cluster.block.ClusterBlockException","error.message":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];","error.stack_trace":"org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized, SERVICE_UNAVAILABLE/2/no master];\n\tat org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:639)\n\tat org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:492)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n\tSuppressed: org.elasticsearch.discovery.MasterNotDiscoveredException\n\t\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:303)\n\t\tat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\t\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\t\tat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)\n\t\t... 4 more\n"}
{"@timestamp":"2022-03-07T17:31:50.147Z", "log.level": "WARN", "message":"address [127.0.0.1:9303], node [null], requesting [false] connection failed: [][127.0.0.1:9303] connect_exception: Connection refused: /127.0.0.1:9303: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][generic][T#3]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs"}
{"@timestamp":"2022-03-07T17:31:50.147Z", "log.level": "WARN", "message":"address [127.0.0.1:9304], node [null], requesting [false] connection failed: [][127.0.0.1:9304] connect_exception: Connection refused: /127.0.0.1:9304: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][generic][T#1]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs"}
{"@timestamp":"2022-03-07T17:31:50.148Z", "log.level": "WARN", "message":"address [127.0.0.1:9302], node [null], requesting [false] connection failed: [][127.0.0.1:9302] connect_exception: Connection refused: /127.0.0.1:9302: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs"}
{"@timestamp":"2022-03-07T17:31:50.149Z", "log.level": "WARN", "message":"address [127.0.0.1:9301], node [null], requesting [false] connection failed: [][127.0.0.1:9301] connect_exception: Connection refused: /127.0.0.1:9301: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][generic][T#5]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs"}
{"@timestamp":"2022-03-07T17:31:50.151Z", "log.level": "WARN", "message":"address [127.0.0.1:9300], node [null], requesting [false] connection failed: [es-master-2.elasticsearch][10.0.17.134:9300] local node found", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es-master-2.elasticsearch][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es-master-2.elasticsearch","elasticsearch.cluster.name":"prod-eks-logs"}
What is wrong in my upgrade process ?