Elasticsearch Installation 8.4

Hi Team,
I am trying to install elasticsearch-8.4 version on a centos vm. The steps i followed are:

  1. Java Installation
  2. Creating a repo file
  3. Install Elasticsearch
  4. Start ES

Note: i have not made any changes in yml file. It is a single node.
However after running the below command, I am getting error. Please suggest.

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200


Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

Hi Warkolm,
Apologies for the same.
I am attaching a new issue.
I am installing elk 8.4 on kubernetes and getting a error. Can you please support?

 ---
clusterName: "attelk"
nodeGroup: "master"

# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""

# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.roles=master
# https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles
roles:
  master: "true"
  data: "true"
  data: "true"

replicas: 3
minimumMasterNodes: 2

esMajorVersion: ""

# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
#  elasticsearch.yml: |
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value

createCert: true

esJvmOptions: {}
#  processors.options: |
#    -XX:ActiveProcessorCount=3

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# Allows you to load environment variables from kubernetes secret or config map
envFrom: []
# - secretRef:
#     name: env-secret
# - configMapRef:
#     name: config-map

# Disable it to use your own elastic-credential Secret.
secret:
  enabled: true
  password: "password" # generated randomly if not defined

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: elastic-certificates
#    secretName: elastic-certificates
#    path: /usr/share/elasticsearch/config/certs
#    defaultMode: 0755

hostAliases: []
#- ip: "127.0.0.1"
#  hostnames:
#  - "foo.local"
#  - "bar.local"

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "8.2.3"
imagePullPolicy: "IfNotPresent"

podAnnotations: {}
# iam.amazonaws.com/role: es-cluster

# additionals labels
labels: {}

esJavaOpts: "-Xmx6G -Xms6G" # example: "-Xmx1g -Xms1g"

resources:
  requests:
    cpu: "3000m"
    memory: "8Gi"
  limits:
    cpu: "3000m"
    memory: "8Gi"

initResources: {}
# limits:
#   cpu: "25m"
#   # memory: "128Mi"
# requests:
#   cpu: "25m"
#   memory: "128Mi"

#networkHost: "0.0.0.0"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "gp2"
  resources:
    requests:
      storage: 20Gi

rbac:
  create: false
  serviceAccountAnnotations: {}
  serviceAccountName: ""
  automountToken: true

podSecurityPolicy:
  create: false
  name: ""
  spec:
    privileged: true
    fsGroup:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
      - secret
      - configMap
      - persistentVolumeClaim
      - emptyDir

persistence:
  enabled: true
  labels:
    # Add default labels for the volumeClaimTemplate of the StatefulSet
    enabled: false
  annotations: {}

extraVolumes: []
# - name: extras
#   emptyDir: {}

extraVolumeMounts: []
# - name: extras
#   mountPath: /usr/share/extras
#   readOnly: true

extraContainers: []
# - name: do-something
#   image: busybox
#   command: ['do', 'something']

extraInitContainers: []
# - name: do-something
#   image: busybox
#   command: ['do', 'something']

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "soft"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

# The environment variables injected by service links are not used, but can lead to slow Elasticsearch boot times when
# there are many services in the current namespace.
# If you experience slow pod startups you probably want to set this to `false`.
enableServiceLinks: true

protocol: https
httpPort: 9200
transportPort: 9300

service:
  enabled: true
  labels: {}
  labelsHeadless: {}
  type: ClusterIP
  # Consider that all endpoints are considered "ready" even if the Pods themselves are not
  # https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec
  publishNotReadyAddresses: false
  nodePort: ""
  annotations: {}
  httpPortName: http
  transportPortName: transport
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000

securityContext:
  capabilities:
    drop:
      - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""

imagePullSecrets: []
nodeSelector: {}
tolerations: []

# Enabling this will publicly expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: false
  annotations: {}
  # kubernetes.io/ingress.class: nginx
  # kubernetes.io/tls-acme: "true"
  className: "nginx"
  pathtype: ImplementationSpecific
  hosts:
    - host: chart-example.local
      paths:
        - path: /
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

nameOverride: ""
fullnameOverride: ""
healthNameOverride: ""

lifecycle: {}
# preStop:
#   exec:
#     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
#   exec:
#     command:
#       - bash
#       - -c
#       - |
#         #!/bin/bash
#         # Add a template to adjust number of shards/replicas
#         TEMPLATE_NAME=my_template
#         INDEX_PATTERN="logstash-*"
#         SHARD_COUNT=8
#         REPLICA_COUNT=1
#         ES_URL=http://localhost:9200
#         while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done
#         curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}'

sysctlInitContainer:
  enabled: true

keystore: []

networkPolicy:
  ## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
  ## In order for a Pod to access Elasticsearch, it needs to have the following label:
  ## {{ template "uname" . }}-client: "true"
  ## Example for default configuration to access HTTP port:
  ## elasticsearch-master-http-client: "true"
  ## Example for default configuration to access transport port:
  ## elasticsearch-master-transport-client: "true"

  http:
    enabled: false
    ## if explicitNamespacesSelector is not set or set to {}, only client Pods being in the networkPolicy's namespace
    ## and matching all criteria can reach the DB.
    ## But sometimes, we want the Pods to be accessible to clients from other namespaces, in this case, we can use this
    ## parameter to select these namespaces
    ##
    # explicitNamespacesSelector:
    #   # Accept from namespaces with all those different rules (only from whitelisted Pods)
    #   matchLabels:
    #     role: frontend
    #   matchExpressions:
    #     - {key: role, operator: In, values: [frontend]}

    ## Additional NetworkPolicy Ingress "from" rules to set. Note that all rules are OR-ed.
    ##
    # additionalRules:
    #   - podSelector:
    #       matchLabels:
    #         role: frontend
    #   - podSelector:
    #       matchExpressions:
    #         - key: role
    #           operator: In
    #           values:
    #             - frontend

  transport:
    ## Note that all Elasticsearch Pods can talk to themselves using transport port even if enabled.
    enabled: false
    # explicitNamespacesSelector:
    #   matchLabels:
    #     role: frontend
    #   matchExpressions:
    #     - {key: role, operator: In, values: [frontend]}
    # additionalRules:
    #   - podSelector:
    #       matchLabels:
    #         role: frontend
    #   - podSelector:
    #       matchExpressions:
    #         - key: role
    #           operator: In
    #           values:
    #             - frontend

tests:
  enabled: true

# Deprecated
# please use the above podSecurityContext.fsGroup instead
fsGroup: ""

Blockquote

And getting below error:

cluster.name":"attelk"}
{"@timestamp":"2022-09-12T08:33:32.636Z", "log.level":"ERROR", "message":"uncaught exception in thread [main]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler","elasticsearch.node.name":"attelk-master-0","elasticsearch.cluster.name":"attelk","error.type":"org.elasticsearch.bootstrap.StartupException","error.message":"java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:563)\n\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:509)\n\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:479)\n\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449)\n\tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151)\n\tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56)\n\tat org.elasticsearch.node.Node.<init>(Node.java:454)\n\tat org.elasticsearch.node.Node.<init>(Node.java:291)\n\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)\n\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)\n\tat org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)\n\tat org.elasticsearch.cli.Command.main(Command.java:77)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)\nFor complete error details, refer to the log at /usr/share/elasticsearch/logs/attelk.log\n"}
uncaught exception in thread [main]
java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:563)
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:509)
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:479)
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449)
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151)
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56)
        at org.elasticsearch.node.Node.<init>(Node.java:454)
        at org.elasticsearch.node.Node.<init>(Node.java:291)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
        at org.elasticsearch.cli.Command.main(Command.java:77)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/attelk.log

Can you please support?

//Ankita

I can't see that anywhere in your config, which is odd?

Hi Warkolm,
I have used the link helm-charts/values.yaml at main · elastic/helm-charts · GitHub

Can please guide how to fix this?

//Ankita

Hi, apparently you're using an older version (7.x) of the roles specification (with boolean values instead of a list of roles) in your file, and possibly an old version of the Helm chart. Anyway, I am not sure that the helm chart officially supports versions 8.x.
If you are not forced to use helm chart, I really encourage you to look at Kubernetes operator GitHub - elastic/cloud-on-k8s: Elastic Cloud on Kubernetes which works very well.

Hi Team,
Can somebody from elastic confirm on the same?

//Ankita

I'm not sure that the helm charts works with version 8, there is an open issue about it.

The helm charts won't be updated anymore by Elastic, they will focus on the Kubernet Operator, the ECK, but it I'm not sure if they will release a working version for 8.X or not.

Unless someone from the community steps up to maintain the helm charts, the project will not be updated.

If it's not starting then the logs will give you more information why.

Hi Team,
While adding nodes to elasticsearch cluster using enrollment token, I am getting the following errors.

[root@elk2 ~]# yum install --enablerepo=elasticsearch elasticsearch -y
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
elasticsearch | 1.3 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.x86_64 0:8.4.1-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================
Package Arch Version Repository Size

Installing:
elasticsearch x86_64 8.4.1-1 elasticsearch 540 M

Transaction Summary

Install 1 Package

Total download size: 540 M
Installed size: 1.1 G
Downloading packages:
elasticsearch-8.4.1-x86_64.rpm | 540 MB 00:00:27
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Installing : elasticsearch-8.4.1-1.x86_64 1/1
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : 8=IID8ZEYjBaw+c8SNbr

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token '
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.


NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

You can start elasticsearch service by executing

sudo systemctl start elasticsearch.service
Verifying : elasticsearch-8.4.1-1.x86_64 1/1

Installed:
elasticsearch.x86_64 0:8.4.1-1

Complete!
[root@elk2 ~]# vi /etc/elasticsearch/elasticsearch.yml
[root@elk2 ~]# cd /etc/elasticsearch/
[root@elk2 elasticsearch]# vi /etc/elasticsearch/elasticsearch.yml
[root@elk2 elasticsearch]# ./elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxNzIuNjEuNC4xMTA6OTIwMCJdLCJmZ3IiOiJiZjBjYzZlODA4ZTk3YTFhYzJiOTU5MGYwNjcxM2FkNjIwZTU3OTMyMWQyNzRjNjJjNjE2OWI2Y2M3OWEyNTEzIiwia2V5IjoiMlBjZU40TUIydl9sWkpqQ2VBU2k6OEp0X3A5c1lRMDZma1Z2djNvbkJ0USJ9
-bash: ./elasticsearch-reconfigure-node: No such file or directory
[root@elk2 elasticsearch]# ls
certs elasticsearch.keystore elasticsearch-plugins.example.yml elasticsearch.yml jvm.options jvm.options.d log4j2.properties role_mapping.yml roles.yml users users_roles
[root@elk2 elasticsearch]# cd /usr/share/elasticsearch/bin/
[root@elk2 bin]# ./elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxNzIuNjEuNC4xMTA6OTIwMCJdLCJmZ3IiOiJiZjBjYzZlODA4ZTk3YTFhYzJiOTU5MGYwNjcxM2FkNjIwZTU3OTMyMWQyNzRjNjJjNjE2OWI2Y2M3OWEyNTEzIiwia2V5IjoiMlBjZU40TUIydl9sWkpqQ2VBU2k6OEp0X3A5c1lRMDZma1Z2djNvbkJ0USJ9

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically:

  • Security auto configuration will be removed from elasticsearch.yml
  • The [certs] config directory will be removed
  • Security auto configuration related secure settings will be removed from the elasticsearch.keystore
    Do you want to continue with the reconfiguration process [y/N]y

ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [172.61.4.110:9200] were attempted.
[root@elk2 bin]# ./elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxNzIuNjEuNC4xMTA6OTIwMCJdLCJmZ3IiOiJiZjBjYzZlODA4ZTk3YTFhYzJiOTU5MGYwNjcxM2FkNjIwZTU3OTMyMWQyNzRjNjJjNjE2OWI2Y2M3OWEyNTEzIiwia2V5IjoiMlBjZU40TUIydl9sWkpqQ2VBU2k6OEp0X3A5c1lRMDZma1Z2djNvbkJ0USJ9
Generates all the necessary security configuration for a node in a secured cluster

Option Description


-E Configure a setting
--enrollment-token The enrollment token to use
-h, --help Show help
-s, --silent Show minimal output
-v, --verbose Show verbose output

ERROR: Aborting enrolling to cluster. This node doesn't appear to be auto-configured for security. Expected configuration is missing from elasticsearch.yml.
[root@elk2 bin]# ./elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxNzIuNjEuNC4xMTA6OTIwMCJdLCJmZ3IiOiJiZjBjYzZlODA4ZTk3YTFhYzJiOTU5MGYwNjcxM2FkNjIwZTU3OTMyMWQyNzRjNjJjNjE2OWI2Y2M3OWEyNTEzIiwia2V5IjoiMlBjZU40TUIydl9sWkpqQ2VBU2k6OEp0X3A5c1lRMDZma1Z2djNvbkJ0USJ9^C
[root@elk2 bin]#
[root@elk2 bin]#
[root@elk2 bin]#
[root@elk2 bin]# vi /var/log/elasticsearch/elasticsearch.log
[root@elk2 bin]# /usr/share/elasticsearch/bin/elasticsearch --enrollment-token eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxNzIuNjEuNC4xMTA6OTIwMCJdLCJmZ3IiOiJiZjBjYzZlODA4ZTk3YTFhYzJiOTU5MGYwNjcxM2FkNjIwZTU3OTMyMWQyNzRjNjJjNjE2OWI2Y2M3OWEyNTEzIiwia2V5IjoiMlBjZU40TUIydl9sWkpqQ2VBU2k6OEp0X3A5c1lRMDZma1Z2djNvbkJ0USJ9

ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [172.61.4.110:9200] were attempted.

However, telnet shows port is connecting.

[root@ip-172-61-6-28 bin]# telnet 172.61.4.110 9200
Trying 172.61.4.110...
Connected to 172.61.4.110.
Escape character is '^]'.

Please support.