Elasticsearch 8.10.2 synonyms not working when synonyms_path is used

[smritibhandari91 (Smriti)]

We successfully deployed Elasticsearch 8.10.2 using the ECK operator. However, we encountered an issue when trying to access the synonyms_path during the index creation process.

Error: The problem is that the index creation fails with an IOException, indicating a problem with the path for synonyms_path_path - IOException while reading synonyms_path_path:.

Current Setup: we created a volume named 'x' using emptyDir as the source for temporary storage under the podTemplate section of the elasticsearch.yml file for Elasticsearch deployment via ECK. This volume is intended for storing synonym configuration file. We then mounted the 'x' volume at the path '/a/b/elasticsearch/config/x' within the 'xyz' elasticsearch container, ensuring that the contents of the 'x' volume are accessible to the 'xyz' container during index creation.

Although the file is present at the correct path on the volume claim, an error occurs when trying to access the path during the index creation process

[smritibhandari91 (Smriti)]

We have tried another approach as recommended by ES documentation 8.10.2 via config maps as well. ES recommendation documents: Custom configuration files and plugins | Elastic Cloud on Kubernetes [2.10] | Elastic and Provisioning of custom configuration files · Issue #1571 · elastic/cloud-on-k8s · GitHub

We are adding a synonyms file for the synonym token filter in Elasticsearch by mounting synonyms file into the configuration directory of Elasticsearch via config map.
The config map has the all synonyms defined inline.

Attaching some code snippets for reference, the code snippet is just for the data node but we have similarly defined master and client nodes in elasticsearch.yml and synonyms file exists on each node as expected.

- configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: synonyms-config
data:
  synonyms_en.txt: |
    ipod, i-pod, i pod
    ...
  synonyms_fr.txt: |
    personal computer => pc
    ...

- elasticsearch.yml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-eck
spec:
  version: 8.10.2
  volumeClaimDeletePolicy: DeleteOnScaledownOnly
  updateStrategy:
    changeBudget:
      maxSurge: -1
      maxUnavailable: 0
  http:
    service:
      spec:
        type: ClusterIP
        ports:
          - name: http
            port: 9200
            targetPort: 9200
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: data
    count: 1
    config:
      node.roles: ["data", "ingest"]
    podTemplate:
      metadata:
        labels:
          app: elasticsearch-eck
      spec:
        volumes:
          - name: synonyms-volume
            configMap:
              name: synonyms-config
        initContainers:
        - name: configure-sysctl
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          securityContext:
            runAsUser: 0
            privileged: true
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - elasticsearch-eck
              topologyKey: kubernetes.io/hostname
        containers:
        - name: elasticsearch-eck
          volumeMounts:
            - name: synonyms-volume
              mountPath: /usr/share/elasticsearch/config/dictionaries
          env:
            - name: ES_JAVA_OPTS
              value: "-Xms8g -Xmx8g"
          resources:
            requests:
              cpu: 2000m
              memory: 12Gi
            limits:
              cpu: 8000m
              memory: 16Gi
          image: "docker.elastic.co/elasticsearch/elasticsearch:8.10.2"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
            runAsUser: 1000
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: standard
  podDisruptionBudget:
    spec:
      maxUnavailable: 0
      selector:
        matchLabels:
          app: elasticsearch-eck

At "/usr/share/elasticsearch/config/dictionaries" location in our data, client and master nodes, the synonyms file exists with the expected data.

Attaching some screenshot representing files in the ES cluster:

Our index creation definition looks like this:

                    ...
                    "english_synonym_filter": {
                        "type": "synonym_graph",
                        "lenient": true,
                        "synonyms_path": "/usr/share/elasticsearch/config/dictionaries/synonyms_en.txt"
                    },
.....


**Note:** We have also tried relative path in the index creation template: 

                "english_synonym_filter": {
                    "type": "synonym_graph",
                    "lenient": true,
                    "synonyms_path": "dictionaries/synonyms_en.txt"
                },

However, we get the no such file exception in both cases.

Error:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "IOException while reading synonyms_path_path: /usr/share/elasticsearch/config/dictionaries/synonyms_fr.txt"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "IOException while reading synonyms_path_path: /usr/share/elasticsearch/config/dictionaries/synonyms_fr.txt",
        "caused_by": {
            "type": "no_such_file_exception",
            "reason": "/usr/share/elasticsearch/config/dictionaries/synonyms_fr.txt"
        }
    },
    "status": 400
}