Logstash: Variable substitution FAILS for "api_key" in elasticsearch output

Elastic Stack Logstash
1

Hello,

I have configured logstash to use logstash.keystore to store secure credentials.
The ${ES_PASSWD} substitutions are working perfectly on the following configuration:

elasticsearch {
          ...
          user => "elastic"
          password => "${ES_PASSWD}"
          ssl => true
          ssl_certificate_verification => true
          ...
       }

But when I am trying to use a secure keystore variable for api_key it doesn't get replaced/substituted with the value inside the secret store:

elasticsearch {
          ...
          api_key => "${ES_API_KEY_SECRET}"
          ssl => true
          ssl_certificate_verification => true
          ...
       }

Obviously the secret api key value is correct because when I use it as plaintext it works like a charm.

Is there any reason that vars from logstash.keystore are not replaced inside the api_key attribute?

I am using logstash 8.4.1.1 on RHEL 8.6.

Any input is much appreciated.

Logstash Secure Keystore procedure followed: Secrets keystore for secure settings | Logstash Reference [8.4] | Elastic

Thank you,

Georgios

2

Problem resolved. Passwd had ' at the end dispite the fact we didn't put it in there. We reset tha passwd on the keystore and it worked.