Metricbeat --setup dashboards throws error for keystore not created

Elastic Stack Beats
1

Hi Experts,

We are developing automation script to setup metricbeat before start the metricbeat.
Our automation script does the following:

  1. Download and untar the metricbeat package (7.6.1) from elastic site.
  2. Enable system and oracle module through metricbeat enable command.
  3. Update system.yml file to enable additional metricsets.
  4. Update oracle.yml file to add oracle connection string.
  5. Update metricbeat.yml with elasticsearch and kibana host and port number. Also update if user provides username and password details for elasticsearch.
  1. Create a new directory structure to copy system and oracle visualization json files
  2. Setup dashboards using metricbeat --setup dashboards command.

NOTE : Currently elasticsearch and kibana running as a different docker container with no user credentials support.

Our automation script runs successfully with the above mentioned steps. Also if we setup the user and password for elasticsearch by updating in metricbeat.yml file and our automation script runs successfully.

Now we are changing our automation script to support providing key name instead of username and password for elasticsearch.
i.e. We are updating KEY name for username and password fields in metricbeat.yml
instead of actual username and password.

Now our automation script is failing if we provide username KEY and password KEY because of key is not created.

I have few questions on this issue currently facing

  1. When metricbeat --setup dashboards command is running, I assume metricbeat connects to kibana to load the visualization and not connecting to elasticsearch.
    Please confirm my assumption is correct or not correct ?

  1. Our elasticsearch container is running without user credentials support and eventhough if we provide some dummy username and password in metricbeat.yml and setup dashboard command is successfull. But if we provide KEY name, setup dashboard command is failing. Do we really need to create a KEY before setup dashboard command runs ? Also If elasticsearch is configured with user credentials support then do we need to create KEY before hand prior to load the dashboard?

  2. Currently we are planning to update this user and password key in metricbeat.yml after ran the setup dashboard command. Is that fine ?

Error Received:
./metricbeat setup --dashboards
Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.username' (source:'metricbeat.yml')

If we configure actual username and password, then we did not got the above error and dashboard loaded successfully.

Please respond. Thanks in advance.

2

Hey!

About q1 your assumption is correct.
Can you elaborate more on what KEY stands for please?

C.

3

@ChrsMark
Thanks for the response.
We are supporting keystore to store the value of elastic credentials.
In existing metricbeat.yml, we store username and password directly in metricbeat.yml

output.elasticsearch:
hosts: ["svmondev1_elasticsearch_1:9200"]
username: "elastic"
password: "changeme"

With this configuration, I am not getting issue while loading dashboards.

But When i configure with key name for user and password like below in metricbeat.yml

output.elasticsearch:
hosts: ["svmondev1_elasticsearch_1:9200"]
username: "${ES_USER}"
password: "${ES_PASSWD}"

I am getting this error
Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.username' (source:'metricbeat.yml')

If ./metricbeat --setup dashboard command does only connect to kibana and not to elasticsearch, then no use of accessing elasticsearch user and password key right ?

Please confirm whether this can be fixed from metricbeat side.

4

Ok! Have you created keystore and store ES_USER value?

Regarding your question about access to ES and Kibana. In order to setup dashboards Metribceat will only connect to Kibana (iirc) but many other setup commands connect to ES too. In many cases Kibana and ES are sharing the creds and you are able to connect to Kibana using ES creds.

5

Nope @ChrsMark. I cannot create at this moment. Because as i mentioned above, my automation script only download the package and complete till setup dashboards.

Once my automation script is completed, then only i can able to create key ES_USER and ES_PASSWD to store the credentials.
Keystore creation and addition is post setup activity.

So in this case, do we need to download and untar the package and create key store first before proceeding with the setup?

6

Then can you try with something similar to env var default values? https://www.elastic.co/guide/en/beats/metricbeat/current/using-environ-vars.html

Something like ${VAR:default_value}.

7

Ok. Thanks @ChrsMark.

So in this case either we need to go with creating environment variable or Perform download & untar package and keystore creation before the automation setup ?

But one advantage in keystore is the value entered is not visible to anyone and it is secretly stored, where as in env variable, it is visible to user.

8

I think that providing a default value in case keystore is not defined will work too. Something like username: "${ES_USER:elastic}". So no need to switch to env vars. Can you give it a shot?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.