ELK Autoprovision with X-Pack Security in Docker container

lungati · January 25, 2020, 12:48pm

I have the following bash script that I'd like to run while building the elk image (sebp/elk) so that it can create the builtin users using credentials I've saved in a file (the current setup doesn't allow for a better way to save the credentials, but I'm open to ideas). I'd like suggestions on how to improve the script so that it works as expected.

What I expect:

1 - It will run the elasticsearch-setup-passwords script and output the passwords into a file

2 - That I can then use the file to get the elastic user password which I use to generate the Basic Authentication base64 string

3 - Then I use the ELK's REST APIs to change the passwords with those I have in a file

4 - I then paste the new passwords in the config files (for kibana and logstash)

What I'm getting:

1 - There's no builtin_users file, although I get the login prompt when I access localhost:9200, so the passwords must have changed (I'm assuming)

The script (running in Bash 4.4.19):

#!/bin/bash
#let script exit if a command fails
set -o errexit 

#let script exit if an unsed variable is used
set -o nounset

# Save builtin user passwords
function save_users(){
    touch $ES_HOME/builtin_users.txt;
    $ES_HOME/bin/./elasticsearch-setup-passwords auto -b 1> $ES_HOME/builtin_users.txt;
}


# Get the elastic user password
function get_elastic_pwd(){
    readonly elastic_pwd=$(awk '/PASSWORD elastic/ { print $4 }' $ES_HOME/builtin_users.txt);
}


# Get new user passwords
function get_new_pwds(){
    readonly apm_system_pwd=$(awk '/apm_system/ { print $2 }' $ES_HOME/creds.txt)
    readonly kibana_pwd=$(awk '/kibana/ { print $2 }' $ES_HOME/creds.txt)
    readonly logstash_system_pwd=$(awk '/logstash_system/ { print $2 }' $ES_HOME/creds.txt)
    readonly beats_system_pwd=$(awk '/beats_system/ { print $2 }' $ES_HOME/creds.txt)
    readonly remote_monitoring_user_pwd=$(awk '/remote_monitoring_user/ { print $2 }' $ES_HOME/creds.txt)
    readonly elastic_new_pwd=$(awk '/elastic/ { print $2 }' $ES_HOME/creds.txt)
}


# Change builtin user passwords
function change_pwds(){
    curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic $(echo -n elastic:$elastic_pwd | base64)" -fSL http://localhost:9200/_security/user/apm_system/_password -d '{"password": "'$apm_system_pwd'"}'

    curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic $(echo -n elastic:$elastic_pwd | base64)" -fSL http://localhost:9200/_security/user/kibana/_password  -d '{"password": "'$kibana_pwd'"}'

    curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic $(echo -n elastic:$elastic_pwd | base64)" -fSL http://localhost:9200/_security/user/logstash_system/_password -d '{"password": "'$logstash_system_pwd'"}'

    curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic $(echo -n elastic:$elastic_pwd | base64)" -fSL http://localhost:9200/_security/user/beats_system/_password -d '{"password": "'$beats_system_pwd'"}'

    curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic $(echo -n elastic:$elastic_pwd | base64)" -fSL http://localhost:9200/_security/user/remote_monitoring_user/_password -d '{"password": "'$remote_monitoring_user_pwd'"}'

    curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic $(echo -n elastic:$elastic_pwd | base64)" -fSL http://localhost:9200/_security/user/elastic/_password -d '{"password": "'$elastic_new_pwd'"}'

}

# Set the new passwords in required files
function set_new_pwds(
    find ${KIBANA_HOME}/config/kibana.yml | xargs -L1 bash -c 'sed "s;kibana_pwd;$kibana_pwd;g" $1 > /tmp/.intermediate-file-233; cp /tmp/.intermediate-file-233 $1;' --
    find ${LOGSTASH_PATH_CONF}/logstash.yml | xargs -L1 bash -c 'sed "s;logstash_system_pwd;$logstash_system_pwd;g" $1 > /tmp/.intermediate-file-234; cp /tmp/.intermediate-file-234 $1;' --
)

# Create the script log file
touch $ES_HOME/setup_users.log;
save_users() || echo "Error saving builtin users" >> $ES_HOME/setup_users.log;
get_elastic_pwd() || echo "Error setting elastic pwd" >> $ES_HOME/setup_users.log;
get_new_pwds() || echo "Error getting new pwds" >> $ES_HOME/setup_users.log;
change_pwds() || echo "Error changing pwds" >> $ES_HOME/setup_users.log;
set_new_pwds() || echo "Error setting new pwds" >> $ES_HOME/setup_users.log;

exit 0

My Dockerfile:

FROM sebp/elk

RUN apt-get update && apt-get install -y ssh libnss3 

# This section is for installing logstash plugins
#
WORKDIR ${LOGSTASH_HOME}
RUN gosu logstash bin/logstash-plugin install logstash-input-syslog && \
    gosu logstash bin/logstash-plugin install logstash-input-tcp && \
	gosu logstash bin/logstash-plugin install logstash-output-elasticsearch && \
	gosu logstash bin/logstash-plugin install logstash-filter-mutate && \
	gosu logstash bin/logstash-plugin install logstash-filter-multiline && \
	gosu logstash bin/logstash-plugin install logstash-filter-json && \
	gosu logstash bin/logstash-plugin install logstash-filter-split && \
	gosu logstash bin/logstash-plugin install logstash-filter-csv && \
	gosu logstash bin/logstash-plugin install logstash-filter-geoip && \
	mkdir -p ${LOGSTASH_PATH_CONF}/templates

# Add global configs
ADD --chown=elasticsearch:elasticsearch config/jvm.options ${ES_PATH_CONF}/jvm.options
ADD --chown=logstash:logstash config/logstash.yml ${LOGSTASH_PATH_CONF}/logstash.yml
ADD --chown=logstash:logstash config/pipelines.yml ${LOGSTASH_PATH_CONF}/pipelines.yml
ADD --chown=kibana:kibana config/kibana.yml ${KIBANA_HOME}/config/kibana.yml
ADD --chown=elasticsearch:elasticsearch config/elasticsearch.yml ${ES_PATH_CONF}/elasticsearch.yml

# Add templates
ADD --chown=logstash:logstash templates/beats.yml ${LOGSTASH_PATH_CONF}/templates/beats.yml

# Add pipeline configs
RUN rm ${LOGSTASH_PATH_CONF}/conf.d/*
ADD --chown=logstash:logstash config/02-input-beats.conf ${LOGSTASH_PATH_CONF}/conf.d/02-input-beats.conf
ADD --chown=logstash:logstash config/03-input-tcp.conf ${LOGSTASH_PATH_CONF}/conf.d/03-input-tcp.conf
ADD --chown=logstash:logstash config/20-filter-beats.conf ${LOGSTASH_PATH_CONF}/conf.d/20-filter-beats.conf
ADD --chown=logstash:logstash config/21-filter-haproxy.conf ${LOGSTASH_PATH_CONF}/conf.d/21-filter-haproxy.conf
ADD --chown=logstash:logstash config/22-filter-tcp.conf ${LOGSTASH_PATH_CONF}/conf.d/22-filter-tcp.conf
ADD --chown=logstash:logstash config/29-filter-post.conf ${LOGSTASH_PATH_CONF}/conf.d/29-filter-post.conf
ADD --chown=logstash:logstash config/30-output.conf ${LOGSTASH_PATH_CONF}/conf.d/30-output.conf

# Setting permissions
RUN mkdir -p ${LOGSTASH_HOME}/data/queue && \
    mkdir -p ${LOGSTASH_HOME}/data/dead_letter_queue && \
    chown -R logstash:logstash ${LOGSTASH_HOME}/data && \
    chmod 777 -R ${LOGSTASH_HOME}/data

# Test config
RUN ${LOGSTASH_HOME}/bin/logstash --config.test_and_exit --path.config ${LOGSTASH_PATH_CONF}/conf.d/

# Adjust network details
RUN echo "fs.inotify.max_user_instances=8192 \
fs.mqueue.msg_max=8192 \
net.ipv4.ip_local_port_range=15000 61000 \
net.ipv4.tcp_fin_timeout=30 \
net.ipv4.tcp_tw_reuse=1 \
net.core.somaxconn=8192 \
net.core.netdev_max_backlog=4096 \
net.ipv4.tcp_max_syn_backlog=4096 \
" >> /etc/sysctl.conf
RUN sysctl -p

# Prepare for startup
COPY config/startup.sh /startup.sh
RUN chmod 755 /startup.sh

# Create inbuilt users
COPY --chown=elasticsearch:elasticsearch config/creds.txt ${ES_HOME}/creds.txt
COPY --chown=elasticsearch:elasticsearch config/setup-users.sh ${ES_HOME}/setup_users.sh
RUN chmod 755 ${ES_HOME}/setup_users.sh

CMD [ "/startup.sh" ]

ikakavas · January 25, 2020, 2:25pm

elasticsearch-setup-passwords is not really geared to be used in an automated/scripted manner.

You can pass the environment variable ELASTIC_PASSWORD to set the password for the elastic user ( see https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls-docker.html for an example ) and use that then as credentials to make the API calls to set the password for other builtin users.

lungati · January 26, 2020, 6:57pm

Thanks. I’ll try that out

lungati · January 27, 2020, 8:42am

Hi @ikakavas ,

Thanks for responding earlier.

I've tried setting ENV variables ELASTIC_PASSWORD or even ELASTIC_BOOTSTRAP_PASSWORD in my Dockerfile but doesn't seem to set the initial password for elastic user.

I'm using this image: sebp/elk

ikakavas · January 27, 2020, 7:51pm

I'm not aware of this image, why don't you use the one we provide ?

lungati · February 5, 2020, 1:19am

Yes, plan on changing. Thanks

system · March 4, 2020, 1:19am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Bash Scripting ElasticSearch and Kibana Passwords Kibana elastic-stack-security	9	691	January 16, 2023
Automate Elasticsearch-reset-password in bash script Elasticsearch elastic-stack-security	11	2315	November 24, 2022
Scripting password setup after install Elasticsearch elastic-stack-security	4	4515	October 31, 2019
Set Elastic Password during deployment Elasticsearch elastic-stack-security , docker	4	447	November 23, 2022
How to create build in user without interactive mode or auto Elasticsearch	5	6120	June 27, 2019

ELK Autoprovision with X-Pack Security in Docker container

Related topics