How to import a config file into Docker Container of Logstash?

[redapplesonly (RAO)]

Hi Logstash Ninja Masters,

I recently spun up a Logstash docker container (v 7.4.0), to replace a local-app version of LS on the same Ubuntu box. I‘m confused how to instruct the Docker LS to use the configuration that the local-app LS used?

Some details:

My host machine is Ubuntu 16.04.4, and my Docker version is 17.09.0-ce. For the moment, I want a simple standalone LS docker instance. By that, I mean I don’t want to worry about pipeline files, I don’t want to use docker-compose. Here’s my docker run command:

docker run --rm -it -d \
    -v /usr/share/logstash/config/logstash.conf:/home/me/logstash/logstash-7.4.0/config/myConfig.conf
    --name myLogstash \
    --net myNet \
    -e XPACK.MONITORING_ELASTICSEARCH_HOSTS="http://10.10.10.100:9200" \
    docker.elastic.co/logstash/logstash:7.4.0

That second line is my clumsy attempt to tell Docker LS, “Hey, be sure to import and use this local config file, okay?”

The container comes up great:

me@ubuntu1:~# docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                                              NAMES
8ec488b628f5        c2c1ac6b995b          "/usr/local/bin/do..."   2 hours ago         Up 2 hours          5044/tcp, 9600/tcp                                                                 myLogstash
me@ubuntu1:~#

So… the container comes up, but I don’t think its using my config file.

How do I force the Docker LS to do that? I’ve tried copying the desired file into the container’s /usr/share/logstash/config/ directory; no effect. Can I do that and somehow restart the service?

Some other questions I have that aren’t address in the documentation… From running the local version of LS, I learned that you launched the LS service like this:

~bin/logstash -r -f config/myConfig.conf

Basically, you execute the logstash app, suppling your config file as input. The config file has a .conf suffix. So why does the LS Docker documentation talk about .yml files as config files? Are they the same?

Final Jeopardy Question: Once my config file is (hopefully) used, how do I verify Docker LS is running it?

Anyway, if you can see the error of my ways, I’d appreciate any help you can offer. Thanks!

[furiabhavesh (Bhavesh Furia)]

@redapplesonly : I created a separate Dockerfile and added config file in it so that I can also version control my changes.

Reference : https://github.com/bhaveshf-cuelogic/logstash-docker/blob/master/Dockerfile

According to my experience in last few days .yml file is used to configure Logstash's behavior like bootup configs, ssl config, metrics etc. whereas .conf files are used to configure what input it should expect, how it should process and where it should output after processing.

I can have 2 .conf files if I want to separate out input from 2 different applications for better management.

[redapplesonly (RAO)]

Thanks furiabhavesh,

I appreciate you taking the time to write about this, sorry for the delayed response on my part.

Is it your experience that importing the config file can only be done through a Dockerfile (ie, through docker-compose)? Because I don't have docker-compose running on my machine for resource purposes. I was hoping to learn of a way to impose the config file and behavior I want on by Logstash container by other means.

 Many thanks!

[redapplesonly (RAO)]

Hey all,

I worked out the solution, which may help anyone else who is following in my footsteps. All that I document below was in the online Logstash 7.4.0 documentation, just spread around in different locations.

The first thing you must do is edit a settings file here: /usr/share/logstash/config/logstash.yml

You need to add a line telling Logstash where your configuration files is:

http.host: 0.0.0.0
path.config: /usr/share/logstash/config/logstash.conf                   <<========================
xpack.monitoring.elasticsearch.hosts: http://10.10.10.10:9200

That second line is what I added. Then, I made sure the config file I wanted was in that location. Finally, I restarted the LS docker container. When the container restarted, it was using the config file I wanted.

[furiabhavesh (Bhavesh Furia)]

Yes @redapplesonly, I was comfortable with basic Docker so that's the quick solution I came across.