Migration of elk on docker

Gaurav_Agarwal · February 19, 2018, 6:09am

Hello
I am new to docker also kind of a beginner for elk.
Till now I was using windows for elk now was willing to migrate to docker can some one guide me how can I migrate and run my config file of logstash from my local host to docker on a server.

Regards
Gaurav

magnusbaeck · February 19, 2018, 7:15am

Have you read the documentation?

https://www.elastic.co/guide/en/logstash/current/docker.html
https://www.elastic.co/guide/en/logstash/current/docker-config.html

Gaurav_Agarwal · February 19, 2018, 7:25am

I m working on a server now on the server i have dragged drop my conf files using winscp now when i tr7y to use the commands as per the documentation it shows this error can u please let me know where i m going wrong.

magnusbaeck · February 19, 2018, 7:38am

Please don't post screenshots. Use copy/paste.

What files do you have in the current directory when you run the command above?

Gaurav_Agarwal · February 19, 2018, 7:42am

[gagarwal3@]/% docker run -h logstash --name logstash --link elasticsearch:elasticsearch -it --rm -v "$PWD":/remote/users/gagarwal3/tmp logstash -f /remote/users/gagarwal3/tmp/uuuu.conf
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
07:13:16.761 [main] INFO logstash.modules.scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
07:13:16.767 [main] INFO logstash.modules.scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
07:13:16.776 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
07:13:16.778 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
07:13:16.814 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"f70e2899-6cad-4e66-a3e2-64e513613432", :path=>"/var/lib/logstash/uuid"}
07:13:16.913 [LogStash::Runner] INFO logstash.agent - No config files found in path {:path=>"/remote/users/gagarwal3/tmp/uuuu.conf"}
07:13:16.916 [LogStash::Runner] ERROR logstash.agent - failed to fetch pipeline configuration {:message=>"No config files found: /remote/users/gagarwal3/tmp/uuuu.conf. Can you make sure this path is a logstash config file?"}

this is the error

I created a new folder named tmp in the server directory where in i dragged dropped the conf files from the local logstash bin folder_emphasized text_

magnusbaeck · February 19, 2018, 8:00am

Please answer the question: What files do you have in the current directory when you run the command above?

Gaurav_Agarwal · February 19, 2018, 9:01am

in remote/users/gagarwal3/tmp there is only one file which is the config file named uuuu.conf

magnusbaeck · February 19, 2018, 9:09am

That's odd. What if you start a shell using the Logstash image,

docker run --entrypoint /bin/sh -it --rm -v "$PWD":/remote/users/gagarwal3/tmp logstash

and try step into /remote/users/gagarwal3/tmp. Does the directory exist? Does it contain uuuu.conf?

Gaurav_Agarwal · February 19, 2018, 9:17am

[gagarwal3@nceaptlht04]/% docker run --entrypoint /bin/sh -it --rm -v "$PWD":/remote/users/gagarwal3/tmp logstash

/remote/users/gagarwal3/tmp

/bin/sh: 1: /remote/users/gagarwal3/tmp: Permission denied

ls -lrt

total 72
drwxr-xr-x 2 root root 4096 Nov 19 15:25 home
drwxr-xr-x 2 root root 4096 Nov 19 15:25 boot
drwxr-xr-x 2 root root 4096 Feb 13 00:00 srv
drwxr-xr-x 2 root root 4096 Feb 13 00:00 opt
drwxr-xr-x 2 root root 4096 Feb 13 00:00 mnt
drwxr-xr-x 2 root root 4096 Feb 13 00:00 media
lrwxrwxrwx 1 root root 33 Feb 15 11:08 docker-java-home -> /usr/lib/jvm/java-8-openjdk-amd64
-rwxrwxr-x 1 root root 311 Feb 15 18:54 docker-entrypoint.sh
drwxr-xr-x 2 root root 4096 Feb 16 08:58 bin
drwxr-xr-x 8 root root 4096 Feb 16 08:58 lib
drwx------ 2 root root 4096 Feb 16 08:58 root
drwxr-xr-x 2 root root 4096 Feb 16 08:58 lib64
drwxr-xr-x 4 root root 4096 Feb 16 08:58 run
drwxr-xr-x 2 root root 4096 Feb 16 08:58 sbin
drwxrwxrwt 3 root root 4096 Feb 16 08:58 tmp
drwxr-xr-x 10 root root 4096 Feb 16 08:59 usr
drwxr-xr-x 11 root root 4096 Feb 16 08:59 var
drwxr-xr-x 1 root root 4096 Feb 19 09:10 etc
dr-xr-xr-x 153 root root 0 Feb 19 09:10 proc
dr-xr-xr-x 13 root root 0 Feb 19 09:10 sys
drwxr-xr-x 3 root root 4096 Feb 19 09:10 remote
drwxr-xr-x 5 root root 360 Feb 19 09:10 dev

magnusbaeck · February 19, 2018, 10:00am

What user does Logstash run as, root? Does root on your machine have access to /remote/users/gagarwal3?

system · March 19, 2018, 10:01am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.