Setting Up Logstash In Docker-Compose For Bulk Ingest Of CSV Files In Local Machine

Ethan777100 · November 11, 2023, 5:16pm

CONTINUATION FROM kibana-8-11-0-failed-to-start-exit-code-1

My use case is to bulk ingest csv files into Elasticsearch. Understand i need Logstash to do it.

Not sure how to start.

Should I be using a default or custom configuration that looks like

A sep docker image to talk to the main environment.

Any need for Port connection/config if my data is residing in my File Explorer?

Am using csv data I want to pipe in from File Explorer. The data is sorted by year, month and each file exists as 1 day's worth.

Will this be 1 index per day or 1 index for everything?

Ethan777100 · November 13, 2023, 4:28pm

This is my docker-env setup now.

I was advised to seperate logstash and run on its own by @stephenb . Need help or further explanation on what this means, - seperate docker-envs?

And how to setup and make sure both docker environments link up with each other?

I need my logstash to be a pipeline into my main docker-env for the Data.

Thanks in Advance.

stephenb · November 13, 2023, 5:43pm

@Ethan777100

Just set up a separate compose with just the logstash in it... that should work... take out the lines from the other... create a new directory, and put the compose file in it.

Take out the depends on...

You can leave in the network part I think...

If the dockert networks are shared should not be a problem... if they are not, you just use the host IP with the correct ports...

networks:
  default:
    name: elastic
    external: false <!--- You might want to set that to true in the logstash compose

Ethan777100 · November 14, 2023, 8:17am

I just created a new docker-compose.yml. Also copied the .env file over

Is this looking right? ^^

Here is the docker-compose.yml for logstash. Not entirely sure what else needs to be removed. I already removed the es01 and kibana setup lines. But some portions of the bash script I'm not sure how to act on them.

version: "3.8"

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: true
    
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  logstash:
  
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - logstashdata01:/usr/share/logstash/data
      - certs:/usr/share/logstash/certs
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - NODE_NAME="logstash"
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD={ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

Ethan777100 · November 15, 2023, 7:37am

I urgently need help.

When I follow this,
Build a Custom Docker Image for Logstash Ali Younes

I realised I need this elasticsearch-ca.pem that needs to be generated from securing the Elasticsearch cluster.

Securing the Elasticsearch Cluster - YouTube

Else i cannot setup the Docker Image in the first place as per Custom Docker Image.

 => ERROR [ 3/29] COPY --chmod=755 elasticsearch-ca.pem /usr/share/logstash/certs                                                                                               0.0s 
 => CACHED [ 4/29] RUN apt-get install -y sudo                                                                                                                                  0.0s 
 => CACHED [ 5/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.11.0-amd64.deb                                                           0.0s 
 => CACHED [ 6/29] RUN sudo dpkg -i metricbeat-8.11.0-amd64.deb                                                                                                                 0.0s 
 => CACHED [ 7/29] RUN rm metricbeat-8.11.0-amd64.deb                                                                                                                           0.0s 
 => CACHED [ 8/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-amd64.deb                                                               0.0s 
 => CACHED [ 9/29] RUN sudo dpkg -i filebeat-8.11.0-amd64.deb                                                                                                                   0.0s
 => CACHED [10/29] RUN rm filebeat-8.11.0-amd64.deb                                                                                                                             0.0s 
 => ERROR [11/29] COPY --chmod=755 elasticsearch-ca.pem /etc/metricbeat/                                                                                                        0.0s 
 => ERROR [12/29] COPY --chmod=755 elasticsearch-ca.pem /etc/filebeat/                                                                                                          0.0s 
 => CACHED [13/29] RUN metricbeat modules enable logstash-xpack                                                                                                                 0.0s 
 => CACHED [14/29] RUN metricbeat modules disable system                                                                                                                        0.0s 
 => CACHED [15/29] RUN rm -f /etc/metricbeat/metricbeat.yml                                                                                                                     0.0s 
 => CACHED [16/29] RUN rm -f /etc/metricbeat/modules.d/logstash-xpack.yml                                                                                                       0.0s 
 => ERROR [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/                                                                                                   0.0s 
 => ERROR [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/                                                                                     0.0s 
 => CACHED [19/29] RUN filebeat modules enable logstash                                                                                                                         0.0s 
 => CACHED [20/29] RUN filebeat modules disable system                                                                                                                          0.0s 
 => CACHED [21/29] RUN rm -f /etc/filebeat/filebeat.yml                                                                                                                         0.0s 
 => CACHED [22/29] RUN rm -f /etc/filebeat/modules.d/logstash.yml                                                                                                               0.0s 
 => ERROR [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/                                                                                                         0.0s 
 => ERROR [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/                                                                                               0.0s 
------
 > [ 3/29] COPY --chmod=755 elasticsearch-ca.pem /usr/share/logstash/certs:
------
------
 > [11/29] COPY --chmod=755 elasticsearch-ca.pem /etc/metricbeat/:
------
------
 > [12/29] COPY --chmod=755 elasticsearch-ca.pem /etc/filebeat/:
------
------
 > [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/:
------
------
 > [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/:
------
------
 > [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/:
------
------
 > [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/:
------
Dockerfile:53
--------------------
  51 |     RUN rm -f /etc/filebeat/modules.d/logstash.yml
  52 |     COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/
  53 | >>> COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/
  54 |
  55 |     # Remove the default log4j2.properties file and copy the custom one to the container
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 90cab661-b7e8-4991-b930-2712e8309ff4::3lj53r6q7ydmz77z5kh58xdtv: "/filebeat/logstash.yml": not found

I'm trying to make the Certs as per Set up basic security for the Elastic Stack | Elasticsearch Guide [8.11] | Elastic

Set up basic security for the Elastic Stack plus secured HTTPS traffic | Elasticsearch Guide [8.11] | Elastic

But the YouTube video is going by Oct 2022 configurations. I am stuck in trying to make the certs because the ES Docker file structure has evolved over the past 1 year.

Please help me.

I feel I'm going into deeper holes unnecessarily. Why does it feel like as if I must jump so many hoops just to get a simple pipeline logstash running?

I reasonably wanna bypass not making these certs. But this means I have to go into Ali Younges configuration as-is on Github and make even more deletions and modifications and incorporate other lines from other Youtubers into the various .ymls just to not encounter errors regarding missing certs.

I'm so tired and drained trying to compare other users environment, official Elastic Docs contexts wrt to mine and making the necessary adjustments to get my configuration set up.

I. need. help.

It would have been so much easier if there wasn't this stupid memory issue with logstash and going thru so much hassle to seperate logstash into a seperate Docker Container.

I dont even know what's right and what's the direction because I'm running out of time and cannot afford to get entangled in a deep web of trial, error and confusion.

I am very scared to accidentally break something now.

stephenb · November 15, 2023, 3:46pm

Hi @Ethan777100

Wow.... that is quite a journey... not sure why you are trying to build a whole docker images and every thing...

Here is what I did....

in a separate directory from the where you did the elasticsearch compose.

Here are the files

$ ls -la
total 12
drwxr-xr-x   5 sbrown  staff  160 Nov 15 07:36 ./
drwxr-xr-x  10 sbrown  staff  320 Nov 15 07:15 ../
-rw-r--r--   1 sbrown  staff  843 Nov 15 07:24 .env
-rw-r--r--   1 sbrown  staff  399 Nov 15 07:35 docker-compose.yml
-rw-r--r--   1 sbrown  staff  373 Nov 15 07:19 logstash.conf

Here is the docker-compose.yml
It uses the network you previously defined
note I also put in the -r which means you can edit the logstash.conf file while logstash is running and logstash will reload it... good for debug test...

networks:
  default:
    name: elastic
    external: true # Says use the other network
    
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    environment:
      - xpack.monitoring.enabled=false
    volumes:
      - ./:/usr/share/logstash/pipeline/
    command: logstash -r -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

here is logstash.conf
for the time being I told it not to validate the cert, can show you that later

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://es01:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
    action => "create"
    user => "elastic"
    password => "mypassword"
    ssl_verification_mode=> "none"
  }
}

This starts up and works and connects to elastic and listens for a beat to connect... but I think you want to read csvs etc... But this should work now you need to work on loading your CSVs.

hyperion:logstash sbrown$ docker-compose up 
[+] Building 0.0s (0/0)                                                                             docker:desktop-linux
[+] Running 1/0
 ✔ Container logstash-logstash-1  Created                                                                           0.0s 
Attaching to logstash-logstash-1
logstash-logstash-1  | 2023/11/15 15:52:09 Setting 'xpack.monitoring.enabled' from environment.
logstash-logstash-1  | Using bundled JDK: /usr/share/logstash/jdk
logstash-logstash-1  | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
.....
logstash-logstash-1  | [2023-11-15T15:52:22,579][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash-logstash-1  | [2023-11-15T15:52:22,623][INFO ][org.logstash.beats.Server][main][d9dee40397240d93c16edb2f2e3e225edeebcbe1ac13e47294feb7bbc611d2ee] Starting server on port: 5044

Ethan777100 · November 15, 2023, 4:00pm

Thanks for the response. I was actually spending the last 5-6 hours doing further work on my own. Now i wonder if I shld try merge your changes into what I've done so far.

This actually my logstash.conf as of now. I hard coded the location of all my csv files where I parked them in my D drive.

Your method is to use beats on Port 5044. Do both methods yield same result?

For the index wise, this set of csv file I'm porting in are about 1.5years of data of same system. 1 day, 1 csv file output. I was intending to pipe them in as 1 elasticsearch index called ats-logs. Is this the correct way to index your data?

input { 
    file { 
        path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv" 
        start_position => "beginning" 
        sincedb_path => "NULL"
    } 
}
filter { 
    csv { 
        separator => ","
        columns => [ "id","uniqueid","alarm","eventtype","system","subsystem","sourcetime","operator","alarmvalue","value","equipment","location","severity","description","state","mmsstate","zone","graphicelement"] 
    } 
}

output { 
    elasticsearch { 
        index => "ats-logs" 
        hosts => ["https://es01:9200"]
        user => "elastic"
        password => "elastic123"
        ssl_enabled => true
        cacert => "/usr/share/logstash/certs/ca/ca.crt"
    }
    stdout{} 
}

In future, if I receive a different set of csv files of another category, and I want to pipe them into elasticsearch, do I need to change anything in logstash.conf in order to port in new data subsequently?

stephenb · November 15, 2023, 4:05pm

No that was just an example to show you how to run the whole process.

What you are doing is the normal way to read / parse CSV

It depends on what you want to do.... but sure that is fine.

You should also learn about mappings... this will work, but you will get a default mapping for each fields both text and keyword lets chat about that later.

Well sure .. the path.. and if they are a different format you will need to change the columns etc..etc...

I see you learned that good!

Get it working then refine...

Ethan777100 · November 15, 2023, 4:11pm

sincedb_path => "NULL"

Thanks for the affirmation~~~ I was actually just following YouTube video. Smth about allowing the logstash pointer to continue where I left off and etc should the instance crash.

Tbh, I was following this video the entire afternoon (now its 0007hrs in Singapore), and now at here working to transfer the generated elasticsearch-ca.pem to kibana.

I'm going thru this whole entire process because in
Build a Custom Docker Image for Logstash Ali Younes (youtube.com)

The provided github repo requires elasticsearch-ca.pem before creating the docker volume

So much effort I spent just to generate elasticsearch-ca.pem . hence why I detoured so intensively just because I wanted to follow his video step by step since the logstash workspace already given to me.

However, should I just scratch that and just follow what you've provided? Dk whether its not too late to turn back.

Ethan777100 · November 15, 2023, 4:29pm

Alright @stephenb I decided to boldly just follow the configs u provided. Bypassed those certs and dared to docker-compose up my logstash

logstash.conf

input { 
    file { 
        path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv" 
        start_position => "beginning" 
        sincedb_path => "NULL"
    } 
}
filter { 
    csv { 
        separator => ","
        columns => [ "id","uniqueid","alarm","eventtype","system","subsystem","sourcetime","operator","alarmvalue","value","equipment","location","severity","description","state","mmsstate","zone","graphicelement"] 
    } 
}

output { 
    elasticsearch { 
        index => "ats-logs" 
        hosts => ["https://es01:9200"]
        manage_template => false
        user => "elastic"
        password => "elastic123"
        ssl_verification_mode=> "none"
        # ssl_enabled => true
        # cacert => "/usr/share/logstash/certs/ca/ca.crt"
    }
    stdout{} 
}

docker-compose.yml

version: "3.8"

volumes:
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: true # Says use the other network
    
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    environment:
      - xpack.monitoring.enabled=false
    volumes:
      - ./:/usr/share/logstash/pipeline/
    command: logstash -r -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

Good news is that the logstash docker image is healthy. never crash on me.

However, the pipeline is not setup and has error.

2023-11-16 00:26:06 2023/11/15 16:26:06 Setting 'xpack.monitoring.enabled' from environment.
2023-11-16 00:26:06 Using bundled JDK: /usr/share/logstash/jdk
2023-11-16 00:26:40 Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2023-11-16 00:26:40 [2023-11-15T16:26:40,130][WARN ][deprecation.logstash.runner] NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
2023-11-16 00:26:40 [2023-11-15T16:26:40,148][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
2023-11-16 00:26:40 [2023-11-15T16:26:40,150][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.11.0", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [x86_64-linux]"}
2023-11-16 00:26:40 [2023-11-15T16:26:40,155][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
2023-11-16 00:26:40 [2023-11-15T16:26:40,174][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
2023-11-16 00:26:40 [2023-11-15T16:26:40,178][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
2023-11-16 00:26:40 [2023-11-15T16:26:40,576][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2023-11-16 00:26:40 [2023-11-15T16:26:40,596][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"93c92697-1707-49a1-bf50-640c85f4879e", :path=>"/usr/share/logstash/data/uuid"}
2023-11-16 00:26:41 [2023-11-15T16:26:41,887][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-11-16 00:26:42 [2023-11-15T16:26:42,403][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [^\\r\\n], \"\\r\", \"\\n\" at line 46, column 4 (byte 1133) after # }", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
2023-11-16 00:26:45 [2023-11-15T16:26:45,730][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [^\\r\\n], \"\\r\", \"\\n\" at line 46, column 4 (byte 1133) after # }", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}

I know I need to see json object entry for every csv row being piped in - to confirm that I have encountered SUCCESS.

Provided current state of my .conf and ymls for your Inspection.

Ethan777100 · November 15, 2023, 4:45pm

If I follow Build a Custom Docker Image for Logstash - YouTube now, this is where I get stuck when I try to build the docker image.

PS C:\Users\ethan\OneDrive - Nanyang Technological University\NTU Documents\EEE\Y5S1\EE4080 FINAL YEAR PROJECT\docker-env\logstash-docker-image> docker build -t logstash-8.11.0 .
[+] Building 1.0s (28/33)                                                                                                                                                        docker:default
 => [internal] load .dockerignore                                                                                                                                                          0.1s
 => => transferring context: 2B                                                                                                                                                            0.0s 
 => [internal] load build definition from Dockerfile                                                                                                                                       0.1s 
 => => transferring dockerfile: 2.60kB                                                                                                                                                     0.0s 
 => [internal] load metadata for docker.elastic.co/logstash/logstash:8.11.0                                                                                                                0.0s 
 => [internal] load build context                                                                                                                                                          0.2s
 => => transferring context: 1.86kB                                                                                                                                                        0.2s 
 => CACHED [ 1/29] FROM docker.elastic.co/logstash/logstash:8.11.0                                                                                                                         0.0s 
 => CANCELED [ 2/29] RUN mkdir /usr/share/logstash/certs                                                                                                                                   0.8s 
 => CACHED [ 3/29] COPY --chmod=755 elasticsearch-ca.pem /usr/share/logstash/certs                                                                                                         0.0s
 => CACHED [ 4/29] RUN apt-get install -y sudo                                                                                                                                             0.0s 
 => CACHED [ 5/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.11.0-amd64.deb                                                                      0.0s 
 => CACHED [ 6/29] RUN sudo dpkg -i metricbeat-8.11.0-amd64.deb                                                                                                                            0.0s 
 => CACHED [ 7/29] RUN rm metricbeat-8.11.0-amd64.deb                                                                                                                                      0.0s 
 => CACHED [ 8/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-amd64.deb                                                                          0.0s 
 => CACHED [ 9/29] RUN sudo dpkg -i filebeat-8.11.0-amd64.deb                                                                                                                              0.0s
 => CACHED [10/29] RUN rm filebeat-8.11.0-amd64.deb                                                                                                                                        0.0s 
 => CACHED [11/29] COPY --chmod=755 elasticsearch-ca.pem /etc/metricbeat/                                                                                                                  0.0s 
 => CACHED [12/29] COPY --chmod=755 elasticsearch-ca.pem /etc/filebeat/                                                                                                                    0.0s 
 => CACHED [13/29] RUN metricbeat modules enable logstash-xpack                                                                                                                            0.0s 
 => CACHED [14/29] RUN metricbeat modules disable system                                                                                                                                   0.0s 
 => CACHED [15/29] RUN rm -f /etc/metricbeat/metricbeat.yml                                                                                                                                0.0s 
 => CACHED [16/29] RUN rm -f /etc/metricbeat/modules.d/logstash-xpack.yml                                                                                                                  0.0s 
 => ERROR [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/                                                                                                              0.0s 
 => ERROR [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/                                                                                                0.0s 
 => CACHED [19/29] RUN filebeat modules enable logstash                                                                                                                                    0.0s 
 => CACHED [20/29] RUN filebeat modules disable system                                                                                                                                     0.0s 
 => CACHED [21/29] RUN rm -f /etc/filebeat/filebeat.yml                                                                                                                                    0.0s 
 => CACHED [22/29] RUN rm -f /etc/filebeat/modules.d/logstash.yml                                                                                                                          0.0s 
 => ERROR [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/                                                                                                                    0.0s 
 => ERROR [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/                                                                                                          0.0s 
------
 > [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/:
------
------
 > [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/:
------
------
 > [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/:
------
------
 > [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/:
------
Dockerfile:53
--------------------
  51 |     RUN rm -f /etc/filebeat/modules.d/logstash.yml
  52 |     COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/
  53 | >>> COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/
  54 |
  55 |     # Remove the default log4j2.properties file and copy the custom one to the container
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 90cab661-b7e8-4991-b930-2712e8309ff4::vp33ej6ywl29mlkb4oh1mirvd: "/filebeat/logstash.yml": not found

Snapshot of my file directory logstash-docker-image

The 1st 3 folders are those logstash plugins I downloaded from GitHub. I havent got to implementing them yet. Unless you can help me with this now as well?

stephenb · November 15, 2023, 4:46pm

Why are you building a custom image?

Confused I through you were following my approach... you have lost me...

Apologies but I am not going to help build a custom image... sorry.

back to my method.

What is your directory structure? assumes the file

logstash.conf is in the same directory of the compose file

You will also need to mount where your .csv files are so logstash can find them

Logstash in Docker will not be able to access

path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv"

in the compose

- /path/to/your/csv_files:/csv_files

Also I know windows but I would remove all the spaces and special characters in the path...

so perhaps

- D:/ATS_Event_Logs/Logstash:D:/Logs/csv_files

Then in your .conf

path => "D:/Logs/csv_files/*.csv"

Something like that

Ethan777100 · November 15, 2023, 4:54pm

Fresh setup of standard logstash workspace with the 3 folders of plugins i downloaded from Github.

Is this better?

Logstash in Docker will not be able to access

path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv"

Oh. I parked my Data in the D drive whilst working on C Drive.

Ethan777100 · November 15, 2023, 4:59pm

stephenb · November 15, 2023, 5:03pm

Yeah but I am a bit worried about all the : I am not on windows... but yes try that.

I would do a very simple test to load a file make sure the framework is there etc...

mounts and files are found etc

But yeah looking close to what you want... and muuuuuchhh simpler

Ohh wait !!!

That is wrong the docker is unix

Hold on lets try in the compose

- D:/ATS_Event_Logs/Logstash:/logs/csv_files

First part is Windows Path to Real Files:/unix/path/to/directory

then in the .conf its the UNIX path

path => "/logs/csv_files/*.csv"

Remember logstash in docker is actually running on Unix
So all paths in the confs are unix paths that are mounted volumes to your real windows file system

Ethan777100 · November 15, 2023, 5:13pm

ethan777@LAPTOP-PESEJ3MG:/mnt/c/Users/ethan/OneDrive - Nanyang Technological University/NTU Documents/EEE/Y5S1/EE4080 FINAL YEAR PROJECT/docker-env/logstash-standard$ docker-compose up -d
validating /mnt/c/Users/ethan/OneDrive - Nanyang Technological University/NTU Documents/EEE/Y5S1/EE4080 FINAL YEAR PROJECT/docker-env/logstash-standard/docker-compose.yml: services.logstash Additional property paths is not allowed

I think the wsl terminal does not allow the extra line
paths: - D:/ATS_Event_Logs/Logstash:/logs/csv_files

in docker-compose.yml

stephenb · November 15, 2023, 5:14pm

Your using a Cloud Drive?... I would use local only...

Can you show me the whole volumes section... in the compose

One volume line per line.

    volumes:
      - ./:/usr/share/logstash/pipeline/
      - D:/ATS_Event_Logs/Logstash:/logs/csv_files

Ethan777100 · November 15, 2023, 5:16pm

version: "3.8"

volumes:
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: true # Says use the other network
    
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    environment:
      - xpack.monitoring.enabled=false
    volumes:
      - ./:/usr/share/logstash/pipeline/
    command: logstash -r -f /usr/share/logstash/pipeline/logstash.conf
    paths:
    - D:/ATS_Event_Logs/Logstash:/logs/csv_files
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

stephenb · November 15, 2023, 5:17pm

    volumes:
      - ./:/usr/share/logstash/pipeline/
      - "D:\ATS_Event_Logs\Logstash:/logs/csv_files"

the - means a elements in a list

Could be a windows... path thing ... need to look at the docker docs

Think you have to Quote it

  - "D:\ATS_Event_Logs\Logstash:/logs/csv_files"

BUT I will tell you I am not a WSL person... are there Unix Paths if so use them... think you are going to need how to figure that out...

/mnt/d/pathtofiles

if you can figure out the UNIX paths that may be better... sorry not my area of expertise...

Looks like you have

/mnt/c/Users/ethan/....

and I have NO CLUE what effect one drive has on that... I would use a local drive... if it were me and I was testing... (you keep throwing me curveballs )

Although that may work ... The error was basically saying bad syntax on this line

paths: - D:/ATS_Event_Logs/Logstash:/logs/csv_files

gotta run... I will check in later

Ethan777100 · November 15, 2023, 5:32pm

Sorry abt the OneDrive and curveballs. But you really help me immensely idk how to Thank you.

I'm actually working from my Desktop, with a remote-in to my laptop where the Docker action is happening.

For the timebeing, in case I needed to access the same work space from my Desktop, I can do so via OneDrive.

Most of my school life I was using the University provided OneDrive to store all my files related to Uni.

Maybe later on when things stabilise can I just copy out the entire workspace into local C Drive and then start the docker-composes again? Shld still work right?

Anyway