# Setting Up Logstash In Docker-Compose For Bulk Ingest Of CSV Files In Local Machine

**URL:** https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916
**Category:** Logstash
**Tags:** docker
**Created:** [November 11, 2023, 5:16pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916 "2023-11-11T17:16:21Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 11, 2023, 5:16pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/1 "2023-11-11T17:16:21Z")

</div>

CONTINUATION FROM [kibana-8-11-0-failed-to-start-exit-code-1](https://discuss.elastic.co/t/kibana-8-11-0-failed-to-start-exit-code-1/346893/20)

My use case is to bulk ingest csv files into Elasticsearch. Understand i need Logstash to do it.

Not sure how to start.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/5/759e47beebadb8329bdda92b3222410cafded1ce.png)

Should I be using a default or custom configuration that looks like  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/0/506e2c6beb7fb479a9268a5c3082f5de012ab35b.png)  
A sep docker image to talk to the main environment.

Any need for Port connection/config if my data is residing in my File Explorer?

Am using csv data I want to pipe in from File Explorer. The data is sorted by year, month and each file exists as 1 day's worth.

Will this be 1 index per day or 1 index for everything?

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 13, 2023, 4:28pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/2 "2023-11-13T16:28:38Z")

</div>

This is my docker-env setup now.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/70e48f0976e8072a7b9729c0c1244c744cb01f41.png)

I was advised to seperate logstash and run on its own by @stephenb . Need help or further explanation on what this means, - seperate docker-envs?

And how to setup and make sure both docker environments link up with each other?

I need my logstash to be a pipeline into my main docker-env for the Data.

Thanks in Advance.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 13, 2023, 5:43pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/3 "2023-11-13T17:43:56Z")

</div>

@Ethan777100

Just set up a separate compose with just the logstash in it... that should work... take out the lines from the other... create a new directory, and put the compose file in it.

Take out the depends on...

You can leave in the network part I think...

> [@Ethan777100](#):
>
> And how to setup and make sure both docker environments link up with each other?

If the dockert networks are shared should not be a problem... if they are not, you just use the host IP with the correct ports...

> **[Networking in Compose](https://docs.docker.com/compose/networking/#use-a-pre-existing-network)**
>
> How Docker Compose sets up networking between containers

```auto
networks:
  default:
    name: elastic
    external: false <!--- You might want to set that to true in the logstash compose 
    

```

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 14, 2023, 8:17am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/4 "2023-11-14T08:17:36Z")

</div>

I just created a new docker-compose.yml. Also copied the .env file over

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0af3c8e76e798e440cdf6f48826d72a33f31b5de.png)

Is this looking right? ^^

Here is the docker-compose.yml for logstash. Not entirely sure what else needs to be removed. I already removed the es01 and kibana setup lines. But some portions of the bash script I'm not sure how to act on them.

```auto
version: "3.8"

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: true
    
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [x${ELASTIC_PASSWORD} == x]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [x${KIBANA_PASSWORD} == x]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [! -f config/certs/ca.zip]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [! -f config/certs/certs.zip]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          " - name: es01\n"\
          " dns:\n"\
          " - es01\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[-f config/certs/es01/es01.crt]"]
      interval: 1s
      timeout: 5s
      retries: 120

  logstash:
  
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - logstashdata01:/usr/share/logstash/data
      - certs:/usr/share/logstash/certs
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
    environment:
      - NODE_NAME="logstash"
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD={ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
    command: logstash -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

```

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 7:37am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/5 "2023-11-15T07:37:22Z")

</div>

I urgently need help.

When I follow this,  
[Build a Custom Docker Image for Logstash Ali Younes](https://https://www.youtube.com/watch?v=weHiB7JJMaQ&list=PLe0UsEbF-5meLIZJaAOF5QDKey-v1PEvw&index=22&pp=iAQB)

I realised I need this elasticsearch-ca.pem that needs to be generated from securing the Elasticsearch cluster.

[Securing the Elasticsearch Cluster - YouTube](https://www.youtube.com/watch?v=w3taEk9tAQ4&list=PLe0UsEbF-5meLIZJaAOF5QDKey-v1PEvw&index=11)

Else i cannot setup the Docker Image in the first place as per Custom Docker Image.

```auto
 => ERROR [3/29] COPY --chmod=755 elasticsearch-ca.pem /usr/share/logstash/certs 0.0s 
 => CACHED [4/29] RUN apt-get install -y sudo 0.0s 
 => CACHED [5/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.11.0-amd64.deb 0.0s 
 => CACHED [6/29] RUN sudo dpkg -i metricbeat-8.11.0-amd64.deb 0.0s 
 => CACHED [7/29] RUN rm metricbeat-8.11.0-amd64.deb 0.0s 
 => CACHED [8/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-amd64.deb 0.0s 
 => CACHED [9/29] RUN sudo dpkg -i filebeat-8.11.0-amd64.deb 0.0s
 => CACHED [10/29] RUN rm filebeat-8.11.0-amd64.deb 0.0s 
 => ERROR [11/29] COPY --chmod=755 elasticsearch-ca.pem /etc/metricbeat/ 0.0s 
 => ERROR [12/29] COPY --chmod=755 elasticsearch-ca.pem /etc/filebeat/ 0.0s 
 => CACHED [13/29] RUN metricbeat modules enable logstash-xpack 0.0s 
 => CACHED [14/29] RUN metricbeat modules disable system 0.0s 
 => CACHED [15/29] RUN rm -f /etc/metricbeat/metricbeat.yml 0.0s 
 => CACHED [16/29] RUN rm -f /etc/metricbeat/modules.d/logstash-xpack.yml 0.0s 
 => ERROR [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/ 0.0s 
 => ERROR [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/ 0.0s 
 => CACHED [19/29] RUN filebeat modules enable logstash 0.0s 
 => CACHED [20/29] RUN filebeat modules disable system 0.0s 
 => CACHED [21/29] RUN rm -f /etc/filebeat/filebeat.yml 0.0s 
 => CACHED [22/29] RUN rm -f /etc/filebeat/modules.d/logstash.yml 0.0s 
 => ERROR [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/ 0.0s 
 => ERROR [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/ 0.0s 
------
 > [3/29] COPY --chmod=755 elasticsearch-ca.pem /usr/share/logstash/certs:
------
------
 > [11/29] COPY --chmod=755 elasticsearch-ca.pem /etc/metricbeat/:
------
------
 > [12/29] COPY --chmod=755 elasticsearch-ca.pem /etc/filebeat/:
------
------
 > [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/:
------
------
 > [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/:
------
------
 > [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/:
------
------
 > [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/:
------
Dockerfile:53
--------------------
  51 | RUN rm -f /etc/filebeat/modules.d/logstash.yml
  52 | COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/
  53 | >>> COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/
  54 |
  55 | # Remove the default log4j2.properties file and copy the custom one to the container
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 90cab661-b7e8-4991-b930-2712e8309ff4::3lj53r6q7ydmz77z5kh58xdtv: "/filebeat/logstash.yml": not found

```

I'm trying to make the Certs as per [Set up basic security for the Elastic Stack | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup.html)

[Set up basic security for the Elastic Stack plus secured HTTPS traffic | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html)

But the YouTube video is going by Oct 2022 configurations. I am stuck in trying to make the certs because the ES Docker file structure has evolved over the past 1 year.

Please help me.

I feel I'm going into deeper holes unnecessarily. Why does it feel like as if I must jump so many hoops just to get a simple pipeline logstash running?

I reasonably wanna bypass not making these certs. But this means I have to go into Ali Younges configuration as-is on Github and make even more deletions and modifications and incorporate other lines from other Youtubers into the various .ymls just to not encounter errors regarding missing certs.

I'm so tired and drained trying to compare other users environment, official Elastic Docs contexts wrt to mine and making the necessary adjustments to get my configuration set up.

I. need. help.

It would have been so much easier if there wasn't this stupid memory issue with logstash and going thru so much hassle to seperate logstash into a seperate Docker Container.

I dont even know what's right and what's the direction because I'm running out of time and cannot afford to get entangled in a deep web of trial, error and confusion.

I am very scared to accidentally break something now.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 15, 2023, 3:46pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/6 "2023-11-15T15:46:56Z")

</div>

Hi @Ethan777100

Wow.... that is quite a journey... not sure why you are trying to build a whole docker images and every thing...

Here is what I did....

in a separate directory from the where you did the elasticsearch compose.

Here are the files

```auto
$ ls -la
total 12
drwxr-xr-x 5 sbrown staff 160 Nov 15 07:36 ./
drwxr-xr-x 10 sbrown staff 320 Nov 15 07:15 ../
-rw-r--r-- 1 sbrown staff 843 Nov 15 07:24 .env
-rw-r--r-- 1 sbrown staff 399 Nov 15 07:35 docker-compose.yml
-rw-r--r-- 1 sbrown staff 373 Nov 15 07:19 logstash.conf

```

Here is the docker-compose.yml  
It uses the network you previously defined  
note I also put in the `-r` which means you can edit the logstash.conf file while logstash is running and logstash will reload it... good for debug test...

```auto
networks:
  default:
    name: elastic
    external: true # Says use the other network
    
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    environment:
      - xpack.monitoring.enabled=false
    volumes:
      - ./:/usr/share/logstash/pipeline/
    command: logstash -r -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

```

here is logstash.conf  
for the time being I told it not to validate the cert, can show you that later

```auto
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://es01:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
    action => "create"
    user => "elastic"
    password => "mypassword"
    ssl_verification_mode=> "none"
  }
}

```

This starts up and works and connects to elastic and listens for a beat to connect... but I think you want to read csvs etc... But this should work now you need to work on loading your CSVs.

```auto
hyperion:logstash sbrown$ docker-compose up 
[+] Building 0.0s (0/0) docker:desktop-linux
[+] Running 1/0
 ✔ Container logstash-logstash-1 Created 0.0s 
Attaching to logstash-logstash-1
logstash-logstash-1 | 2023/11/15 15:52:09 Setting 'xpack.monitoring.enabled' from environment.
logstash-logstash-1 | Using bundled JDK: /usr/share/logstash/jdk
logstash-logstash-1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
.....
logstash-logstash-1 | [2023-11-15T15:52:22,579][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash-logstash-1 | [2023-11-15T15:52:22,623][INFO][org.logstash.beats.Server][main][d9dee40397240d93c16edb2f2e3e225edeebcbe1ac13e47294feb7bbc611d2ee] Starting server on port: 5044

```

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 4:00pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/7 "2023-11-15T16:00:16Z")

</div>

Thanks for the response. I was actually spending the last 5-6 hours doing further work on my own. Now i wonder if I shld try merge your changes into what I've done so far.

This actually my logstash.conf as of now. I hard coded the location of all my csv files where I parked them in my D drive.

Your method is to use beats on Port 5044. Do both methods yield same result?

For the index wise, this set of csv file I'm porting in are about 1.5years of data of same system. 1 day, 1 csv file output. I was intending to pipe them in as 1 elasticsearch index called ats-logs. Is this the correct way to index your data?

```auto
input { 
    file { 
        path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv" 
        start_position => "beginning" 
        sincedb_path => "NULL"
    } 
}
filter { 
    csv { 
        separator => ","
        columns => ["id","uniqueid","alarm","eventtype","system","subsystem","sourcetime","operator","alarmvalue","value","equipment","location","severity","description","state","mmsstate","zone","graphicelement"] 
    } 
}

output { 
    elasticsearch { 
        index => "ats-logs" 
        hosts => ["https://es01:9200"]
        user => "elastic"
        password => "elastic123"
        ssl_enabled => true
        cacert => "/usr/share/logstash/certs/ca/ca.crt"
    }
    stdout{} 
}

```

In future, if I receive a different set of csv files of another category, and I want to pipe them into elasticsearch, do I need to change anything in logstash.conf in order to port in new data subsequently?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 15, 2023, 4:05pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/8 "2023-11-15T16:05:34Z")

</div>

> [@Ethan777100](#):
>
> Your method is to use beats on Port 5044. Do both methods yield same result?

No that was just an example to show you how to run the whole process.

What you are doing is the normal way to read / parse CSV

> [@Ethan777100](#):
>
> For the index wise, this set of csv file I'm porting in are about 1.5years of data of same system. 1 day, 1 csv file output. I was intending to pipe them in as 1 elasticsearch index called ats-logs. Is this the correct way to index your data?

It depends on what you want to do.... but sure that is fine.

You should also learn about mappings... this will work, but you will get a default mapping for each fields both `text` and `keyword` lets chat about that later.

> [@Ethan777100](#):
>
> In future, if I receive a different set of csv files of another category, and I want to pipe them into elasticsearch, do I need to change anything in logstash.conf in order to port in new data subsequently?

Well sure .. the path.. and if they are a different format you will need to change the columns etc..etc...

> [@Ethan777100](#):
>
> ` sincedb_path => "NULL"`

I see you learned that good! 🙂

Get it working then refine...

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 4:11pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/9 "2023-11-15T16:11:20Z")

</div>

`sincedb_path => "NULL"`

Thanks for the affirmation~~~ I was actually just following YouTube video. Smth about allowing the logstash pointer to continue where I left off and etc should the instance crash.

Tbh, I was following this video the entire afternoon (now its 0007hrs in Singapore), and now at here working to transfer the generated `elasticsearch-ca.pem` to kibana.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/9/99438e08c27d7521601af46a63b671b7af0b09c2.jpeg)

I'm going thru this whole entire process because in  
[Build a Custom Docker Image for Logstash Ali Younes (youtube.com)](https://www.youtube.com/watch?v=weHiB7JJMaQ&list=PLe0UsEbF-5meLIZJaAOF5QDKey-v1PEvw&index=22&pp=iAQB)

The provided github repo requires `elasticsearch-ca.pem` before creating the docker volume

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/6/46bab9e0cabbb8af4dd7ef25e788675ef33c7398.png)

So much effort I spent just to generate `elasticsearch-ca.pem` . hence why I detoured so intensively just because I wanted to follow his video step by step since the logstash workspace already given to me.

However, should I just scratch that and just follow what you've provided? Dk whether its not too late to turn back.

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 4:29pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/10 "2023-11-15T16:29:19Z")

</div>

Alright @stephenb I decided to boldly just follow the configs u provided. Bypassed those certs and dared to docker-compose up my logstash

`logstash.conf`

```auto
input { 
    file { 
        path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv" 
        start_position => "beginning" 
        sincedb_path => "NULL"
    } 
}
filter { 
    csv { 
        separator => ","
        columns => ["id","uniqueid","alarm","eventtype","system","subsystem","sourcetime","operator","alarmvalue","value","equipment","location","severity","description","state","mmsstate","zone","graphicelement"] 
    } 
}

output { 
    elasticsearch { 
        index => "ats-logs" 
        hosts => ["https://es01:9200"]
        manage_template => false
        user => "elastic"
        password => "elastic123"
        ssl_verification_mode=> "none"
        # ssl_enabled => true
        # cacert => "/usr/share/logstash/certs/ca/ca.crt"
    }
    stdout{} 
}

```

`docker-compose.yml`

```auto
version: "3.8"

volumes:
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: true # Says use the other network
    
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    environment:
      - xpack.monitoring.enabled=false
    volumes:
      - ./:/usr/share/logstash/pipeline/
    command: logstash -r -f /usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

```

Good news is that the logstash docker image is healthy. never crash on me.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/0/507066df339cf1a68c6c18e9be2fb6c1b5f106f4.png)

However, the pipeline is not setup and has error.

```auto
2023-11-16 00:26:06 2023/11/15 16:26:06 Setting 'xpack.monitoring.enabled' from environment.
2023-11-16 00:26:06 Using bundled JDK: /usr/share/logstash/jdk
2023-11-16 00:26:40 Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2023-11-16 00:26:40 [2023-11-15T16:26:40,130][WARN][deprecation.logstash.runner] NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
2023-11-16 00:26:40 [2023-11-15T16:26:40,148][INFO][logstash.runner] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
2023-11-16 00:26:40 [2023-11-15T16:26:40,150][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.11.0", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [x86_64-linux]"}
2023-11-16 00:26:40 [2023-11-15T16:26:40,155][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
2023-11-16 00:26:40 [2023-11-15T16:26:40,174][INFO][logstash.settings] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
2023-11-16 00:26:40 [2023-11-15T16:26:40,178][INFO][logstash.settings] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
2023-11-16 00:26:40 [2023-11-15T16:26:40,576][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2023-11-16 00:26:40 [2023-11-15T16:26:40,596][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=>"93c92697-1707-49a1-bf50-640c85f4879e", :path=>"/usr/share/logstash/data/uuid"}
2023-11-16 00:26:41 [2023-11-15T16:26:41,887][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-11-16 00:26:42 [2023-11-15T16:26:42,403][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [^\\r\\n], \"\\r\", \"\\n\" at line 46, column 4 (byte 1133) after # }", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
2023-11-16 00:26:45 [2023-11-15T16:26:45,730][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [^\\r\\n], \"\\r\", \"\\n\" at line 46, column 4 (byte 1133) after # }", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}

```

I know I need to see json object entry for every csv row being piped in - to confirm that I have encountered SUCCESS.

Provided current state of my .conf and ymls for your Inspection.

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 4:45pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/11 "2023-11-15T16:45:42Z")

</div>

If I follow [Build a Custom Docker Image for Logstash - YouTube](https://www.youtube.com/watch?v=weHiB7JJMaQ&list=PLe0UsEbF-5meLIZJaAOF5QDKey-v1PEvw&index=23) now, this is where I get stuck when I try to build the docker image.

```auto
PS C:\Users\ethan\OneDrive - Nanyang Technological University\NTU Documents\EEE\Y5S1\EE4080 FINAL YEAR PROJECT\docker-env\logstash-docker-image> docker build -t logstash-8.11.0 .
[+] Building 1.0s (28/33) docker:default
 => [internal] load .dockerignore 0.1s
 => => transferring context: 2B 0.0s 
 => [internal] load build definition from Dockerfile 0.1s 
 => => transferring dockerfile: 2.60kB 0.0s 
 => [internal] load metadata for docker.elastic.co/logstash/logstash:8.11.0 0.0s 
 => [internal] load build context 0.2s
 => => transferring context: 1.86kB 0.2s 
 => CACHED [1/29] FROM docker.elastic.co/logstash/logstash:8.11.0 0.0s 
 => CANCELED [2/29] RUN mkdir /usr/share/logstash/certs 0.8s 
 => CACHED [3/29] COPY --chmod=755 elasticsearch-ca.pem /usr/share/logstash/certs 0.0s
 => CACHED [4/29] RUN apt-get install -y sudo 0.0s 
 => CACHED [5/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.11.0-amd64.deb 0.0s 
 => CACHED [6/29] RUN sudo dpkg -i metricbeat-8.11.0-amd64.deb 0.0s 
 => CACHED [7/29] RUN rm metricbeat-8.11.0-amd64.deb 0.0s 
 => CACHED [8/29] RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-amd64.deb 0.0s 
 => CACHED [9/29] RUN sudo dpkg -i filebeat-8.11.0-amd64.deb 0.0s
 => CACHED [10/29] RUN rm filebeat-8.11.0-amd64.deb 0.0s 
 => CACHED [11/29] COPY --chmod=755 elasticsearch-ca.pem /etc/metricbeat/ 0.0s 
 => CACHED [12/29] COPY --chmod=755 elasticsearch-ca.pem /etc/filebeat/ 0.0s 
 => CACHED [13/29] RUN metricbeat modules enable logstash-xpack 0.0s 
 => CACHED [14/29] RUN metricbeat modules disable system 0.0s 
 => CACHED [15/29] RUN rm -f /etc/metricbeat/metricbeat.yml 0.0s 
 => CACHED [16/29] RUN rm -f /etc/metricbeat/modules.d/logstash-xpack.yml 0.0s 
 => ERROR [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/ 0.0s 
 => ERROR [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/ 0.0s 
 => CACHED [19/29] RUN filebeat modules enable logstash 0.0s 
 => CACHED [20/29] RUN filebeat modules disable system 0.0s 
 => CACHED [21/29] RUN rm -f /etc/filebeat/filebeat.yml 0.0s 
 => CACHED [22/29] RUN rm -f /etc/filebeat/modules.d/logstash.yml 0.0s 
 => ERROR [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/ 0.0s 
 => ERROR [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/ 0.0s 
------
 > [17/29] COPY --chmod=755 metricbeat/metricbeat.yml /etc/metricbeat/:
------
------
 > [18/29] COPY --chmod=755 metricbeat/logstash-xpack.yml /etc/metricbeat/modules.d/:
------
------
 > [23/29] COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/:
------
------
 > [24/29] COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/:
------
Dockerfile:53
--------------------
  51 | RUN rm -f /etc/filebeat/modules.d/logstash.yml
  52 | COPY --chmod=755 filebeat/filebeat.yml /etc/filebeat/
  53 | >>> COPY --chmod=755 filebeat/logstash.yml /etc/filebeat/modules.d/
  54 |
  55 | # Remove the default log4j2.properties file and copy the custom one to the container
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 90cab661-b7e8-4991-b930-2712e8309ff4::vp33ej6ywl29mlkb4oh1mirvd: "/filebeat/logstash.yml": not found 

```

Snapshot of my file directory logstash-docker-image

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c643edbdd14708eb3ca6938bcefbbaa5be9f115e.png)

The 1st 3 folders are those logstash plugins I downloaded from GitHub. I havent got to implementing them yet. Unless you can help me with this now as well?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 15, 2023, 4:46pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/12 "2023-11-15T16:46:09Z")

</div>

Why are you building a custom image?

Confused I through you were following my approach... you have lost me...

Apologies but I am not going to help build a custom image... sorry.

back to my method.

What is your directory structure? assumes the file

`logstash.conf` is in the same directory of the compose file

You will also need to mount where your .csv files are so logstash can find them

Logstash in Docker will not be able to access

` path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv"`

in the compose

` - /path/to/your/csv_files:/csv_files`

Also I know windows but I would remove all the spaces and special characters in the path...

so perhaps

` - D:/ATS_Event_Logs/Logstash:D:/Logs/csv_files`

Then in your .conf

`path => "D:/Logs/csv_files/*.csv"`

Something like that

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 4:54pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/13 "2023-11-15T16:54:06Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/a/8/a8be0360457cff2e23eb67a8bb973f9121439564.png)  
Fresh setup of standard logstash workspace with the 3 folders of plugins i downloaded from Github.

Is this better?

> Logstash in Docker will not be able to access
> 
> `path => "D:/ATS Event Logs/For-Logstash (ML)/*.csv"`

Oh. I parked my Data in the D drive whilst working on C Drive.

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 4:59pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/14 "2023-11-15T16:59:36Z")

</div>

> [@stephenb](#):
>
> - D:/ATS\_Event\_Logs/Logstash:D:/Logs/csv\_files

![image](https://us1.discourse-cdn.com/elastic/original/3X/5/0/5063a05bca4bd5604043b93a216f4116ce7ead5a.png)

![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/707bbb1b567b206c0af61e1ab1c9b0c457bb3f74.png)

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 15, 2023, 5:03pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/15 "2023-11-15T17:03:34Z")

</div>

Yeah but I am a bit worried about all the `:` I am not on windows... but yes try that.

I would do a very simple test to load a file make sure the framework is there etc...

mounts and files are found etc

But yeah looking close to what you want... and muuuuuchhh simpler

Ohh wait !!!

That is wrong the docker is unix

Hold on lets try in the compose

`- D:/ATS_Event_Logs/Logstash:/logs/csv_files`

First part is `Windows Path to Real Files:/unix/path/to/directory`

then in the .conf its the UNIX path

`path => "/logs/csv_files/*.csv"`

Remember logstash in docker is actually running on Unix  
So all paths in the confs are unix paths that are mounted volumes to your real windows file system

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 5:13pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/16 "2023-11-15T17:13:13Z")

</div>

```auto
ethan777@LAPTOP-PESEJ3MG:/mnt/c/Users/ethan/OneDrive - Nanyang Technological University/NTU Documents/EEE/Y5S1/EE4080 FINAL YEAR PROJECT/docker-env/logstash-standard$ docker-compose up -d
validating /mnt/c/Users/ethan/OneDrive - Nanyang Technological University/NTU Documents/EEE/Y5S1/EE4080 FINAL YEAR PROJECT/docker-env/logstash-standard/docker-compose.yml: services.logstash Additional property paths is not allowed

```

I think the wsl terminal does not allow the extra line  
` paths: - D:/ATS_Event_Logs/Logstash:/logs/csv_files`

in `docker-compose.yml`

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 15, 2023, 5:14pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/17 "2023-11-15T17:14:04Z")

</div>

Your using a Cloud Drive?... I would use local only...

Can you show me the whole volumes section... in the compose

One volume line per line.

```auto
    volumes:
      - ./:/usr/share/logstash/pipeline/
      - D:/ATS_Event_Logs/Logstash:/logs/csv_files

```

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 5:16pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/18 "2023-11-15T17:16:02Z")

</div>

```auto
version: "3.8"

volumes:
  logstashdata01:
    driver: local

networks:
  default:
    name: elastic
    external: true # Says use the other network
    
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    environment:
      - xpack.monitoring.enabled=false
    volumes:
      - ./:/usr/share/logstash/pipeline/
    command: logstash -r -f /usr/share/logstash/pipeline/logstash.conf
    paths:
    - D:/ATS_Event_Logs/Logstash:/logs/csv_files
    ports:
      - "5044:5044"
    mem_limit: ${LS_MEM_LIMIT}

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 15, 2023, 5:17pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/19 "2023-11-15T17:17:03Z")

</div>

> [@Ethan777100](#):
>
> ```auto
> volumes:
> - ./:/usr/share/logstash/pipeline/
> 
> ```

```auto
    volumes:
      - ./:/usr/share/logstash/pipeline/
      - "D:\ATS_Event_Logs\Logstash:/logs/csv_files"

```

the `-` means a elements in a list

Could be a windows... path thing ... need to look at the docker docs

Think you have to Quote it

```
  - "D:\ATS_Event_Logs\Logstash:/logs/csv_files"

```

> <https://stackoverflow.com/questions/41334021/volume-binding-using-docker-compose-on-windows>

BUT I will tell you I am not a WSL person... are there Unix Paths if so use them... think you are going to need how to figure that out...

`/mnt/d/pathtofiles`

if you can figure out the UNIX paths that may be better... sorry not my area of expertise...

Looks like you have

`/mnt/c/Users/ethan/....`

and I have NO CLUE what effect one drive has on that... I would use a local drive... if it were me and I was testing... (you keep throwing me curveballs 🙂 )

Although that may work ... The error was basically saying bad syntax on this line

`paths: - D:/ATS_Event_Logs/Logstash:/logs/csv_files`

gotta run... I will check in later

---

<div class="post-metadata">

### Author: ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)
#### Post date: [November 15, 2023, 5:32pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/20 "2023-11-15T17:32:12Z")

</div>

Sorry abt the OneDrive and curveballs. But you really help me immensely idk how to Thank you.

I'm actually working from my Desktop, with a remote-in to my laptop where the Docker action is happening.

For the timebeing, in case I needed to access the same work space from my Desktop, I can do so via OneDrive.

Most of my school life I was using the University provided OneDrive to store all my files related to Uni.

Maybe later on when things stabilise can I just copy out the entire workspace into local C Drive and then start the docker-composes again? Shld still work right?

Anyway

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a659d38518e8032d20cb35ee7c91312b48fc46f0.png)

[Next page](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916.md?page=2)