# Setting Up Logstash In Docker-Compose For Bulk Ingest Of CSV Files In Local Machine

**URL:** <https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [November 11, 2023, 5:16pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916 "2023-11-11T17:16:21Z")\
**Posts on this page:** 14\
**Page:** 6

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 22, 2023, 6:46pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/105 "2023-11-22T18:46:40Z")

</div>

Okay I don't know how each of the \_source ended up being different. This is crazy

I didn't save my data as .xlsx . I always ensured to never save my opened csv files if I viewed them thru Excel

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 22, 2023, 6:50pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/106 "2023-11-22T18:50:44Z")

</div>

> [@Ethan777100](#):
>
> Okay I don't know how each of the \_source ended up being different. This is crazy
> 
> I didn't save my data as .xlsx . I always ensured to never save my opened csv files if I viewed them thru Excel

That was me showing you an example to demonstrate that the `\r` definitely is a problem. I did 1 with the `\r` and one without..

That data is definitely coming in with `\r` from your CSV .. it is right there in your logstash debug above...

> [@Ethan777100](#):
>
> ```auto
> {
> "@timestamp" => 2023-11-22T15:43:21.000370288Z,
> "log" => {
> "file" => {
> "path" => "/usr/share/logstash/ats-logs-mainline/2023/01-Jan-23/events2023-01-01.csv"
> }
> },
> "@version" => "1",
> "message" => "54146161,OCC_54146161,OCC_0,DIAG_IAllDoorModeStatus[2],RSC,DRS,2023-01-01 00:00:34.672,null,0,>=1 OPEN,90390/RSC/DRS/ALL,90390,0,Summary of Train Doors Status with Open,5,0,-1,-1\r",
> "host" => {
> "name" => "41d44a8cb286"
> },
> "event" => {
> "original" => "54146161,OCC_54146161,OCC_0,DIAG_IAllDoorModeStatus[2],RSC,DRS,2023-01-01 00:00:34.672,null,0,>=1 OPEN,90390/RSC/DRS/ALL,90390,0,Summary of Train Doors Status with Open,5,0,-1,-1\r"
> }
> }
> 
> ```

Not Sure what is happening... but there is some "[entropy](https://en.wikipedia.org/wiki/Entropy)" going on 🙂

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 22, 2023, 7:00pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/107 "2023-11-22T19:00:01Z")

</div>

Before `description` keyword

```auto
2023-11-17 11:16:45 "host" => {
2023-11-17 11:16:45 "name" => "94e10b4ac1ae"
2023-11-17 11:16:45 },
2023-11-17 11:16:45 "log" => {
2023-11-17 11:16:45 "file" => {
2023-11-17 11:16:45 "path" => "/usr/share/logstash/csv_files/events2022-01-01.csv"
2023-11-17 11:16:45 }
2023-11-17 11:16:45 },
2023-11-17 11:16:45 "graphicelement" => "-1",
2023-11-17 11:16:45 "sourcetime" => "2022-01-01 01:03:31.885",
2023-11-17 11:16:45 "@timestamp" => 2023-11-17T03:16:38.290874500Z,
2023-11-17 11:16:45 "zone" => "-1",
2023-11-17 11:16:45 "equipment" => "90370/RSC/DRS/ALL",
2023-11-17 11:16:45 "subsystem" => "DRS",
2023-11-17 11:16:45 "system" => "RSC",
2023-11-17 11:16:45 "uniqueid" => "OCC_11668339",
2023-11-17 11:16:45 "operator" => "null",
2023-11-17 11:16:45 "alarmvalue" => "0",
2023-11-17 11:16:45 "alarm" => "OCC_0",
2023-11-17 11:16:45 "eventtype" => "DIAG_IAllDoorModeStatus[1]",
2023-11-17 11:16:45 "event" => {
2023-11-17 11:16:45 "original" => "11668339,OCC_11668339,OCC_0,DIAG_IAllDoorModeStatus[1],RSC,DRS,2022-01-01 01:03:31.885,null,0,CLOSED & LOCKED,90370/RSC/DRS/ALL,90370,0,Summary of Train Doors Status with Closed & Locked,5,0,-1,-1"
2023-11-17 11:16:45 },
2023-11-17 11:16:45 "message" => "11668339,OCC_11668339,OCC_0,DIAG_IAllDoorModeStatus[1],RSC,DRS,2022-01-01 01:03:31.885,null,0,CLOSED & LOCKED,90370/RSC/DRS/ALL,90370,0,Summary of Train Doors Status with Closed & Locked,5,0,-1,-1",
2023-11-17 11:16:45 "state" => "5",
2023-11-17 11:16:45 "severity" => "0",
2023-11-17 11:16:45 "id" => "11668339",
2023-11-17 11:16:45 "value" => "CLOSED & LOCKED",
2023-11-17 11:16:45 "location" => "90370",
2023-11-17 11:16:45 "description" => "Summary of Train Doors Status with Closed & Locked",
2023-11-17 11:16:45 "mmsstate" => "0",
2023-11-17 11:16:45 "@version" => "1"
2023-11-17 11:16:45 }

```

After `description` keyword (Now)

```auto
{
       "message" => "54161229,CCC_54161229,CCC_0,DIAG_ISkipStOdrRecAto,SIG,CTC,2023-01-01 01:51:51.515,null,0,SKIP ACTIVE,CC062/CCG/AAA/TDMS_SYS,90062,0,Skip ddd reported by AAA,5,0,-1,-1",  
          "host" => {
        "name" => "3ddc142e035e"
    },
           "log" => {
        "file" => {
            "path" => "/usr/share/logstash/ats-logs-mainline/2023/01-Jan-23/events2023-01-01.csv"
        }
    },
         "event" => {
        "original" => "54161229,CCC_54161229,CCC_0,DIAG_ISkipStOdrRecAto,SIG,CTC,2023-01-01 01:51:51.515,null,0,SKIP ACTIVE,CC062/CCG/AAA/TDMS_SYS,90062,0,Skip ddd reported by AAA,5,0,-1,-1" 
    },
    "@timestamp" => 2023-11-22T18:53:08.192485038Z,
      "@version" => "1"
}

```

Now I am getting traction to ingest files via original csv.

Curious but I noticed that now the file ingesting result on the console is different from how it was originally.

Is this because I made `description` keyword ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 22, 2023, 7:12pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/108 "2023-11-22T19:12:21Z")

</div>

> [@Ethan777100](#):
>
> ```auto
> 2023-11-17 11:16:45 "host" => {
> 2023-11-17 11:16:45 "name" => "94e10b4ac1ae"
> 2023-11-17 11:16:45 },
> 2023-11-17 11:16:45 "log" => {
> 2023-11-17 11:16:45 "file" => {
> 2023-11-17 11:16:45 "path" => "/usr/share/logstash/csv_files/events2022-01-01.csv"
> 2023-11-17 11:16:45 }
> 2023-11-17 11:16:45 },
> 2023-11-17 11:16:45 "graphicelement" => "-1",
> 2023-11-17 11:16:45 "sourcetime" => "2022-01-01 01:03:31.885",
> 2023-11-17 11:16:45 "@timestamp" => 2023-11-17T03:16:38.290874500Z,
> 2023-11-17 11:16:45 "zone" => "-1",
> 2023-11-17 11:16:45 "equipment" => "90370/RSC/DRS/ALL",
> 2023-11-17 11:16:45 "subsystem" => "DRS",
> 2023-11-17 11:16:45 "system" => "RSC",
> 2023-11-17 11:16:45 "uniqueid" => "OCC_11668339",
> 2023-11-17 11:16:45 "operator" => "null",
> 2023-11-17 11:16:45 "alarmvalue" => "0",
> 2023-11-17 11:16:45 "alarm" => "OCC_0",
> 2023-11-17 11:16:45 "eventtype" => "DIAG_IAllDoorModeStatus[1]",
> 2023-11-17 11:16:45            
> 
> ```

So if you are seeing this out of logstash **it means you still have the filter section in the logstash.conf because logstsash SHOULD not be doing the CSV**... the ingest pipeline it... I think you are loading old code... NOTE My logsstash.conf.

```auto
input { 
    file { 
        path => "/usr/share/logstash/csv_files/events2023-06-01.csv"
        start_position => "beginning" 
        sincedb_path => "/dev/null"
    } 
}

output { 
    elasticsearch {
        index => "ats-events-2023-06" 
        hosts => ["https://es01:9200"]
        user => "elastic"
        password => "mypassword"
        pipeline => "ats-events-pipeline"
        ssl_verification_mode=> "none"
    }
    stdout{ codec => "rubydebug"} 
}

```

and yes the output of logstash should like...

```auto
logstash-logstash-1 | {
logstash-logstash-1 | "@version" => "1",
logstash-logstash-1 | "event" => {
logstash-logstash-1 | "original" => "3432482,CCC_3432482,CCC_0,DIAG_IHndovr1Radio,SSS,ATC,2023-06-01 00:00:04.614,null,0,ALARM,xx022/SSS/ATC/AAA_SYS,xx022,0,Auto Norm: performed handover with one radio only,5,0,-1,-1"
logstash-logstash-1 | },
logstash-logstash-1 | "message" => "3432482,CCC_3432482,CCC_0,DIAG_IHndovr1Radio,SSS,ATC,2023-06-01 00:00:04.614,null,0,ALARM,xx022/SSS/ATC/AAA_SYS,xx022,0,Auto Norm: performed handover with one radio only,5,0,-1,-1",
logstash-logstash-1 | "host" => {
logstash-logstash-1 | "name" => "6e71e0dd4966"
logstash-logstash-1 | },
logstash-logstash-1 | "log" => {
logstash-logstash-1 | "file" => {
logstash-logstash-1 | "path" => "/usr/share/logstash/csv_files/events2023-06-01.csv"
logstash-logstash-1 | }
logstash-logstash-1 | },
logstash-logstash-1 | "@timestamp" => 2023-11-22T19:08:27.540865711Z
logstash-logstash-1 | }

```

NOTE : Unparsed, because the parsing happens in the ingest pipeline...

**Do you understand that we are NOT using logstash to parse the CSV...**

**The logstash simpl reads the file line by line and sends the `message` fields (and some others) to elasicsearch and the ingest pipeline running in elasticsearch parses the CSV...**

I am not sure what you are doing... I have given you the exact code ... each time...

Gotta run... good luck.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 22, 2023, 7:14pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/109 "2023-11-22T19:14:08Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/f/4/f40788357486fb7203fe5a7af0bf11622f509bc1.png)

I swear by it. I was never on the filters the past few hours.

Thanks for the tip

```auto
{
    "@timestamp" => 2023-11-22T19:13:06.671293416Z,
          "host" => {
        "name" => "9ad435a480de"
    },
       "message" => "54147799,OCC_54147799,OCC_1442517,ALARM_NORMALIZED,RSC,ATI,2023-01-01 00:12:05.168,null,1,NORMAL,90712/RSC/ATI,90712,3,ATI System Mode,2,4,-1,-1\r",
      "@version" => "1",
         "event" => {
        "original" => "54147799,OCC_54147799,OCC_1442517,ALARM_NORMALIZED,RSC,ATI,2023-01-01 00:12:05.168,null,1,NORMAL,90712/RSC/ATI,90712,3,ATI System Mode,2,4,-1,-1\r"
    },
           "log" => {
        "file" => {
            "path" => "/usr/share/logstash/ats-logs-mainline/2023/01-Jan-23/events2023-01-01-test.csv"
        }
    }
}

```

I can confirm the root cause is due to the stupid

> The addition of the extra `\r` (carriage return) in the CSV file might have occurred due to the default behavior of the `csv.writer()` method when writing rows using the `csv` module.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 23, 2023, 4:39am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/110 "2023-11-23T04:39:38Z")

</div>

Now spending time trying to improve on my python script such that \r is not appended at the end of every row.

pandas always does this.

do u know of any other library that doesn't append \r ?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [November 23, 2023, 7:40am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/111 "2023-11-23T07:40:00Z")

</div>

You can use regex to replace or gub in LS.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 23, 2023, 5:01pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/112 "2023-11-23T17:01:40Z")

</div>

I found this from [CSV Filter - Quote character causing \_csvparsefailure - Elastic Stack / Logstash - Discuss the Elastic Stack](https://discuss.elastic.co/t/csv-filter-quote-character-causing-csvparsefailure/121273)

[Mutate filter plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)

Here is my improvising in my use-case.

I wrote this in logstash config. `"graphicelement"` is where I have the` \r` and I replace it with nothing hence `""`.

```auto
input { 
    file { 
        path => "/usr/share/logstash/ats-logs-mainline/2023/01-Jan-23/formatted_events2023-01-01.csv"
        start_position => "beginning" 
    } 
    mutate {
            gsub => ["graphicelement", "\r", ""]
       }
}

```

Oh no I need help to install my mutate filter plugin. I'm not sure how to do it I just read the readme. My use case is within docker environment.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 23, 2023, 5:11pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/113 "2023-11-23T17:11:56Z")

</div>

> [@Ethan777100](#):
>
> Oh no I need help to install my mutate filter plugin.

The mutate filter is bundled by default, you do not need to install it. Note, however, that because you are parsing the csv in the ingest pipeline the [graphicelelement] field will not exist in logstash, you will need to mutate the [message] field.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 23, 2023, 5:16pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/114 "2023-11-23T17:16:46Z")

</div>

Edited to [message] - makes sense. Coz we are doing the mutate on the logstash output to which the json output has the `message: xxxx`

```auto
[2023-11-23T17:15:13,140][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"mutate"}

```

I think I'm putting the mutate json in the wrong place.

I actually have pipeline template where the filtering happens. Am i supposed to put it there?

EDIT: console doesn't allow.

Correct. I logically reasoned `filter.`

Thanks tho. Do bear with me. Sometimes still clueless on where certain parts are supposed to go (to which files?)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 23, 2023, 5:22pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/115 "2023-11-23T17:22:51Z")

</div>

Indeed, the mutate filter needs to be in the filter section, not the input section.

```
input { 
    file { 
        path => "/usr/share/logstash/ats-logs-mainline/2023/01-Jan-23/formatted_events2023-01-01.csv"
        start_position => "beginning" 
    } 
}
filter { 
    mutate {
            gsub => ["message", "\r", ""]
       }
}

```

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 23, 2023, 5:48pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/116 "2023-11-23T17:48:47Z")

</div>

```auto
[2023-11-23T17:46:56,709][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"{\" at line 10, column 7 (byte 256) after filter", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}

```

Now I get this.

EDIT: My bad. I wrote `filters` and not `filter` **strong text**

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 23, 2023, 5:55pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/117 "2023-11-23T17:55:14Z")

</div>

> [@Ethan777100](#):
>
> `Expected one of [\\t\\r\\n], \"#\", \"{\" at line 10, column 7 (byte 256) after filter"`

OK, so logstash is objecting to whatever comes immediately after the word filter in your configuration. So what does the configuration look like?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2023, 5:58pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/119 "2023-12-21T17:58:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916.md?page=5)