# Setting Up Logstash In Docker-Compose For Bulk Ingest Of CSV Files In Local Machine

**URL:** <https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [November 11, 2023, 5:16pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916 "2023-11-11T17:16:21Z")\
**Posts on this page:** 20\
**Page:** 4

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 20, 2023, 5:27pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/63 "2023-11-20T17:27:00Z")

</div>

Cool. These are really the small hacks not mentioned in Documentation.

This is essentially for ppl like me who don't know how to create or modify a mapping or pipeline from scratch.

So the hack is to upload a csv using the provided UI, get the mapping and pipeline created inside Elastic.

Then use the logstash container to ingest the rest of the files.

PS: I'm closely tracking my **ext4.vhdx** file size.  
10.5GB -\> 14.46 GB at 1.2million rows.  
For 1 month of logs, I'm looking at ~4.122 million rows. (assuming 137.4k rows per csv file)

Guess it now beats the question : why was the default schema that I used earlier on just "inflated" and "bulky" when the current mapping and pipelines are honestly more streamlined?

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 20, 2023, 6:11pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/64 "2023-11-20T18:11:06Z")

</div>

The ingestion stopped for unknown reason.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b679ed3fef0220cfa0f968666e33510b54390088.png)

The logs ingested was from 01-June-2023 08:00:00 to 12-June-2023 02:55:00.

My full range of logs is 01-June-23 00:00:00 to 30-June-23 23:59:59  
I'm not sure whats going on. My logstash docker is still running, no errors raised.

Nothing raised in ES and Kib containers as well.

Oh no. My disk is still growing despite not ingesting anything.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/5/65c8c788c2a2e0369f4a219bb3b7adeac6c37d6d.png)

Restarted logstash several times. The Logs are now completely empty and smth is fishy. The pipeline doesn't even start upon container startup.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/5/25444b79616b5b1bd96c05b0f23abec213fdb0c1.png)

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 20, 2023, 7:25pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/65 "2023-11-20T19:25:50Z")

</div>

I also learnt from the past 1 hour that when logstash is restarted, the pipeline re-reads everything into my index.

Now, this cycle of parsing has caused a duplicate of rows for the data `01-June-2023 08:00:00 to 12-June-2023 02:55:00.`

This time round logstash continued ingesting files beyond 12-June-2023 02:55:00 (smooth-sailing case).

How can I delete off the 1st section of logs?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/f/dfcf325c7eac0adaafead603d75c12a0dc118c49.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 20, 2023, 7:35pm UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/66 "2023-11-20T19:35:21Z")

</div>

> [@Ethan777100](#):
>
> I also learnt from the past 1 hour that when logstash is restarted, the pipeline re-reads everything into my index.

We discussed this

> [@stephenb](#):
>
> ` sincedb_path => "/dev/null"`

take that out... that resets "logstash" memory of what it has read already...

> [@Ethan777100](#):
>
> How can I delete off the 1st section of logs?

I think this is where you start to read the docs...

> **[Delete by query API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html)**

But me... I would be loading the data in "batches" and putting them in indcice like

`ats-logs-2022-01`  
`ats-logs-2022-02`

Then create an Data view .. `ats-logs-*` that way I would have more control... and could delete and reload "sections" of the data better than putting in all the data in a single index.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 5:45am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/67 "2023-11-21T05:45:09Z")

</div>

```auto
2023-11-21 13:36:50 [2023-11-21T05:36:50,943][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-11-21 13:36:52 [2023-11-21T05:36:52,189][INFO][org.reflections.Reflections] Reflections took 252 ms to scan 1 urls, producing 132 keys and 464 values
2023-11-21 13:36:52 [2023-11-21T05:36:52,727][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:36:52 [2023-11-21T05:36:52,743][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:516)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:36:56 [2023-11-21T05:36:56,136][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:36:56 [2023-11-21T05:36:56,143][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:36:59 [2023-11-21T05:36:59,065][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:36:59 [2023-11-21T05:36:59,068][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:37:02 [2023-11-21T05:37:02,163][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:37:02 [2023-11-21T05:37:02,166][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:37:04 [2023-11-21T05:37:04,996][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:37:04 [2023-11-21T05:37:04,998][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:37:07 [2023-11-21T05:37:07,967][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:37:07 [2023-11-21T05:37:07,969][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:37:11 [2023-11-21T05:37:11,000][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:37:11 [2023-11-21T05:37:11,004][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:37:13 [2023-11-21T05:37:13,995][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:37:13 [2023-11-21T05:37:13,996][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
2023-11-21 13:37:16 [2023-11-21T05:37:16,946][ERROR][logstash.outputs.elasticsearch] Unknown setting 'mapping' for elasticsearch
2023-11-21 13:37:16 [2023-11-21T05:37:16,948][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}

```

decided to add a pipeline and mapping directory to the json files I obtained from the mapping and ingest pipeline.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0b72c01de513f1443093f107b2d64e228de82dc2.png)

Not sure if this is the right format. `logstash.conf`

```auto
output { 
    elasticsearch { 
        index => "ats-mainline-logs-2023-06" 
        hosts => ["https://es01:9200"]
        manage_template => false
        user => "elastic"
        password => "elastic123"
        ssl_verification_mode=> "none"
        pipeline => "./ats-logs-mainline/ats-logs-ingest pipeline.json"
        mapping => "./ats-logs-mainline/ats-logs-mapping.json"

```

Regarding

### [Delete by query API | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html)

Is there a way to delete rows based on a range of timestamp? I tried to do that but the format isn't right.

format is `1/6/2023 00:00:00` - but as you can see, the display format is just different

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4b75462c8ae735b2a5b825378f83a68629ca943f.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:10am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/68 "2023-11-21T06:10:06Z")

</div>

> [@Ethan777100](#):
>
> ```auto
> pipeline => "./ats-logs-mainline/ats-logs-ingest pipeline.json"
> mapping => "./ats-logs-mainline/ats-logs-mapping.json"
> 
> ```

That is not correct, that is not how it works... You can't just point to a json file for the pipeline and mapping.

Like on this page then use its name in the logstash conf

> **[Ingest pipelines | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html)**

In the Dev Tools  
You need to PUT the ingest pipeline

For the mapping you need to PUT template

> **[Index templates | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html)**

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 6:17am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/69 "2023-11-21T06:17:02Z")

</div>

Meanwhile, I'm having this weird behaviour where

My csv files are being ingested as per usual.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af4f8745be3ddf4f96a8508a6e4730d5b1469b84.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/9/19553af2270711122ad18e548c099c8e2b84fd31.png)

The index is yet not growing with files.

This was an index I created using file upload, then with Console API to delete the 137000+ rows of data. to prevent duplicate entries for 1 June 2023.

Now I start the logstash container to ingest the data as per the pipeline and mapping json that was generated.

I checked double and triple that all my pointers and namings have been correct before starting the logstash container.

Idk where my data is being piped to now.

At the same time,

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86378407dea6d7a73a6ed5157a5744b98e05fefc.png)

I'm getting quite a few bad rows who cannot be ingested into Elastic. Which is weird because i thought - we got the mapping and pipeline configs right ?

The log output is flashing by too fast to copy just in time

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:20am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/70 "2023-11-21T06:20:34Z")

</div>

Go to Dev Tools

`GET _cat/indices?v`

Pasting images of text sigh...

But the error says it can not index because date time format exception

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 6:22am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/71 "2023-11-21T06:22:18Z")

</div>

> [@stephenb](#):
>
> GET \_cat/indices?v

```auto
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green open .internal.alerts-observability.logs.alerts-default-000001 O54Ee8Z_TaeI4TSNYbPXwQ 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.uptime.alerts-default-000001 yg2Q9o8KR_a03cKGBqx6Dw 1 0 0 0 249b 249b 249b
green open .internal.alerts-ml.anomaly-detection.alerts-default-000001 voKYtP5LRC23_X-skrAOfg 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.slo.alerts-default-000001 eHQPEP6QTx6O1f1rPVko4A 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.apm.alerts-default-000001 AgnDZUQ9RPS2csA4MUkL7g 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.metrics.alerts-default-000001 WKqpRWVtRtitYRdyFolR8g 1 0 0 0 249b 249b 249b
green open .kibana-observability-ai-assistant-conversations-000001 3MzK0r5rT72EKYePE2meOw 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.threshold.alerts-default-000001 TBSeIwJ3RFCGvKjYhgJ34Q 1 0 0 0 249b 249b 249b
yellow open ats-mainline-logs-2023-06 UnuGNRFKTsGI85YILs-scA 1 1 0 0 2.2mb 2.2mb 2.2mb
green open .kibana-observability-ai-assistant-kb-000001 TJIEehIuSJebmgUrV9VUzA 1 0 0 0 249b 249b 249b
green open .internal.alerts-security.alerts-default-000001 7nSFEQuwTwqlh6OXmg-b8Q 1 0 0 0 249b 249b 249b
green open .internal.alerts-stack.alerts-default-000001 R854JGu-TsSo4ivd04w4zw 1 0 0 0 249b 249b 249b

```

The pipeline file

```auto
    {
      "date": {
        "field": "sourcetime",
        "timezone": "Asia/Singapore",
        "formats": [
          "yyyy-MM-dd HH:mm:ss.SSS",
          "yyyy-MM-dd HH:mm:ss.SS",
          "yyyy-MM-dd HH:mm:ss",
          "yyyy-MM-dd HH:mm:ss.S"
        ]
      }
    },

```

And mapping file

```auto
     "sourcetime": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss.SS||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.S"

```

So many formats were already accounted for and detected from File Upload tho.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:25am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/72 "2023-11-21T06:25:48Z")

</div>

It's failing and not ingesting the data

I can't really see / read the error because you pasted and image... But the error tells you what the issue is...

Bad timezone or something  
.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 6:26am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/73 "2023-11-21T06:26:52Z")

</div>

However some rows appear to be passing through.

Yeah i noticed this this morning. Was wondering if this could explain why my data only ingested from 08:00:00 instead of 00:00:00 ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:27am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/74 "2023-11-21T06:27:17Z")

</div>

> [@Ethan777100](#):
>
> `"timezone": "Asia/Singapore",`

That is probably not right..  
Probably `SST` ... the 3 letter code

Hmmm seems like what you have should work too.

Take out timezone and test .. the .fix

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 6:34am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/75 "2023-11-21T06:34:01Z")

</div>

SST ? 3-letter country code in accordance with ISO 3166-1 alpha-3?

Do you mean SGP - Singapore?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:35am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/76 "2023-11-21T06:35:07Z")

</div>

[Singapore Time - Wikipedia](https://en.m.wikipedia.org/wiki/Singapore_Time#:~:text=Singapore%20Time%20(SGT)%2C%20also,UTC%2B08%3A00)).

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 6:48am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/77 "2023-11-21T06:48:50Z")

</div>

the pipeline config file was generated by File Upload tho.

I didn't define "timezone".

So it looks like the Elastic-generated files may not be fully accurate to my needs? I thought i had my breakthrough.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:53am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/78 "2023-11-21T06:53:56Z")

</div>

Or there is some other error.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 21, 2023, 6:55am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/79 "2023-11-21T06:55:08Z")

</div>

Wait... What does your log stash config look like...

Did you leave the CSV filter in there?

Or take it out? The ingest pipeline does the CSV parsing. Are you trying to do both?

If you're using the ingest pipeline, you should no longer be using the CSV part in logstash conf

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 6:56am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/80 "2023-11-21T06:56:26Z")

</div>

```auto
input { 
    file { 
        path => "/usr/share/logstash/ats-logs-mainline/2023/06-Jun-23/events2023-06-*.csv"
        start_position => "beginning" 
    } 
}

filter { 
    csv { 
        separator => ","
        columns => ["id","uniqueid","alarm","eventtype","system","subsystem","sourcetime","operator","alarmvalue","value","equipment","location","severity","description","state","mmsstate","zone","graphicelement"] 
    } 
}

output { 
    elasticsearch { 
        index => "ats-mainline-logs-2023-06" 
        hosts => ["https://es01:9200"]
        manage_template => false
        user => "elastic"
        password => "elastic123"
        ssl_verification_mode=> "none"
        pipeline => "ats-logs-mainline-pipeline"
    }
    stdout{ codec => "rubydebug"} 
}

```

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 7:18am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/81 "2023-11-21T07:18:09Z")

</div>

```auto
{
          "host" => {
        "name" => "d682149381e4"
    },
         "event" => {
        "original" => "7164267,OCC_7164267,OCC_0,DIAG_IAllDoorModeStatus[1],RSC,DRS,2023-06-01 08:03:23.223,null,0,NOT CLOSED & LOC,xxx/RSC/DRS/ALL,xxx,0,Summary of Train Doors Status with Closed & Locked,5,0,-1,-1"
    },
       "message" => "7164267,xxx_7164267xxx0,DIAG_IAllDoorModeStatus[1],RSC,DRS,2023-06-01 08:03:23.223,null,0,NOT CLOSED & LOC,xxx/RSC/DRS/ALL,xxxx,0,xxxx,5,0,-1,-1",
    "@timestamp" => 2023-11-21T07:10:22.735149884Z,
           "log" => {
        "file" => {
            "path" => "/usr/share/logstash/ats-logs-mainline/2023/06-Jun-23/events2023-06-01.csv"
        }
    },
      "@version" => "1"

```

Ingesting screws up again after I removed the filter component.

---

<div class="post-metadata">

**Author:** ![Ethan777100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethan777100/32/70366_2.png) [@Ethan777100](https://discuss.elastic.co/u/Ethan777100)\
**Post date:** [November 21, 2023, 8:51am UTC](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916/82 "2023-11-21T08:51:47Z")

</div>

> [@stephenb](#):
>
> GET \_cat/indices?v

```auto
yellow open ats-mainline-logs-2023-06 DOaSsm34RfS6It7t6E5dnA 1 1 3614158 0 1.1gb 1.1gb 1.1gb

```

1st month of logs ingested.

[Previous page](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916.md?page=3)

[Next page](https://discuss.elastic.co/t/setting-up-logstash-in-docker-compose-for-bulk-ingest-of-csv-files-in-local-machine/346916.md?page=5)