# Unable to read the txt file through Logstash **URL:** \ **Category:** Logstash\ **Created:** [October 5, 2020, 11:44pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057 "2020-10-05T23:44:53Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 5, 2020, 11:44pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/1 "2020-10-05T23:44:54Z")
Hi Guys, I am trying to read sample EDI log from the below path using logstash .conf file C:\Users\skkoganti\Documents\log\EDI Log-2019-05-27.txt this is my sample input file format input { file { path =\> ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27"] start\_position =\> "beginning" sincedb\_path =\> "NULL" } } output { stdout {codec =\> rubydebug} elasticsearch { index=\> "edi\_logs" hosts =\> ["localhost:9200"] } } but i don't see any index created on my kibana. Can anyone please suggest me on this. logstash output after i ran 🙂 C:\Users\skkoganti\Documents\Elastic\elastic\_download\logstash-7.9.1\logstash-7.9.1\config\>logstash -f iphone.conf Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties [2020-10-05T19:38:20,405][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.9.1", "jruby.version"=\>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0\_261-b12 +indy +jit [mswin32-x86\_64]"} [2020-10-05T19:38:20,690][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-10-05T19:38:22,842][INFO][org.reflections.Reflections] Reflections took 50 ms to scan 1 urls, producing 22 keys and 45 values [2020-10-05T19:38:25,792][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2020-10-05T19:38:26,024][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [2020-10-05T19:38:26,096][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es\_version=\>7} [2020-10-05T19:38:26,103][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7} [2020-10-05T19:38:26,184][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [2020-10-05T19:38:26,293][INFO][logstash.outputs.elasticsearch][main] Using a default mapping template {:es\_version=\>7, :ecs\_compatibility=\>:disabled} [2020-10-05T19:38:26,352][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>1000, "pipeline.sources"=\>["C:/Users/skkoganti/Documents/Elastic/elastic\_download/logstash-7.9.1/logstash-7.9.1/config/iphone.conf"], :thread=\>"#\"} [2020-10-05T19:38:26,384][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s", "number\_of\_shards"=\>1}, "mappings"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}} [2020-10-05T19:38:27,279][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=\>0.92} [2020-10-05T19:38:27,870][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=\>"main"} [2020-10-05T19:38:27,946][INFO][filewatch.observingtail][main][5d1e9063f71372cd43b3d1a5773212aeb482e63ccd792a234be6d0096907cf78] START, creating Discoverer, Watch with file and sincedb collections [2020-10-05T19:38:27,952][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} [2020-10-05T19:38:28,321][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 6, 2020, 2:43am UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/2 "2020-10-06T02:43:29Z")
> [@sukumar.koganti](#): > > sincedb\_path =\> "NULL" On Windows that should be "NUL", on UNIX it should be "/dev/null" ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 12:54pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/3 "2020-10-06T12:54:25Z")
Thanks Badger, I changed to sincedb\_path =\> "NUL" but after ran again i am getting the same output. C:\Users\skkoganti\Documents\Elastic\elastic\_download\logstash-7.9.1\logstash-7.9.1\config\>logstash -f iphone.conf Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties [2020-10-06T08:49:01,367][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.9.1", "jruby.version"=\>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0\_261-b12 +indy +jit [mswin32-x86\_64]"} [2020-10-06T08:49:01,658][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-10-06T08:49:03,816][INFO][org.reflections.Reflections] Reflections took 46 ms to scan 1 urls, producing 22 keys and 45 values [2020-10-06T08:49:06,478][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2020-10-06T08:49:06,706][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [2020-10-06T08:49:06,766][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es\_version=\>7} [2020-10-06T08:49:06,772][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7} [2020-10-06T08:49:06,846][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [2020-10-06T08:49:06,912][INFO][logstash.outputs.elasticsearch][main] Using a default mapping template {:es\_version=\>7, :ecs\_compatibility=\>:disabled} [2020-10-06T08:49:06,981][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>1000, "pipeline.sources"=\>["C:/Users/skkoganti/Documents/Elastic/elastic\_download/logstash-7.9.1/logstash-7.9.1/config/iphone.conf"], :thread=\>"#\"} [2020-10-06T08:49:07,007][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s", "number\_of\_shards"=\>1}, "mappings"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}} [2020-10-06T08:49:07,933][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=\>0.93} [2020-10-06T08:49:08,529][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=\>"main"} [2020-10-06T08:49:08,594][INFO][filewatch.observingtail][main][ccc290222d51a93cf499be94ceda483818c7289c6274c84d9ca62971ff6a1691] START, creating Discoverer, Watch with file and sincedb collections [2020-10-06T08:49:08,597][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} [2020-10-06T08:49:08,951][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 1:28pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/4 "2020-10-06T13:28:01Z")
Hey Badger, it's working now thanks for your help. path =\> ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt"] after given the .txt at end of the file then only it's working. but i have one more question like i don't see column names on kibana whatever i had in my .txt file. This is my sample .txt file Time|Application|Message 2019-05-27 08:59:30|b\_GetOtherExcelData\_BEEM|Started 2019-05-27 08:59:33|b\_GetOtherExcelData\_BEEM|2019 Spring Bookings.xlsx Copied to \pneus.touchette\documents\EDI\BEEM\Development\Outbound\Wholesale\toSend 2019-05-27 08:59:33|b\_GetOtherExcelData\_BEEM|Successfully Complete 2019-05-27 13:07:12|ProcessContiGold\_contiGOLD|Started logstash output. [2020-10-06T08:57:04,017][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} [2020-10-06T08:57:04,436][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} { "path" =\> "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt", "message" =\> "2019-05-27 08:59:30|b\_GetOtherExcelData\_BEEM|Started\r", "host" =\> "GTAPIDEV", "@version" =\> "1", "@timestamp" =\> 2020-10-06T12:57:04.598Z } { "path" =\> "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt", "message" =\> "2019-05-27 13:23:42|ProcessContiGold\_contiGOLD|Started\r", "host" =\> "GTAPIDEV", "@version" =\> "1", "@timestamp" =\> 2020-10-06T12:57:04.602Z } { "path" =\> "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt", "message" =\> "2019-05-27 13:54:21|SendContiGOLD\_contiGOLD|Started\r", "host" =\> "GTAPIDEV", "@version" =\> "1", "@timestamp" =\> 2020-10-06T12:57:04.605Z =\> i want to read column names aswell, Can you help me badger. Thanks, Suresh Koganti. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 6, 2020, 1:44pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/5 "2020-10-06T13:44:20Z")
Have you tried using a csv filter? ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 1:51pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/6 "2020-10-06T13:51:23Z")
Yes i tried like below, but i am reading .txt file right will it work? input { file { path =\> ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt"] start\_position =\> "beginning" sincedb\_path =\> "NUL" } } filter { ``` csv{ separator => "," columns["Time","Application","Message"] #autodetect_column_names => true } } ``` output { stdout {codec =\> rubydebug} elasticsearch { index=\> "edi\_logs" hosts =\> ["localhost:9200"] } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 6, 2020, 2:28pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/7 "2020-10-06T14:28:01Z")
I would expect that to work. What happened when you tried it? ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 2:31pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/8 "2020-10-06T14:31:31Z")
when i tied this i am getting like below and i don't see my column names. ``` "@version" => "1", "host" => "GTAPIDEV", "path" => "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt", "2019-05-27 08:59:33|b_GetOtherExcelData_BEEM|2019 Spring Bookings.xlsx Copied to \\\\pneus.touchette\\documents\\EDI\\BEEM\\Development\\Outbound\\Wholesale\\toSend" => "2019-05-27 13:13:24|SendContiGOLD_contiGOLD|Started", "message" => "2019-05-27 13:13:24|SendContiGOLD_contiGOLD|Started\r", "@timestamp" => 2020-10-06T14:19:00.237Z ``` } { "@version" =\> "1", "host" =\> "GTAPIDEV", "path" =\> "C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt", "2019-05-27 08:59:33|b\_GetOtherExcelData\_BEEM|2019 Spring Bookings.xlsx Copied to \\pneus.touchette\documents\EDI\BEEM\Development\Outbound\Wholesale\toSend" =\> "2019-05-27 13:43:15|BuildData\_contiGOLD|Started", "message" =\> "2019-05-27 13:43:15|BuildData\_contiGOLD|Started\r", "@timestamp" =\> 2020 ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 6, 2020, 2:40pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/9 "2020-10-06T14:40:10Z")
> [@sukumar.koganti](#): > > ``` > "message" => "2019-05-27 13:13:24|SendContiGOLD_contiGOLD|Started\r", > > ``` Those are not comma separated values, so I do not understand why you would expect a csv filter to parse them. ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 2:48pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/10 "2020-10-06T14:48:35Z")
Yeah those are pipe (|) separated values and coming from the .txt file like below. **Time|Application|Message** **2019-05-27 08:59:30|b\_GetOtherExcelData\_BEEM|Started** **2019-05-27 08:59:33|b\_GetOtherExcelData\_BEEM|2019 Spring Bookings.xlsx Copied to \pneus.touchette\documents\EDI\BEEM\Development\Outbound\Wholesale\toSend** **2019-05-27 08:59:33|b\_GetOtherExcelData\_BEEM|Successfully Complete** **2019-05-27 13:07:12|ProcessContiGold\_contiGOLD|Started** **2019-05-27 13:07:12|BuildData\_contiGOLD|Started** ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 6, 2020, 2:54pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/11 "2020-10-06T14:54:00Z")
> [@sukumar.koganti](#): > > `separator => ","` Maybe you should try ``` separator => "|" ``` instead. ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 3:39pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/12 "2020-10-06T15:39:52Z")
Thanks Badger it's working now. ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 6, 2020, 4:46pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/13 "2020-10-06T16:46:42Z")
Hi Badger, again... 🙂 How to read the latest file based on date and time automatically.. input { file { path =\> ["C:/Users/skkoganti/Documents/log/EDI Log-2019-05-27.txt"] start\_position =\> "beginning" sincedb\_path =\> "NUL" ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [October 6, 2020, 4:59pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/14 "2020-10-06T16:59:35Z")
By default Logstash sorts the files by "last\_modified" attribute, you could change this behavior with the "file\_sort\_by" and "fiel\_sort\_direction" options. > **[File input plugin | Logstash Reference \[7.9\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-file_sort_by)** ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 7, 2020, 3:38pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/15 "2020-10-07T15:38:55Z")
> [@Iker](#): > > stash so Thanks Iker, it's working now. any idea how to fetch the files every week (7 days) from the path. **input** **{** **file** **{** \*_path =\> ["[//pneus.touchettedocuments/EDI/Logs/Preproduction/](https://pneus.touchettedocuments/EDI/Logs/Preproduction/)txt"]_ **start\_position =\> "beginning"** **sincedb\_path =\> "NUL"** **}** **}** **filter {** \*\* csv {\*\* \*\* separator =\> "|"\*\* \*\* columns =\> ["Time","Application","Message"]\*\* \*\* autodetect\_column\_names =\> true\*\* \*\* autogenerate\_column\_names =\> true\*\* \*\* file\_sort\_direction =\> ["asc"]\*\* \*\* }\*\* ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 7, 2020, 5:28pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/16 "2020-10-07T17:28:21Z")
Hi Badger, When i use this code i am getting below error. filter { csv { ``` separator => "|" columns => ["Time","Application","Message"] autodetect_column_names => true autogenerate_column_names => true ``` } mutate { ``` convert => { "Time" => "Date" } rename => { "Time" => "Date" } ``` } * * * C:\Users\skkoganti\Documents\Elastic\elastic\_download\logstash-7.9.1\logstash-7.9.1\config\>logstash -f edi\_development.conf Sending Logstash logs to C:/ELK/logstash-7.9.1/logstash-7.9.1/logs which is now configured via log4j2.properties [2020-10-07T13:25:47,702][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.9.1", "jruby.version"=\>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.261-b12 on 1.8.0\_261-b12 +indy +jit [mswin32-x86\_64]"} [2020-10-07T13:25:47,983][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-10-07T13:25:49,446][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "=\>" at line 25, column 8 (byte 445) after filter {\r\n csv {\r\n separator =\> "|"\r\n columns =\> ["Time","Application","Message"]\r\n autodetect\_column\_names =\> true\r\n autogenerate\_column\_names =\> true\r\n }\r\n mutate {\r\n\t convert =\> {\r\n "Time" =\> "Date"\r\n }\r\n\t rename =\> { "Time" =\> "Date" }\r\n}\r\noutput\r\n{\r\nstdout ", :backtrace=\>["C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/pipeline\_action/create.rb:52:in `execute'", "C:/ELK/logstash-7.9.1/logstash-7.9.1/logstash-core/lib/logstash/agent.rb:357:in `block in converge\_state'"]} [2020-10-07T13:25:49,862][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2020-10-07T13:25:54,621][INFO][logstash.runner] Logstash shut down. [2020-10-07T13:25:54,640][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 7, 2020, 5:29pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/17 "2020-10-07T17:29:41Z")
actually i am trying to convert the columns and along with the data type. mutate { ``` convert => { "Time" => "Date" } rename => { "Time" => "Date" } ``` } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 7, 2020, 6:25pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/18 "2020-10-07T18:25:17Z")
> [@sukumar.koganti](#): > > :message=\>"...mutate {\r\n\t convert =\> {\r\n "Time" =\> "Date"\r\n }\r\n\t rename =\> { "Time" =\> "Date" }\r\n}\r\noutput\r\n{\r\nstdout " You are missing a } to close the filter section. Also, mutate cannot do a "Date" conversion. You would need a date filter to parse a string into a date. ---
**Author:** ![sukumar.koganti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sukumar.koganti/32/74111_2.png) [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Post date:** [October 7, 2020, 7:22pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/19 "2020-10-07T19:22:07Z")
> [@Badger](#): > > date filter to parse a string into a date Hmmm, bit confusing with this can we write like this. filter { csv { separator =\> "|" columns =\> ["Time","Application","Message"] autodetect\_column\_names =\> true autogenerate\_column\_names =\> true } filter { date { convert =\> { "string" =\> "Date" } } mutate { rename =\> { "Time" =\> "Date" } } } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 7, 2020, 7:56pm UTC](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057/20 "2020-10-07T19:56:55Z")
You have to tell the date filter what the format of the string is. If yours is 2019-05-27 08:59:30 then you would use ``` date { match => ["Time", "YYYY-MM-dd HH:mm:ss"] } ``` [Next page](https://discuss.elastic.co/t/unable-to-read-the-txt-file-through-logstash/251057.md?page=2)