# Logstash failing to convert from csv to json and to forward to opensearch **URL:** \ **Category:** Logstash\ **Created:** [October 23, 2022, 4:28am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264 "2022-10-23T04:28:17Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![D\_Go](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_go/32/112417_2.png) [@D\_Go](https://discuss.elastic.co/u/D_Go)\ **Post date:** [October 23, 2022, 4:28am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/1 "2022-10-23T04:28:17Z")
Hi, I've been at this for a while and cant seem to load a csv file to AWS opensearch from logstash. Any help will be appreciated. """ [ERROR] 2022-10-23 04:09:20.063 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [A-Za-z0-9\_-], [\t\r\n], "#", "=\>" at line 20, column 20 (byte 528) after output {\n\telasticsearch {\n \t\thosts =\> ["[https://vpc-testdomain-recasfbwzdjs34opg4u1l6ul5skrke4.us...](https://vpc-testdomain-recasfbwzdjs34opg4u1l6ul5skrke4.us-east-2.es.amazonaws.com/?fbclid=IwAR2JZUnxs-9fWM0D5MCt3jx6mbRSS1DplBYUHRzQJwOy62yXg3sGWrul9f0)"]\n \t\tindex =\> "billing-log"\n \t\tdocument\_type =\> "CSV"\n \t\tuser =\> "useradmin"\n ilm", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:210:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge\_state'"]} [INFO] 2022-10-23 04:09:20.157 [LogStash::Runner] runner - Logstash shut down. """ If I disable or set line to true then I get:: [ERROR] 2022-10-23 04:22:09.417 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [A-Za-z0-9\_-], [\t\r\n], "#", "=\>" at line 20, column 20 (byte 528) after output {\n\telasticsearch {\n \t\thosts =\> ["https://[https://vpc-testdomain-recasfbwzdjs34opg4u1l6ul5skrke4.us...](https://vpc-testdomain-recasfbwzdjs34opg4u1l6ul5skrke4.us-east-2.es.amazonaws.com/?fbclid=IwAR00u4tks43klqjOILmPsA0271ySR7q3IJRRF30OyrtZp4nzOl1IzHd3eEQ)"]\n \t\tindex =\> "billing-log"\n \t\tdocument\_type =\> "CSV"\n \t\tuser =\> "useradmin"\n ilm", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:210:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge\_state'"]} [INFO] 2022-10-23 04:22:09.549 [LogStash::Runner] runner - Logstash shut down. csv as: Month,Redshift,Relational Database Service,EC2-Instances,EC2-Other,Total 2/1/22,"$4,300 ","$7,000 ","$7,000 ","$7,000 ","$116,300 " 3/1/22,"$29,353 ","$12,325 ","$4,494 ","$3,299 ","$55,092 " 4/1/22,"$4,692 ","$2,025 ",$693 ,$257 ,"$8,240 " 5/1/22,"$4,865 ","$2,083 ",$727 ,$258 ,"$8,533 " 6/1/22,"$4,866 ","$2,025 ",$710 ,$262 ,"$8,472 " 7/1/22,"$5,033 ","$2,083 ",$747 ,$273 ,"$8,859 " 8/1/22,"$5,033 ","$2,083 ",$772 ,$272 ,"$9,478 " 9/1/22,"$4,864 ","$2,025 ",$846 ,"$1,976 ","$11,511 " ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 23, 2022, 4:28am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/2 "2022-10-23T04:28:17Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![D\_Go](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_go/32/112417_2.png) [@D\_Go](https://discuss.elastic.co/u/D_Go)\ **Post date:** [October 23, 2022, 5:04am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/3 "2022-10-23T05:04:58Z")
Thank you! ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 23, 2022, 5:20am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/4 "2022-10-23T05:20:27Z")
And also that error generally indicates a syntax error in your logstash conf file which you did not share Plus if you want to write to Opensearch you will need to use their plugin. > **[GitHub - opensearch-project/logstash-output-opensearch: A Logstash plugin...](https://github.com/opensearch-project/logstash-output-opensearch)** > > A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index. - GitHub - opensearch-project/logstash-output-opensearch: A Logstash plugin that sends event data to a OpenS... With the OSS version of logstash > **[Download Logstash Free | Get Started Now](https://www.elastic.co/downloads/logstash-oss)** > > Download Logstash or the complete Elastic Stack (formerly ELK stack) for free and start collecting, searching, and analyzing your data with Elastic in minutes. ---
**Author:** ![D\_Go](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_go/32/112417_2.png) [@D\_Go](https://discuss.elastic.co/u/D_Go)\ **Post date:** [October 24, 2022, 1:22pm UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/5 "2022-10-24T13:22:38Z")
Stephen, I’ve tried to follow AWS documentation and obviously i’m missing more things. Thank you for the reply and information. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 24, 2022, 2:44pm UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/6 "2022-10-24T14:44:00Z")
If you show us your logstash pipeline configuration file we might be able help with the first error / the syntax error... Also perhaps you should just try the Free / Open Basic Elasticsearch 🙂 ---
**Author:** ![D\_Go](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_go/32/112417_2.png) [@D\_Go](https://discuss.elastic.co/u/D_Go)\ **Post date:** [October 27, 2022, 3:58am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/7 "2022-10-27T03:58:29Z")
Stephen, I appreciate the assistance. Config- file [ec2-user@ip-172-30-2-251 conf.d]$ cat csvjson.conf ```auto 1 input { 2 file { 3 path => "/home/ec2-user/costs101722_csv1.csv" 4 start_position => "beginning" 5 sincedb_path => "NULL" 6 } 7 } 8 filter { 9 csv { 10 columns => ["Month","Redshift","Relational Database Service","EC2-Instances","EC2-Other","Total"] 11 separator => "," 12 } 13 } 14 output { 15 elasticsearch { 16 hosts => ["https://vpc-testdomain-recbwzdjsopgulee6ul5skrke4.us-east-2.es.amazonaws.com:443"] 17 index => "billing-log" 18 document_type => "CSV" 19 user => "dgo" 20 ilm.enabled => false 21 ssl => true 22 password => "4utoknow" 23 } 24 } ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 27, 2022, 4:17am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/8 "2022-10-27T04:17:17Z")
What exact version of logstash? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 27, 2022, 4:20am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/9 "2022-10-27T04:20:36Z")
> [@D\_Go](#): > > `ilm.enabled => false` Should be `ilm_enabled => false` That is your syntax error of course that will not still work for opensearch, But that's the syntax error. So beside that, you're still going to need to use the correct plugins etc. Which you're going to need to work with on that opensearch forum or you can switch over to elasticsearch. ---
**Author:** ![D\_Go](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d_go/32/112417_2.png) [@D\_Go](https://discuss.elastic.co/u/D_Go)\ **Post date:** [October 27, 2022, 4:51am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/10 "2022-10-27T04:51:10Z")
Stephen, Im running Logstash 8.4.3. I've switched to True as per AWS documentation but now it's failing on the compatibly error. I appreciate the assistance here. That kinda change things. Im no longer getting block in converge\_state and instead is giving back again "Could not connect to a compatible version of Elasticsearch" 022-10-27 04:49:23.194 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline\_id=\>"main", :exception=\>#\, :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:245:in `block in healthcheck!'" Im gonna create a new domain and elect 7.10 instead of 1.3. Attempt to configure compatible versions. Logstash 8.4.3 and Elasticsearch 7.10 are compatible? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 27, 2022, 4:51am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/11 "2022-10-27T04:51:10Z")
Elasticsearch 7.10 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 27, 2022, 5:35am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/12 "2022-10-27T05:35:54Z")
> [@D\_Go](#): > > Logstash 8.4.3 and Elasticsearch 7.10 are compatible? Logstash 8.4.3 is not compatible with Elasticsearch 7.10, the last Logstash that is compatible with 7.10 is version 7.17, but there is a catch. The `elasticsearch` output on any Logstash higher than 7.12, if I'm not wrong, will check if the Elasticsearch has a Elastic license, free or paid, they won't work with Elasticsearch 7.10 OSS or any fork like OpenDistro/Opensearch. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 27, 2022, 5:35am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/13 "2022-10-27T05:35:54Z")
Elasticsearch 7.10 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 24, 2022, 5:35am UTC](https://discuss.elastic.co/t/logstash-failing-to-convert-from-csv-to-json-and-to-forward-to-opensearch/317264/14 "2022-11-24T05:35:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.