Logstash failing to convert from csv to json and to forward to opensearch

Elastic Stack Logstash
1

Hi,

I've been at this for a while and cant seem to load a csv file to AWS opensearch from logstash. Any help will be appreciated.

"""

[ERROR] 2022-10-23 04:09:20.063 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 20, column 20 (byte 528) after output {\n\telasticsearch {\n \t\thosts => ["https://vpc-testdomain-recasfbwzdjs34opg4u1l6ul5skrke4.us..."]\n \t\tindex => "billing-log"\n \t\tdocument_type => "CSV"\n \t\tuser => "useradmin"\n ilm", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:210:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}

[INFO ] 2022-10-23 04:09:20.157 [LogStash::Runner] runner - Logstash shut down.

"""

If I disable or set line to true then I get::

[ERROR] 2022-10-23 04:22:09.417 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 20, column 20 (byte 528) after output {\n\telasticsearch {\n \t\thosts => ["https://https://vpc-testdomain-recasfbwzdjs34opg4u1l6ul5skrke4.us..."]\n \t\tindex => "billing-log"\n \t\tdocument_type => "CSV"\n \t\tuser => "useradmin"\n ilm", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:210:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}

[INFO ] 2022-10-23 04:22:09.549 [LogStash::Runner] runner - Logstash shut down.

csv as:
Month,Redshift,Relational Database Service,EC2-Instances,EC2-Other,Total
2/1/22,"$4,300 ","$7,000 ","$7,000 ","$7,000 ","$116,300 "
3/1/22,"$29,353 ","$12,325 ","$4,494 ","$3,299 ","$55,092 "
4/1/22,"$4,692 ","$2,025 ",$693 ,$257 ,"$8,240 "
5/1/22,"$4,865 ","$2,083 ",$727 ,$258 ,"$8,533 "
6/1/22,"$4,866 ","$2,025 ",$710 ,$262 ,"$8,472 "
7/1/22,"$5,033 ","$2,083 ",$747 ,$273 ,"$8,859 "
8/1/22,"$5,033 ","$2,083 ",$772 ,$272 ,"$9,478 "
9/1/22,"$4,864 ","$2,025 ",$846 ,"$1,976 ","$11,511 "

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

Thank you!

4

And also that error generally indicates a syntax error in your logstash conf file which you did not share

Plus if you want to write to Opensearch you will need to use their plugin.

With the OSS version of logstash

1 Like
5

Stephen,

I’ve tried to follow AWS documentation and obviously i’m missing more things.

Thank you for the reply and information.

6

If you show us your logstash pipeline configuration file we might be able help with the first error / the syntax error...

Also perhaps you should just try the Free / Open Basic Elasticsearch :slight_smile:

7

Stephen,

I appreciate the assistance.

Config- file
[ec2-user@ip-172-30-2-251 conf.d]$ cat csvjson.conf

1  input {
     2          file {
     3                  path => "/home/ec2-user/costs101722_csv1.csv"
     4                  start_position => "beginning"
     5                  sincedb_path => "NULL"
     6      }
     7  }
     8  filter {
     9          csv {
    10                  columns => ["Month","Redshift","Relational Database Service","EC2-Instances","EC2-Other","Total"]
    11                  separator => ","
    12          }
    13  }
    14  output {
    15          elasticsearch {
    16                  hosts => ["https://vpc-testdomain-recbwzdjsopgulee6ul5skrke4.us-east-2.es.amazonaws.com:443"]
    17                  index => "billing-log"
    18                  document_type => "CSV"
    19                  user => "dgo"
    20                  ilm.enabled => false
    21                  ssl => true
    22                  password => "4utoknow"
    23  }
    24  }
8

What exact version of logstash?

9

Should be

ilm_enabled => false

That is your syntax error of course that will not still work for opensearch, But that's the syntax error.

So beside that, you're still going to need to use the correct plugins etc. Which you're going to need to work with on that opensearch forum or you can switch over to elasticsearch.

10

Stephen,

Im running Logstash 8.4.3. I've switched to True as per AWS documentation but now it's failing on the compatibly error.

I appreciate the assistance here. That kinda change things. Im no longer getting block in converge_state and instead is giving back again "Could not connect to a compatible version of Elasticsearch"

022-10-27 04:49:23.194 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:245:in `block in healthcheck!'"

Im gonna create a new domain and elect 7.10 instead of 1.3. Attempt to configure compatible versions.
Logstash 8.4.3 and Elasticsearch 7.10 are compatible?

11

Elasticsearch 7.10 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

12

Logstash 8.4.3 is not compatible with Elasticsearch 7.10, the last Logstash that is compatible with 7.10 is version 7.17, but there is a catch.

The elasticsearch output on any Logstash higher than 7.12, if I'm not wrong, will check if the Elasticsearch has a Elastic license, free or paid, they won't work with Elasticsearch 7.10 OSS or any fork like OpenDistro/Opensearch.

2 Likes
13

Elasticsearch 7.10 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.