Logstash Pipeline Error: JSON ParseError

I am collecting asset data for Jira using automations to generate a HTTP POST request to Logstash, the following code then takes the input from the request and filters it and sends it to OpenSearch. I have one yaml file that is working using webhooks to collect issue data however the asset yaml doesn't. The error thrown refers to the logstash.javapipeline. The problem however seems to be with the Ruby filter as hashing out those lines seems to work however doesn't provide any data.

[ERROR][logstash.javapipeline][main] Pipeline error {:pipeline_id="main", :exception=>#JSON::ParserError: unexpected token at ''>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:238:in `register_plugins`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in `run`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start`"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x36246de8 run>"}


input: |-
	http {
		port => 8080
		codec => "json"
filter: |-
	json {
		source => "message"
		target => "assets"
	ruby {
		init => '
			require "net/http"
			require "uri"
			jira_url = ENV["JIRA_DEV_URL"]
			uri = URI(jira_url)
			token = ENV["JIRA_BEARER"]
			request = Net::HTTP::Get.new(uri)
			request["Authorization"] = "Bearer " + token
			response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https") do |http|
			@@json_list = JSON.parse(response.body)
		code => '
				event.set("[assets][environment]", "dev")
				event.get("[assets][object][data]").each do |key, value|
					should_remove = false
					if value.nil?
						should_remove = true
					elsif (value.is_a?(String) || value.is_a?(Array) || value.is_a?(Hash)) && value.empty?
						should_remove = true
					if should_remove
						matching_item = @@json_list.find { |item| item["id"] == key }
						if matching_item
							new_key = matching_item["name"].downcase.gsub(" ", "_")
							event.set("[assets][object][data][#{new_key}]", value)
			rescue Exception => e

output: |-
	opensearch {
		hosts => ["url:port"]
		index => "index_string"
		ssl => true
		ssl_certificate_verification => true
		keystore => "keystore.p12"
		keystore_password => "${KEYSTORE_PASSWORD}"
		truststore => "truststore.jks"
		truststore_password => "${TRUSTSTORE_PASSWORD}"
		manage_template => false
	stdout {}

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

Thanks, the issue doesn't appear to be with the OpenSearch output so I'm not too concerned about that at this point in time.

Where is this file format from? It looks like an template for some automation tool.

Also, is it a typo or your input is missing a closing curly bracket?

Please share the other log lines you have, you will probably have more lines around the one you shared.

@leandrojmp Apologies yes that is a typo, some of the curly brackets seems to have dropped off when I copied the lines.

As for the format, the format is a .yaml file as the Jira, Logstash and OpenSearch instances are all deployed on Kubernetes. I can't provide the entire file however the attributes come from the helm chart similar to this one:

I'll grab some of the other log lines surrounding the one I shared if this will be of use.

This is wrong, it may be your issue, it is mixing a single quote with a double quote.

Apologies @leandrojmp that's another typo.
The way this was originally written was with single quotes which I had to go through and change to double quotes. I seem to have missed this one.

Hi @leandrojmp, please see below for the full list of logging. This pod is in a crashloop now where it seems to get stuck on the pipeline somewehere.

Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[INFO][logstash.runner] Log4J configuration path used is: /usr/share/logstash/config/log4j2.properties
[INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.4.0", "jruby.version"=>"jruby (2.6.8) 2022-06-27 7a2cbcd376 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[INFO][logstash.runner] JVM bootstrap flags: [-Xmslg, Xmxlg, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.edg=file:/dev/urandom, - Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Djruby.regexp.interruptable=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[INFO][logstash.agent] Successfully started Logstash API Endpoint {:port=>9600, :ssl_enabled=>false}
[INFO][org.reflections.Reflections] Reflections took 182 ms to scan 1 urls, producing 125 keys amd 434 values
[INFO][logstash.codecs.json] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[INFO][logstash.javapipeline] Pipeline `main` is configured with `pipeline_ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"Logstash::Outputs::OpenSearch", :hosts=>["<url>:443"]}
[INFO][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[<url>:443/]}}
[WARN][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"<url>:443"}
[INFO][logstash.outputs.opensearch][main] Cluster version determined (2.4.1) {:version=>2}
[ERROR][logstash.javapipeline][main] Pipeline error {:pipeline_id="main", :exception=>#JSON::ParserError: unexpected token at ''>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:238:in `register_plugins`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in `run`", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start`"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x36246de8 run>"}
[INFO][logstash.javapipeline][main] Pipeline terminated {"pipeline.id"=>"main"}
[ERROR][logstash.agent] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[INFO][logstash.runner] Logstash shut down.
[FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

I've replaced sensitive information like so just overlook that. All of these URL's are also https://

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.