Logstash input with Ruby filter to Opensearch output

Elastic Stack Logstash
1

I have rewritten the Ruby filter mentioned here:

The filter now reads as such and I am able to at minimum pass the event to OpenSearch:

input: |-
	http {
		port => 8080
		codec => "json"

filter: |-
	json {
		source => "message"
		target => "assets"
	}
	
	ruby {
		init => '
			require "net/http"
			require "uri"
			
			jira_url = ENV["JIRA_DEV_URL"] + ENV["ASSETS_ENDPOINT"]
			
			uri = URI(jira_url)
			token = ENV["JIRA_BEARER"]
			
			request = Net::HTTP::Get.new(uri)
			request["Authorization"] = "Bearer " + token
			
			response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https") do |http|
				http.request(request)
				return response
			end
			
			@@json_list = JSON.parse(response.body)
		'
		code => '
			begin
				event.set("[assets][environment]", "dev")
				
				event.get('[assets][object][data]").each do |key, value|
					should_remove = false
					if value.nil?
						should_remove = true
					elsif (value.is_a?(String) || value.is_a?(Array) || value.is_a?(Hash)) && value.empty?
						should_remove = true
					end
					if should_remove
						event.remove("[assets][object][data][#{key}]")
					else
						matching_item = @@json_list.find { |item| item["id"] == key }
						if matching_item
							new_key = matching_item["name"].downcase.gsub(" ", "_")
							event.remove("[assets][object][data][#{key}]")
							event.set("[assets][object][data][#{new_key}]", value)
						end
					end
				end
			rescue Exception => e
				logger.error("{e.message}")
			end
		'
	}

output: |-
	opensearch {
		hosts => ["url:port"]
		index => "index_string"
		ssl => true
		ssl_certificate_verification => true
		keystore => "keystore.p12"
		keystore_password => "${KEYSTORE_PASSWORD}"
		truststore => "truststore.jks"
		truststore_password => "${TRUSTSTORE_PASSWORD}"
		manage_template => false
	}
	
	stdout {}

However I am not getting the full JSON output with the following tags added and a bunch of data and/or comments missing.

The original filter managed to retrieve object data in a big json dump but has since stopped functioning hence the rewrite.

ruby {
	init => "
		require 'net/http'
		require 'json'
		require 'openssl'
		
		def fetch_data(uri, token)
			http = Net::HTTP.new.(uri.hostname, uri.port)
			http.use_ssl = true
			request = Net::HTTP::Get.new(uri)
			request['Authorization'] = 'Bearer ' + token
			response = http.request(request)
			return response
		end
	"
	code => "
		begin
			objectId = event.get('[assets][objectId]')
			objectId = objectId.to_s if objectId.is_a?(Integer)
			webhookTrigger = event.get('[assets][event]')
			
			token = ENV['JIRA_BEARER']
			
			# Object Data
			url = 'https://some.thing/jira/rest/assets/latest/object/'+objectId
			uri = URI(url)
			response = fetch_data(uri, token)
			if response.code.to_i == 200
				event.set('[assets][object][data]', response.body)
			end
			
			# Object Comments
			url = 'https://some.thing/jira/rest/assets/latest/comment/object/'+objectId
			uri = URI(url)
			response = fetch_data(uri, token)
			if response.code.to_i == 200
				event.set('[assets][object][comments]', response.body)
			end
		
		rescue Exception => e
			event.set('ruby_exception', e.message)
			event.set('ruby_exception_backtrace', e.backtrace.join('\n'))
		end
	"
}

What am I missing from the above filter, is it an issue with the json parsing or the ruby?

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.