Problem when using pipeline with filebeat (Not LogStash::Event.new)

[nb03briceno (Juan David)]

Hello everyone,
I'm really concerned about a problem that I'm getting in my work. We developed an algorithm that allows us to create a new event in logstash each time that a certain pattern is matched, yet it is not working when we load the files using Filebeat.

The next one is a fragment of the pipeline developed (working properly when we use file input i, but not working when we load with beats.

input{...}

filter{
grok
{
match => {"message" => "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:logLevel} %{GREEDYDATA:result1} %{SPACE} %{GREEDYDATA:result2}:%{GREEDYDATA:result3}%{SPACE}:%{SPACE}<%{GREEDYDATA:max_text1} %{NUMBER:max} %{GREEDYDATA:max_text2}> %{SPACE}<%{GREEDYDATA:nonHeap_text1} %{NUMBER:nonHeap} %{GREEDYDATA:nonHeap_text2}> %{SPACE}<%{GREEDYDATA:total_text1} %{NUMBER:total} %{GREEDYDATA:total_text2}> %{SPACE}<%{GREEDYDATA:free_text1} %{NUMBER:free} %{GREEDYDATA:free_text2}> %{SPACE}<%{GREEDYDATA:consumed_text1} %{NUMBER:consumed} %{GREEDYDATA:consumed_text2}>"}
}

	if ("_grokparsefailure" in [tags]) {  

		mutate {
			add_tag => ["error2"]
			remove_tag => ["_grokparsefailure"]  
		}

    }
	else{

	mutate {
			add_tag => ["memory_value"]  
		}

	ruby {
          init => "@free= '',@free_name='',@consumed='',@consumed_name='', @date='',@total='',@total_name='',@nonHeap='',@nonHeap_name='',@max='',@max_name='' "
		  
		  code => "
                require 'pry'
				@free= event.get('free')
				@free_name= event.get('free_text1')
				@consumed= event.get('consumed')
				@consumed_name= event.get('consumed_text1')
				@date= event.get('logdate')
				@total= event.get('total')
				@total_name= event.get('total_text1')
				@nonHeap= event.get('nonHeap')
				@nonHeap_name= event.get('nonHeap_text1')
				@max= event.get('max')
				@max_name= event.get('max_text1')
				
				generated = LogStash::Event.new
                generated.set('values',@free)
				generated.set('values_names',@free_name)
				generated.set('date_time',@date)
                new_event_block.call(generated)
				.......

output{......}

The problem is the only event stored within the index is the grok matching, yet the generated event "generated = LogStash::Event.new" seems not to be working because is not been added to the index. And when running logstash and filebeat all apears to be good.

There are limitations for creating events in logstash when using filebeat?? or using the ruby code ? The followings are the methods we need to use from ruby:

-generated.set()
-event.get()
-LogStash::Event.new

Thanks so much,

This project is really important for the company I'm working with.