ADD FIELD OVERWRITES DATA

Elastic Stack Logstash
1

Hi guys,

I'm trying to add data in a field like an array, but my grok or mutate overwrites it, could someone take a look?

This is my code:

`filter {
if [message] =~ "TargetID:" {
drop{}
}
grok {
patterns_dir => ["/usr/share/logstash/patterns"]
break_on_match => false
match => {

		"message" =>[
			"%{DATESTAMP:timing}",
			"%{FIRSTLINE:firstline}",
			"%{EVENTPD:eventpd}"
	 
		]
	}
	tag_on_failure => []
 }

    
 if [firstline]{
	grok {
	
		break_on_match => true
		match => ["firstline","%{WORD:test}\s%{WORD:test}\s%{WORD:test}\s%{WORD:test}\s%{WORD:test}\s\[%{USERNAME:sessionId}\]\s%{WORD:test}\s%{WORD:test}\s%{WORD:test}\s\[\[%{WORD:eventCodetmp}"]
			
		
	}
mutate{
	
		#remove_field => ["firstline"]
		remove_field => ["message"]
		remove_field => ["@version"]
		remove_field => ["timing"]
        remove_field => ["test"]
        add_field => ["eventCode" , "%{eventCodetmp}"]
        #remove_field => ["eventCodetmp"]
		
		
		
		rename => {"@timestamp" => "timestamp"}
        rename => {"timing" => "startTime"}
		}
	

 	}

date
{
locale => "es"
match => ["startTime","dd/MM/YY HH:mm:ss.SSS", "ISO8601"]
timezone => "Europe/Vienna"
target => "startTime"
}

 if [eventpd]{
    
	grok {
    break_on_match => true
	match => ["eventpd","SessionID\:\s%{USERNAME:sessionId}\]\s\#\#\#\sEvent\s\[%{WORD:eventCodetmp1}"]
 		
	

	}
		mutate{
		#remove_field => ["eventpd"]
		remove_field => ["message"]
		remove_field => ["@version"]
		remove_field => ["timing"]
		add_field => ["eventCode" , "%{eventCodetmp1}"]
        #remove_field => ["eventCodetmp"]
		rename => {"@timestamp" => "timestamp"}
        rename => {"timing" => "startTime"}
		}
	}
	
 }	`

The tmp variables are being fill with correct data, but always, when eventpd happends it overwrite the value of eventCode instead of add the value as array

2

Use mutate+merge rather than mutate+add_field

BTW, remove_field take an array, so you can do

		remove_field => ["message", "@version", "timing"]

Similarly for

rename => {
    "@timestamp" => "timestamp"
    "timing" => "startTime"
}

Generally I recommend against supplying an option to a filter more than once, since logstash will sometimes do very unexpected things when merging the contents of the two options.

3

I fotgot to say that I used update option to elastic, I use a personal id (sessionId) to update the documents, that is the reason because it is always updating the eventCode value, not adding info, overwriting it. I tried merge , and it is always updating with only one of the results, for example eventCode : %{eventCodetmp}, VALUE or VALUE, %{eventCodetmp1}

any idea for a workaroung? Maybe with ruby but i never used it :frowning:

4

I could manage with script in output config

if (ctx._source.eventCode == null) {ctx._source.eventCode= new ArrayList(); ctx._source.eventCode.add("%{[eventCode]}")} else {ctx._source.eventCode.add("%{[eventCode]}")}

but i sometimes a get this error in log

dynamic method [java.lang.String, add/1 not found

Any help?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.