Mutate Gsub not working


#1

Hello,

I'm currently extracting data from Elasticsearch to a CSV using logstash, however I have some fields that returns a null, so I would have in my CSV, something like: 1,,2,3,4,,,,2.

My configuration is the following:

input {
	elasticsearch {
		hosts => "xxxx"
		index => "xxxx"
		scroll => "1m"
		query => '{"query": {"match_all":{}},"sort": ["item"]}'
	}
}

filter {
  mutate {
    gsub => [
     "color", "39225", "qwerty"
    ]
  }

}
output {
	csv {
		path => "outputNewES.csv"
		fields => ["item","color"]
	}
}

Do you know what I'm doing wrong?


#2

Impossible to say given the limited information you have posted. If you can show a document from output { stdout { codec => rubydebug } } we might hazard a guess.


#3

Hello, thanks for your response.
By putting stdout { codec => rubydebug } I just get the same I get on Postman.
After further testing and investigation I can now replace Strings, however I can not replace null or long values.

For example, in Postman I have this field with the following value:

"vendorId": 450,
"eanProvider": "1111111111111",
"eanInternal": null,

How do i replace the int and the null?
I've tried this without sucess

"vendorId","450","111",
"eanInternal","null","eanInternal"

#4

If you do not need a regular expression then for integers you can test for a particular value and then mutate+replace. For nil values you can mutate+coerce.


#5

Can you please provide me an example?

I've tried:

"vendorNum", "111", "222"
"vendorNum", 111, 222

The first doesn't work and the second says I have a syntax error due to the lack of "".
The same happens when I'm identifying null values.


#6

What I meant is

if [vendorId] == 450 { mutate { replace => { "vendorId" => "111" } }

mutate { coerce { "eanInternal" => "eanInternal" } }

#7

Hmm I see, and where should I put that? Inside a ruby tag?


#8

I've managed to use coerce to replace a string null, but not long null.

Which means:

coerce => {"eanInternal" => 1234569}---> this doesn't
coerce => {"validityEndDate"=>"date"}---> this works

#9

Turns out it was a problem with the if clause. Without the if I don't have any problem with the coerce. Where should I put it?


#10

How about this?

mutate { coerce { "eanInternal" => "1234569" } }
mutate { convert { "eanInternal" => "integer" } }

#11

Turns out the problem was the if clause, it's misplaced. I've putted inside the filter, but outside the mutate. Where should I put it?


#12

The coerce does not need to be conditional, since it only takes effect if the field is nil.

For the mutate+replace my original post shows how the conditional should be written.


#13

My configuration is the following:

input {
	elasticsearch {
		hosts => "xxx"
		index => "xxx"
		scroll => "1m"
		query => '{"query": {"match_all":{}},"sort": ["xxx"]}'
	}
}

filter {
	
	if [vendorId] == 450 { mutate { replace => { "vendorId" => "111" } }
	
  mutate {
    gsub => [
     "xxx", "xxx", "xxx"
	 
    ]
	coerce => {"xxx" => 666}
	
  }

}
output {
	csv {
		path => "outputNewES.csv"
		fields => ["xxx""]
	}
}

I have an error with this configuration due to the if clause


#14

Yes, because you should have three } at the end of the line, not just two.


#15

yap, you're completely correct. Thank you so much for the help


(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.