Mutate Gsub not working

Elastic Stack Logstash
1

Hello,

I'm currently extracting data from Elasticsearch to a CSV using logstash, however I have some fields that returns a null, so I would have in my CSV, something like: 1,,2,3,4,,,,2.

My configuration is the following:

input {
	elasticsearch {
		hosts => "xxxx"
		index => "xxxx"
		scroll => "1m"
		query => '{"query": {"match_all":{}},"sort": ["item"]}'
	}
}

filter {
  mutate {
    gsub => [
     "color", "39225", "qwerty"
    ]
  }

}
output {
	csv {
		path => "outputNewES.csv"
		fields => ["item","color"]
	}
}

Do you know what I'm doing wrong?

2

Impossible to say given the limited information you have posted. If you can show a document from output { stdout { codec => rubydebug } } we might hazard a guess.

3

Hello, thanks for your response.
By putting stdout { codec => rubydebug } I just get the same I get on Postman.
After further testing and investigation I can now replace Strings, however I can not replace null or long values.

For example, in Postman I have this field with the following value:

"vendorId": 450,
"eanProvider": "1111111111111",
"eanInternal": null,

How do i replace the int and the null?
I've tried this without sucess

"vendorId","450","111",
"eanInternal","null","eanInternal"
4

If you do not need a regular expression then for integers you can test for a particular value and then mutate+replace. For nil values you can mutate+coerce.

5

Can you please provide me an example?

I've tried:

"vendorNum", "111", "222"
"vendorNum", 111, 222

The first doesn't work and the second says I have a syntax error due to the lack of "".
The same happens when I'm identifying null values.

6

What I meant is

if [vendorId] == 450 { mutate { replace => { "vendorId" => "111" } }

mutate { coerce { "eanInternal" => "eanInternal" } }
7

Hmm I see, and where should I put that? Inside a ruby tag?

8

I've managed to use coerce to replace a string null, but not long null.

Which means:

coerce => {"eanInternal" => 1234569}---> this doesn't
coerce => {"validityEndDate"=>"date"}---> this works
9

Turns out it was a problem with the if clause. Without the if I don't have any problem with the coerce. Where should I put it?

10

How about this?

mutate { coerce { "eanInternal" => "1234569" } }
mutate { convert { "eanInternal" => "integer" } }
11

Turns out the problem was the if clause, it's misplaced. I've putted inside the filter, but outside the mutate. Where should I put it?

12

The coerce does not need to be conditional, since it only takes effect if the field is nil.

For the mutate+replace my original post shows how the conditional should be written.

13

My configuration is the following:

input {
	elasticsearch {
		hosts => "xxx"
		index => "xxx"
		scroll => "1m"
		query => '{"query": {"match_all":{}},"sort": ["xxx"]}'
	}
}

filter {
	
	if [vendorId] == 450 { mutate { replace => { "vendorId" => "111" } }
	
  mutate {
    gsub => [
     "xxx", "xxx", "xxx"
	 
    ]
	coerce => {"xxx" => 666}
	
  }

}
output {
	csv {
		path => "outputNewES.csv"
		fields => ["xxx""]
	}
}

I have an error with this configuration due to the if clause

14

Yes, because you should have three } at the end of the line, not just two.

15

yap, you're completely correct. Thank you so much for the help

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.