[SOLVED] Mutate string to integer when field type is already integer

[nnet]

Hi, using ELK 5.6.16 on Ubuntu 16.04.

I'm trying to figure out a way to ingest json where a field learn is of type integer. Once in a while an event will come in and that field's value is not an integer but instead a string, yes or no.

I want to change/convert those values to integers 0 or 1 as needed.

Since the field type is integer I evidently cannot use mutate gsub because the field type isn't string.

Various attempts to use mutate convert boolean/integer have no effect on the event data, and the data is ingested unchanged.

Here's a sample of an event:

{"env": "rtl433", "time": "1606424172", "protocol": 36,"model": "Test-Device","id": 1, "battery_ok": "current": 5.099,"interval": 6,"learn": "no","mic": "CHECKSUM","mod": "FSK","freq1": 433.957,"freq2": 433.886,"rssi": -8.079,"snr": 16.212,"noise": -24.291}

I tried:

filter{
  json {
    source => "message"
    id => "rtl433"
    add_tag => [ "JSON", "rtl433" ]
  }
  mutate{
    convert => ["learn","boolean"]
  }
  mutate {
    convert => ["learn","integer]
  }
}

with the expectation that the values of yes and no would be converted per the documentation at https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-mutate.html#plugins-filters-mutate-convert but this has no effect on the event data, and I don't know if it's because the field type is integer according to elasticsearch.

Can anyone shed some insight how I can achieve my goal, thanks!

[Badger]

If I make the JSON valid ("battery_ok" needs a value) then I end up with [learn] being 0 or 1 (integer) using that filter in version 7.10.

Since the field type is integer I evidently cannot use mutate gsub because the field type isn't string .

Not so. In logstash the type is string, because the JSON type is string. The type in elasticsearch may be integer, but that does not make it an integer in logstash.

[nnet]

@Badger There's a typo on my part in the json data above, "battery_ok" has integer value of 0 in my tests. The above was the result of a bad copy/paste. In any case I'm not using ELK 7.x, I'm using 5.6.16.

if mutate gsub doesn't rely on ES field type, I dont understand why this didn't work:

mutate { 
  gsub => [
    "learn","no",0]
}

nor do I understand why the converts dont work in 5.6.16.

[Badger]

@nnet

I dont understand why this didn't work:

mutate { 
  gsub => [
    "learn","no",0]
}

That depends what you mean by "didn't work". If I remember correctly (I cannot test it right now) then the logstash configuration compiler will turn that 0 into "0" (i.e. a string not an integer).

I have used 5.x in the past and I would have expected the sequence of mutate+convert filters to work in 5.x just as well as it works in 7.x.

[nnet]

I tried a simple gsub and its not doing what I expected either:

filter {
  json {
    source => "message"
    id => "rtl433"
    add_tag => [ "JSON", "rtl433" ]
  }
  mutate {
    gsub => ["learn","no","false"]
  }
  mutate {
    gsub => ["learn","yes","true"]
  }
}

Sending

{"env" : "rtl433", "time" : "1606448258", "protocol" : 36, "model" : "Test-Device", "id" : 1, "battery_ok" : 0, "current" : 5.099, "interval" : 6, "learn" : "yes", "mic" : "CHECKSUM", "mod" : "FSK", "freq1" : 433.957, "freq2" : 433.886, "rssi" : -8.079, "snr" : 16.212, "noise" : -24.291}

doesnt change learn value yes to true. I must be doing something wrong. I'm at a loss at this point.

[nnet]

I found my issue and it was total PEBKAC on my part. I no longer ingest that particular data via tcp listener, I use filebeat instead. Since the filter relies on the document_type set in filebeat.yml, it was never processing my gsubs or converts. I added the two converts back into the filter, and removed the gsubs, and the filter now does what I expected.

Thanks you for your time and effort @Badger, once again you've been helpful, its greatly appreciated!

The fix:

filter {
  json {
    source => "message"
    id => "rtl433"
    add_tag => [ "JSON", "rtl433" ]
  }
  mutate {
    convert => ["learn","boolean"]
  }
  mutate {
    convert => ["learn","integer"]
  }
}