If statement not work

Elastic Stack Logstash
1

Hi all,
I'm trying to import a sample json file:

...
{
  "id": 2,
  "timestamp": "2019-08-11T17:55:56Z",
  "paymentType": "Visa",
  "name": "Darby Dacks",
  "gender": "Female",
  "ip_address": "77.72.239.47",
  "purpose": "Shoes",
  "country": "Poland",
  "age": 55
}
{
  "id": 3,
  "timestamp": "2019-07-14T04:48:25Z",
  "paymentType": "Visa",
  "name": "Harri Cayette",
  "gender": "Female",
  "ip_address": "227.6.210.146",
  "purpose": "Sports",
  "country": "Canada",
  "age": 27
}
...

This is my logtash test conf:

input {
  file {
    start_position => "beginning"
    type => "json"
    path => "/tmp/json.txt"
    sincedb_path => "/dev/null"
  }
}
filter{
    if [country] == "Poland" {
        mutate {
                add_field => { "test" => "test" }
        }
    }

    json {
        source => "message"
    }

}
output {
  file {
    path => "/tmp/out.log"
  }
}

I would like to add a field depending on the content of an existing field (country) but the above configuration doesn't work: the field "test" is not added.
What am I doing wrong?

Thank you all

2

I think the code is indendent a level too much. Try this one:

filter {
  if [country] == "Poland" {
      mutate {
              add_field => { "test" => "test" }
      }
  }
 json {
      source => "message"
  }

}
3

Your conditional is correct and should've work, can you share the output you are getting from logstash in the /tmp/out.log file?

Does your source message a single line json or is pretty printed in your source file?

Identation makes no difference in Logstash.

4

This is the full original input json.txt (i used jq for pretty output):

{"id":1,"timestamp":"2019-09-12T13:43:42Z","paymentType":"Amex","name":"Merrill Duffield","gender":"Female","ip_address":"132.150.218.21","purpose":"Toys","country":"United Arab Emirates","age":33}
{"id":2,"timestamp":"2019-08-11T17:55:56Z","paymentType":"Visa","name":"Darby Dacks","gender":"Female","ip_address":"77.72.239.47","purpose":"Shoes","country":"Poland","age":55}
{"id":3,"timestamp":"2019-07-14T04:48:25Z","paymentType":"Visa","name":"Harri Cayette","gender":"Female","ip_address":"227.6.210.146","purpose":"Sports","country":"Canada","age":27}
{"id":4,"timestamp":"2020-02-29T12:41:59Z","paymentType":"Mastercard","name":"Regan Stockman","gender":"Male","ip_address":"139.224.15.154","purpose":"Home","country":"Indonesia","age":34}
{"id":5,"timestamp":"2019-08-03T19:37:51Z","paymentType":"Mastercard","name":"Wilhelmina Polle","gender":"Female","ip_address":"252.254.68.68","purpose":"Health","country":"Ukraine","age":51}

This is the full output (out.log):

{"log":{"file":{"path":"/tmp/json.txt"}},"@version":"1","timestamp":"2019-08-11T17:55:56Z","id":2,"gender":"Female","@timestamp":"2022-07-08T14:15:53.878804Z","event":{"original":"{\"id\":2,\"timestamp\":\"2019-08-11T17:55:56Z\",\"paymentType\":\"Visa\",\"name\":\"Darby Dacks\",\"gender\":\"Female\",\"ip_address\":\"77.72.239.47\",\"purpose\":\"Shoes\",\"country\":\"Poland\",\"age\":55}"},"paymentType":"Visa","host":{"name":"NBK-DAVIDD"},"type":"json","ip_address":"77.72.239.47","age":55,"purpose":"Shoes","message":"{\"id\":2,\"timestamp\":\"2019-08-11T17:55:56Z\",\"paymentType\":\"Visa\",\"name\":\"Darby Dacks\",\"gender\":\"Female\",\"ip_address\":\"77.72.239.47\",\"purpose\":\"Shoes\",\"country\":\"Poland\",\"age\":55}","name":"Darby Dacks","country":"Poland"}
{"log":{"file":{"path":"/tmp/json.txt"}},"@version":"1","timestamp":"2020-02-29T12:41:59Z","id":4,"gender":"Male","@timestamp":"2022-07-08T14:15:53.880454Z","event":{"original":"{\"id\":4,\"timestamp\":\"2020-02-29T12:41:59Z\",\"paymentType\":\"Mastercard\",\"name\":\"Regan Stockman\",\"gender\":\"Male\",\"ip_address\":\"139.224.15.154\",\"purpose\":\"Home\",\"country\":\"Indonesia\",\"age\":34}"},"paymentType":"Mastercard","host":{"name":"NBK-DAVIDD"},"type":"json","ip_address":"139.224.15.154","age":34,"purpose":"Home","message":"{\"id\":4,\"timestamp\":\"2020-02-29T12:41:59Z\",\"paymentType\":\"Mastercard\",\"name\":\"Regan Stockman\",\"gender\":\"Male\",\"ip_address\":\"139.224.15.154\",\"purpose\":\"Home\",\"country\":\"Indonesia\",\"age\":34}","name":"Regan Stockman","country":"Indonesia"}
{"log":{"file":{"path":"/tmp/json.txt"}},"@version":"1","timestamp":"2019-09-12T13:43:42Z","id":1,"gender":"Female","@timestamp":"2022-07-08T14:15:53.852279Z","event":{"original":"{\"id\":1,\"timestamp\":\"2019-09-12T13:43:42Z\",\"paymentType\":\"Amex\",\"name\":\"Merrill Duffield\",\"gender\":\"Female\",\"ip_address\":\"132.150.218.21\",\"purpose\":\"Toys\",\"country\":\"United Arab Emirates\",\"age\":33}"},"paymentType":"Amex","host":{"name":"NBK-DAVIDD"},"type":"json","ip_address":"132.150.218.21","age":33,"purpose":"Toys","message":"{\"id\":1,\"timestamp\":\"2019-09-12T13:43:42Z\",\"paymentType\":\"Amex\",\"name\":\"Merrill Duffield\",\"gender\":\"Female\",\"ip_address\":\"132.150.218.21\",\"purpose\":\"Toys\",\"country\":\"United Arab Emirates\",\"age\":33}","name":"Merrill Duffield","country":"United Arab Emirates"}
{"log":{"file":{"path":"/tmp/json.txt"}},"@version":"1","timestamp":"2019-08-03T19:37:51Z","id":5,"gender":"Female","@timestamp":"2022-07-08T14:15:53.880880Z","event":{"original":"{\"id\":5,\"timestamp\":\"2019-08-03T19:37:51Z\",\"paymentType\":\"Mastercard\",\"name\":\"Wilhelmina Polle\",\"gender\":\"Female\",\"ip_address\":\"252.254.68.68\",\"purpose\":\"Health\",\"country\":\"Ukraine\",\"age\":51}"},"paymentType":"Mastercard","host":{"name":"NBK-DAVIDD"},"type":"json","ip_address":"252.254.68.68","age":51,"purpose":"Health","message":"{\"id\":5,\"timestamp\":\"2019-08-03T19:37:51Z\",\"paymentType\":\"Mastercard\",\"name\":\"Wilhelmina Polle\",\"gender\":\"Female\",\"ip_address\":\"252.254.68.68\",\"purpose\":\"Health\",\"country\":\"Ukraine\",\"age\":51}","name":"Wilhelmina Polle","country":"Ukraine"}
{"log":{"file":{"path":"/tmp/json.txt"}},"@version":"1","timestamp":"2019-07-14T04:48:25Z","id":3,"gender":"Female","@timestamp":"2022-07-08T14:15:53.879848Z","event":{"original":"{\"id\":3,\"timestamp\":\"2019-07-14T04:48:25Z\",\"paymentType\":\"Visa\",\"name\":\"Harri Cayette\",\"gender\":\"Female\",\"ip_address\":\"227.6.210.146\",\"purpose\":\"Sports\",\"country\":\"Canada\",\"age\":27}"},"paymentType":"Visa","host":{"name":"NBK-DAVIDD"},"type":"json","ip_address":"227.6.210.146","age":27,"purpose":"Sports","message":"{\"id\":3,\"timestamp\":\"2019-07-14T04:48:25Z\",\"paymentType\":\"Visa\",\"name\":\"Harri Cayette\",\"gender\":\"Female\",\"ip_address\":\"227.6.210.146\",\"purpose\":\"Sports\",\"country\":\"Canada\",\"age\":27}","name":"Harri Cayette","country":"Canada"}

How you can see there is no "test" field in the output file.

5

Oh, I thought you had codec => "json" in your input, but it is type => "json".

Your if is not working because the field country is not present in the document when your event arrives in that conditional, you are parsing the message just after the conditional, it is only after you parse the message field that the field country will be present in your document and you would be able to use it for conditionals.

Change the order and it will work:

json {
        source => "message"
}
if [country] == "Poland" {
        mutate {
                add_field => { "test" => "test" }
        }
}
6

Thank You Leandro. It's work but i preferred to use "codec" instead of "type" as you suggested so i changed my conf in this way:

input {
  file {
    start_position => "beginning"
    codec => json
    path => "/tmp/json.txt"
    sincedb_path => "/dev/null"
  }
}
filter{
    if [country] == "Poland" {
        mutate {
                add_field => { "test" => "test" }
        }
    }
}

Is this the best practice?

Thank you again

7

Those are different things.

Using type => "json" in the input will add a field named type to your event, you can then use this field to filtering

Using codec => "json" in the input will tell logstash that your message is a json document and it will parse directly in the input stage.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.