Another Filter is not work after json filter

tatdat · March 27, 2018, 7:27am

I have some log with json type. Im using filter json for my log.
My config look like

filter {
  if [type] == "wifishop" {
    json {
      source => "message"
      target => "frt.wifi"
      remove_field => ["message"]
    }

    if [frt][wifi][IPWan] {

      geoip {
        id => "frt-wifi-geoip_IPWan-city"
        source => "[frt][wifi][IPWan]"
        default_database_type => "City"
        target => "[frt][wifi][geoip]"
        database => "/etc/logstash/GeoLite2-City.mmdb"
      }
      geoip {
        id => "frt-wifi-geoip_IPWan-asn"
        source => "[frt][wifi][IPWan]"
        default_database_type => "ASN"
        target => "[frt][wifi][geoip]"
        database => "/etc/logstash/GeoLite2-ASN.mmdb"
      }

      translate {
        regex => true
        dictionary_path => "/etc/logstash/translates/frt-ipshop-bo.yaml"
        field => "[frt][wifi][IPWan]"
        add_field => { "[frt][wifi][value]" => "translated" }
      }
      json {
        source => "translation"
        #remove_field => ["translation"]
      }

      mutate {
        add_field => { "[frt][wifi][checkip]" => "true" }
      }

    } else {
      mutate {
        add_field => { "[frt][wifi][checkip]" => "false" }
      }
    }
    
    mutate {
      convert => { "[frt][wifi][DownloadResult][AverageSpeed]" => "float" }
      convert => { "frt.wifi.DownloadResult.MaxSpeed" => "float" }
      convert => { "frt.wifi.DownloadResult.MinSpeed" => "float" }
      convert => { "frt.wifi.UploadResult.AverageSpeed" => "float" }
      convert => { "frt.wifi.UploadResult.MaxSpeed" => "float" }
      convert => { "frt.wifi.UploadResult.MinSpeed" => "float" }
      convert => { "frt.wifi.WifiResult.Frequency" => "float" }
      convert => { "frt.wifi.WifiResult.LinkSpeed" => "float" }
      add_field => { "frt.wifi.value" => "mutated" }
      #remove_tag => ["beats_input_codec_plain_applied","_grokparsefailure","_geoip_lookup_failure"]
    }
  }
}

And this is log in kibana after filtered

After first json filter

json {
    source => "message"
    target => "frt.wifi"
    remove_field => ["message"]
}

i got all field with prefix "frt.wifi.*" but when i check field [frt][wifi][IPWan], why it jump to else condition

if [frt][wifi][IPWan] {
    ........
} else {
      mutate {
           add_field => { "[frt][wifi][checkip]" => "false" }
       }
 }

What happen? something wrong in my config ?

magnusbaeck · March 27, 2018, 7:33am

Have you tried

target => "[frt][wifi]"

in your json filter?

tatdat · March 27, 2018, 7:59am

Thank for reply,

GeoIP filter worked, but translate filter not work

And i have question. What is diffirent between "frt.wifi" and "[frt].[wifi]" in json filter ?.

magnusbaeck · March 27, 2018, 8:10am

frt.wifi creates a field with that exact name:

{
  ...
  "frt.wifi": "some value",
  ...
}

[frt][wifi] creates a nested field:

{
  ...
  "frt": {
    "wifi": "some value"
  },
  ...
}

tatdat · March 28, 2018, 1:01am

Thank for your information!

Most filters work except translate filter

translate {
    regex => true
    dictionary_path => "/etc/logstash/translates/frt-ipshop-bo.yaml"
    field => "[frt][wifi][IPWan]"
    fallback => "UNKNOW"
}
json {
    source => "translation"
     #remove_field => ["translation"]
}

My translate file look like below

'118.69.201.43': '{"frt": {"wifi": {"shop_id": "0", "shop_name": " Shop1"}}}'
'118.69.183.170': '{"frt": {"wifi": {"shop_id": "1", "shop_name": "Shop2"}}}'
'118.69.60.85': '{"frt": {"wifi": {"shop_id": "2", "shop_name": "Shop3"}}}'
'118.69.108.232': '{"frt": {"wifi": {"shop_id": "3", "shop_name": "Shop4"}}}'

magnusbaeck · March 28, 2018, 6:16am

Please show

all of your pipeline configuration and
an example document produced by Logstash (copy/paste from Kibana's JSON tab).

tatdat · March 28, 2018, 7:00am

My pipeline configuration

input {
  ..........
}

filter {
  if [type] == "wifishop" {
    json {
      source => "message"
      target => "[frt][wifi]"
      remove_field => ["message"]
    }

    if [frt][wifi][IPWan] {
      geoip {
        id => "frt-wifi-geoip_IPWan-city"
        source => "[frt][wifi][IPWan]"
        default_database_type => "City"
        target => "[frt][wifi][geoip]"
        database => "/etc/logstash/GeoLite2-City.mmdb"
      }
      geoip {
        id => "frt-wifi-geoip_IPWan-asn"
        source => "[frt][wifi][IPWan]"
        default_database_type => "ASN"
        target => "[frt][wifi][geoip]"
        database => "/etc/logstash/GeoLite2-ASN.mmdb"
      }

      translate {
        regex => true
        dictionary_path => "/etc/logstash/translates/frt-ipshop-bo.yaml"
        field => "frt.wifi.IPWan"
        fallback => "UNKNOW"
      }
      json {
        source => "translation"
        #remove_field => ["translation"]
      }
    }

    mutate {
      convert => { "frt.wifi.DownloadResult.AverageSpeed" => "float" }
      convert => { "frt.wifi.DownloadResult.MaxSpeed" => "float" }
      convert => { "frt.wifi.DownloadResult.MinSpeed" => "float" }
      convert => { "frt.wifi.UploadResult.AverageSpeed" => "float" }
      convert => { "frt.wifi.UploadResult.MaxSpeed" => "float" }
      convert => { "frt.wifi.UploadResult.MinSpeed" => "float" }
      convert => { "frt.wifi.WifiResult.Frequency" => "float" }
      convert => { "frt.wifi.WifiResult.LinkSpeed" => "float" }
      #remove_tag => ["beats_input_codec_plain_applied","_grokparsefailure","_geoip_lookup_failure"]
    }
  }
}

output {
  if [type] == "wifishop" {
    elasticsearch {
      .........
    }
  }
}

And example document

{
  "_index": "fb-data-wifishop-2018.03",
  "_type": "doc",
  "_id": "LXzdamIBtNkW3GneMn8l",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "wifishop",
    "beattype": "doc",
    "offset": 4855,
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "host": "data-log",
    "@version": "1",
    "beat": {
      "version": "6.2.2",
      "hostname": "data-log",
      "name": "data-log"
    },
    "prospector": {
      "type": "log"
    },
    "frt": {
      "wifi": {
        "PingResult": {
          "AvgTime": "65.324 ms",
          "ResultString": "PING vnexpress.net (111.65.248.132) 56(84) bytes of data.\n",
          "LostPercent": "0 %",
          "Time": "28/03/2018 11:27:21",
          "Url": "vnexpress.net"
        },
        "DownloadResult": {
          "Time": "28/03/2018 11:28:13",
          "MaxSpeed": "2.595304",
          "AverageSpeed": "1.577272",
          "MinSpeed": "0.5592400000000001"
        },
        "StartDate": "28/03/2018 11:27:18",
        "IPAddress": "vnexpress.net/111.65.248.132",
        "TracerResult": {
          "ResultString": "0  Success  10.96.88.1  64.13985 ms\n1  Success  localhost  54.35453 ms\n2  Success  111.65.248.132  58.3 ms\n",
          "IPUrl": "vnexpress.net/111.65.248.132",
          "NumHop": "3",
          "Time": "28/03/2018 11:27:32",
          "Result": "Fail",
          "Url": "vnexpress.net"
        },
        "IPWan": "113.161.29.217",
        "WifiResult": {
          "score": "80",
          "Time": "28/03/2018 11:27:32",
          "SSID": "\"FPTSHOP-NV\"",
          "RSSI": -38,
          "Frequency": "2462",
          "LinkSpeed": "72"
        },
        "UploadResult": {
          "Time": "28/03/2018 11:27:45",
          "MaxSpeed": "41.94304",
          "AverageSpeed": "31.141784",
          "MinSpeed": "20.340528"
        },
        "geoip": {
          "continent_code": "AS",
          "location": {
            "lat": 14.35,
            "lon": 108
          },
          "latitude": 14.35,
          "region_code": "28",
          "ip": "113.161.29.217",
          "country_code2": "VN",
          "country_name": "Vietnam",
          "timezone": "Asia/Ho_Chi_Minh",
          "longitude": 108,
          "country_code3": "VN",
          "region_name": "Kon Tum",
          "asn": 45899,
          "as_org": "VNPT Corp",
          "city_name": "Kon Tum"
        },
        "ConnectIP": {
          "Time": "28/03/2018 11:27:32",
          "Result": "Connection to https://vnexpress.netsuccess \ntime connect: 547",
          "Link": "https://vnexpress.net"
        }
      }
    },
    "@timestamp": "2018-03-28T04:28:39.477Z",
    "source": "/home/fptlogs/uploads/866045038580732_log_28_03_2018_11_27_18.txt",
    "beatname": "fb-data-wifishop"
  },
  "fields": {
    "@timestamp": [
      "2018-03-28T04:28:39.477Z"
    ]
  },
  "sort": [
    1522211319477
  ]
}

magnusbaeck · March 28, 2018, 7:03am

  translate {
   regex => true
   dictionary_path => "/etc/logstash/translates/frt-ipshop-bo.yaml"
   field => "frt.wifi.IPWan"
   fallback => "UNKNOW"
 }

Always use the [frt][wifi][IPWan] notation to reference subfields. Same thing with your mutate filter near the end.

tatdat · March 28, 2018, 7:29am

When i change config, transslate filter worked but all another filter didn't work

I got example log

{
  "_index": "fb-data-wifishop-2018.03",
  "_type": "doc",
  "_id": "iwaAa2IBnIT8tAud5Ksx",
  "_version": 1,
  "_score": null,
  "_source": {
    "frt": {
      "wifi": {
        "shop_name": "HNI 20 Đông Các",
        "shop_id": "30227"
      }
    },
    "beattype": "doc",
    "type": "wifishop",
    "beatname": "fb-data-wifishop",
    "prospector": {
      "type": "log"
    },
    "beat": {
      "version": "6.2.2",
      "name": "data-log",
      "hostname": "data-log"
    },
    "@version": "1",
    "host": "data-log",
    "@timestamp": "2018-03-28T07:27:19.536Z",
    "offset": 23040,
    "tags": [],
    "source": "/home/fptlogs/uploads/864212034198313_log_28_03_2018.txt",
    "message": "{\"StartDate\":\"26\\/03\\/2018 10:15:25\",\"IPWan\":\"118.70.179.236\",\"IPAddress\":\"vnexpress.net\\/111.65.248.132\",\"PingResult\":{\"Time\":\"26\\/03\\/2018 10:15:28\",\"Url\":\"vnexpress.net\",\"IPUrl\":\"vnexpress.net\\/111.65.248.132\",\"ResultString\":\"PING vnexpress.net (111.65.248.132) 56(84) bytes of data.\\n64 bytes from 111.65.248.132: icmp_seq=1 ttl=56 time=13.3 ms\\n64 bytes from 111.65.248.132: icmp_seq=2 ttl=56 time=8.73 ms\\n64 bytes from 111.65.248.132: icmp_seq=3 ttl=56 time=4.42 ms\\n64 bytes from 111.65.248.132: icmp_seq=4 ttl=56 time=25.5 ms\\n\\n--- vnexpress.net ping statistics ---\\n4 packets transmitted, 4 received, 0% packet loss, time 3016ms\\nrtt min\\/avg\\/max\\/mdev = 4.422\\/12.996\\/25.523\\/7.884 ms\\n\",\"LostPercent\":\"0 %\",\"AvgTime\":\"12.996 ms\"},\"TracerResult\":{\"Time\":\"26\\/03\\/2018 10:15:30\",\"Url\":\"vnexpress.net\",\"IPUrl\":\"vnexpress.net\\/111.65.248.132\",\"ResultString\":\"0  Success  192.168.0.1  57.007153 ms\\n1  Success  118.69.185.149  154.47354 ms\\n2  Success  118.69.189.22  169.78107 ms\\n3  Success  118.69.132.135  237.80777 ms\\n4  Success  183.80.132.62  205.93584 ms\\n5  Success  118.69.248.150  268.607 ms\\n6  Success  10.250.4.6  42.02131 ms\\n7  Success  111.65.248.132  2.21 ms\\n\",\"NumHop\":\"8\",\"Result\":\"Success\"},\"WifiResult\":{\"Time\":\"26\\/03\\/2018 10:15:30\",\"SSID\":\"\\\"TEST-WIFI\\\"\",\"RSSI\":-53,\"LinkSpeed\":\"65\",\"Frequency\":\"2412\",\"score\":\"0\"},\"ConnectIP\":{\"Time\":\"26\\/03\\/2018 10:15:31\",\"Link\":\"https:\\/\\/vnexpress.net\",\"Result\":\"Connection to https:\\/\\/vnexpress.netsuccess \\ntime connect: 91\"},\"UploadResult\":{\"Time\":\"26\\/03\\/2018 10:15:45\",\"MinSpeed\":\"11.474144\",\"MaxSpeed\":\"27.38148\",\"AverageSpeed\":\"19.427812\"},\"DownloadResult\":{\"Time\":\"26\\/03\\/2018 10:16:19\",\"MinSpeed\":\"0.5994160000000001\",\"MaxSpeed\":\"3.77496\",\"AverageSpeed\":\"2.187188\"}}"
  },
  "fields": {
    "@timestamp": [
      "2018-03-28T07:27:19.536Z"
    ]
  },
  "sort": [
    1522222039536
  ]
}

magnusbaeck · March 28, 2018, 7:34am

So you're saying that changing

field => "frt.wifi.IPWan"

to

field => "[frt][wifi][IPWan]"

in your translate filter suddenly caused your initial json filter to not run? Sorry, I don't buy it.

tatdat · March 28, 2018, 7:57am

Yep, first json filter not work, but i dont understand that why first json didn'd work but translate filter can be work ...

system · April 25, 2018, 7:57am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.