Logstash elasticsearch filter geoip fields nil

[Len_Rugen]

I'm trying to get some of the geoip fields from a prior event using this code:

          elasticsearch {
              hosts => [ "myes:9200" ]
              user => "elastic"
              password => "changeme"
              index => "exchange-*"
              query => "type:iis AND cs-user:%{[cs-user]}"
              fields => { "@timestamp" => "prior.timestamp"
                          "[geoip][location]" => "prior.location"
                          "[geoip][country_name]" => "prior.country_name"
                          "clientip" => "prior.clientip"
              } 
          }

I've also tried the format "geoip.location" => "prior.location", but both ways all geoip fields return nill. The prior timestamp and clientip seem to be be working correctly.

Any ideas?

Thanks.

[warkolm (Mark Walkom)]

If you run the query directly does it return something?

[Len_Rugen]

Yes, looks like standard geoip section to me.

Frequently, the clientip of the current event will be the same as the prior.clientip. I've already ran the geoip filter on the current ip, so there should be good data in most of the prior events. I know some will have bad IP's, but this sample doesn't.

When I get this working, I'll skip this check if the ip's are the same

[Len_Rugen]

Even more interesting, I changed the fields to this:

              fields => { "@timestamp"            => "[prior][timestamp]"
                          "geoip"                 => "[prior][geoip]"
                          "[geoip][location]"     => "[prior][location]"
                          "[geoip][country_name]" => "[prior][country_name]"
                          "clientip"              => "[prior][clientip]"
              } 

I get the entire geoip structure in prior.geoip, I just can't get the individual fields. It looks like I'm missing the syntax, but I don't know what to try.

  "@timestamp" => 2017-02-09T00:00:00.000Z,
        "s-ip" => "xxx6.1",
       "prior" => {
           "geoip" => {
              "timezone" => "America/Chicago",
                    "ip" => "xxxx.134",
              "latitude" => 38.9517,
        "continent_code" => "NA",
             "city_name" => "Columbia",
         "country_code2" => "US",
          "country_name" => "United States",
              "dma_code" => 604,
         "country_code3" => "US",
           "region_name" => "Missouri",
              "location" => [
            [0] -92.3341,
            [1] 38.9517
        ],
           "postal_code" => "65211",
             "longitude" => -92.3341,
           "region_code" => "MO"
    },
        "clientip" => "xxxxx134",
    "country_name" => nil,
        "location" => nil,
       "timestamp" => 2017-02-16T14:22:37.000Z