Logstash elasticsearch filter error TypeError: can't convert nil into String


(Len Rugen) #1

I'm new to elk, trying to query es from logstash with this:

        elasticsearch {
              hosts => [ "my-elk-server:9200" ]
              user => "elastic"
              password => "changeme"
              index => "exchange-*"
              query => "type:iis AND cs-user:%{[cs-user]}"
              fields => [ { "@timestamp" => "prior-timestamp" },  
                          { "geoip.location" => "prior-location" } ]
          }

Some fields have been sanitized :-), If the user or password is wrong, I get a security failure, when it's right, I get this:

[2017-02-14T19:29:48,818][INFO ][logstash.filters.elasticsearch] Querying elasticsearch for lookup {:params=>{:index=>"exchange-", :q=>"type:iis AND cs-user:somebody", :size=>1, :sort=>"@timestamp:desc"}}
[2017-02-14T19:29:49,129][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"exchange-
", :query=>"type:iis AND cs-user:somebody", :event=>2017-02-09T00:00:00.000Z localhost.localdomain 2017-02-09 00:00:00 xxx.yyy.zz.1 POST /autodiscover/autodiscover.xml &CorrelationID=;&cafeReqId=e1eb2af0-b060-432c-a45e-39f3752210af; 80 somebody zz2.yy2yy2.227 Mac+OS+X/10.10.4+(14E46);+ExchangeWebServices/5.0+(213);+Mail/8.2+(2102) - 401 1 1326 1
, :error=>#<TypeError: can't convert nil into String>}

With/without the index doesn't matter, only one field doesn't matter, I'm stumped, I don't know where to look....

Thanks


Extract document id into another variable in logstash
(Magnus B├Ąck) #2

I suspect you're being hit with the bug described below, even though the error message appears to be slightly different. Have you tried omitting the fields option?


(Len Rugen) #3

Yes it worked without fields. so I tried fields => { "@timestamp" => "prior-timestamp" }, and that worked. It looks like it's a syntax error, I need to remove the [] from the fields option and remove the comma seperating them. It's working better, but getting nil for my prior.location, which I'll have to research.

Without fields, you can't reference anything from the matching event right?

Thanks


(Len Rugen) #4

I'm pretty sure I don't understand how the es filter works yet :slight_smile:

The fields option doesn't refer to fields in the query result does it? You have to reparse the event? But you only have the parts of the event stored, for example, we drop "message".

Here's what I've been trying to do, but examples are hard to find....

For the new event, find the users prior event, get the timestamp and geoip.location from the prior event for further processing and it will be stored with the new event, for now anyway.

THanks.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.