Elasticsearch ip type in ranges?

Elastic Stack Elasticsearch
1

using elasticsearch 6.4 on a Centos7 with latest patches

I currently see a very weird behaviour on my ELK. It's related to a search query on a field which is of type ip.
The strange this is that I get results based on the size of the range. I'm looking for the ip xx.yy.104.65 of the field clientip.
So a query like this does work

curl -g 'http://127.0.0.1:9200/maillog-2018-11-08/_search?q=clientip=[xx.yy.104.0+TO+xx.yy.104.99]'
{"took":124,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":3

whereas the following does NOT return results

curl -g 'http://127.0.0.1:9200/maillog-2018-11-08/_search?q=clientip=[xx.yy.104.0+TO+xx.yy.104.100]'
{"took":134,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":}}

Interestingly thise on (much bigger range) does return the expected (correct) results

curl -g 'http://127.0.0.1:9200/maillog-2018-11-08/_search?q=clientip=[xx.yy.104.0+TO+xx.yy.254.254]'
{"took":152,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":7

by try-and-error I found that the query between xx.yy.104.0 and xx.yy.104.99 does work as well. But no query with last octet greater as 99 does return a result.
Again interestingly if I change the second last octet I get hits again, so this works as well xx.yy.104.0 and xx.yy.105.1 That would explain why I get matches with bigger ranges.
But I have no clue why my queries stop producing hits if I only change last octet to bigger than 99

Thanks for any input

tobi

2

This looks like a problem caused by a wrong field type to me.
Check the index mapping and make sure the type of your clientip field is ip.

3

that what we thought too, as we changed the field type to ip recently. But even if I query it on indices which where created after the change (we use dated indices) I cannot find results as soon as the search goes above .100 in the last octet. I checked the logstash logs and it shows that the ES field clientip is set as type ip. All our logstash templates which set this field set it as type ip. Double checked :slight_smile:
Is there another way to check the index mapping? Except check templates and ensure that logstash shows the correct type in logs when it applies the template?

4

You can see configuration of maillog-2018-11-08 index using

curl 'localhost:9200/maillog-2018-11-08?pretty'
5

Hi,
thanks a lot for the command :slight_smile: It shows me that the field is indeed of type "text"
Now the question arises: why the hack is the template not applied?
My template config in output.conf looks as follows

} else if [type] == "rejects" {
    elasticsearch {
      index => "rejects-%{+YYYY-MM-dd}"
      hosts => "localhost:9200"
      manage_template => true
      template_overwrite => true
      template => "/etc/logstash/config/templates/rejects.json"
      template_name => "rejects"
    }
  }

and in rejects.json I defines clientip as follows

"clientip":{
            "type":"ip"
          },

when starting up logstash it tells me that the correct type is applied:

Using mapping template from {:path=>"/etc/logstash/config/templates/rejects.json"}

Attempting to install template {:manage_template=>{"template"=>"rejects", "order"=>1, "settings"=>{"number_of_shards"=>2, "index.refresh_interval"=>"90s", "analysis"=>{"normalizer"=>{"lowercase"=>{"type"=>"custom", "char_filter"=>, "filter"=>["lowercase", "asciifolding"]}}}}, "rejects"=>{"mappings"=>{"rejects"=>{"properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"text", "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}, "clienthost"=>{"type"=>"text", "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}, "clientip"=>{"type"=>"ip"}, ....

Installing elasticsearch template to _template/rejects

so from log output above all looks fine to me
I have not idea where the template field type got overwritten back to default "text"

6

I think logstash will push an index template to ES only if it does not exist already.

You would be better off defining index template directly in ES. Also note that changes to index template won't affect already existing indexes.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.