Bulk import of IP addresses and showing a map with geo-location : No Compatible Fields


(Hans) #1

Dear All,

My environment ELK 6.6.1
I try to import data and want to show a map with the locations.

I create the index with

PUT /spamcalcgrp
{
  "settings": {
    "index.mapping.total_fields.limit": 1500,
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "doc": {
      "properties": {
        "totalscore": {
          "type": "float"
        },
        "nrules": {
          "type": "integer"
        },
        "xspam": {
          "type": "integer"
        },
        "sspam": {
          "type": "integer"
        },
        "fortiip": {
          "type": "ip"
        },
        "xlarge": {
          "type": "integer"
        }
      }
    }
  },
  "geoip.location": {
    "type": "geo_point"
  }
}

The IP address is stored in "fortiip"
Than I create a pipeline with

PUT /_ingest/pipeline/geoip-info
{
  "description": "Add geoip info",
  "processors": [
    {
      "geoip": {
        "field": "fortiip",
        "target_field": "geoip_tmp",
        "properties": [
          "location"
        ]
      }
    },
    {
      "set": {
        "field": "geoip",
        "value": "{{geoip_tmp.location.lat}}, {{geoip_tmp.location.lon}}"
      }
    },
    {
      "remove": {
        "field": "geoip_tmp"
      }
    }
  ]
}

Finally I do the bulk import with

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/spamcalcgrp/doc/_bulk?pretty' --data-binary @spamcalcgrp.json

For example 2 lines of this file looks like:

{"index":{"_index":"spamcalcgrp","_id":100000048,"pipeline":"geoip-info"}}
{"@timestamp":"2019-01-01T00:01:33.000Z","nrules":"14","qdir":"qdir-2019-01-01-00.01.33-001","totalscore":"15.774","BAYES_00":"-1.9","LOCAL_QQCOM":"7.0","FREEMAIL_FROM":"4.1","FREEMAIL_ENVFROM_END_D":"0.2","MIME_HTML_ONLY":"0.7","HTML_MESSAGE":"0.0","DCC_CHECK":"1.1","FSL_BULK_SIG":"0.0","LOTS_OF_MONEY":"2.1","HTML_MIME_NO_HTML_TAG":"0.4","UNPARSEABLE_RELAY":"0.0","RDNS_NONE":"0.8","FROM_EXCESS_BASE64":"1.0","FREEMAIL_DISPTO":"0.2","xspam":"1","sspam":"1","fortiip":"114.239.144.252","xlarge":"0","mailfrom":"=?utf-8?B?6LS65qyi6Imz?= <3306194147@qq.com>","rcptto":"<some.user@anywhere>","gateway":"some_gateway"}

Looking at "Discover" I can find a text field geoip for this record
t geoip 32.0617, 118.7778

But when I try to create a new visualization "Coordinate Map" selecting "Geo Coordinates" and aggregation "Geohash" I get the error:
" No Compatible Fields: The spamcalcgrp* index pattern does not contain any of the following field types: geo_point"

Any help is welcome.

Kind regards
Hans

--


(Hans) #2

I fixed the issue. It was during creating the index. Here is the correct definition

PUT /spamcalcgrp
{
  "settings": {
    "index.mapping.total_fields.limit": 1500,
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "doc": {
      "properties": {
        "totalscore": {
          "type": "float"
        },
        "nrules": {
          "type": "integer"
        },
        "xspam": {
          "type": "integer"
        },
        "sspam": {
          "type": "integer"
        },
        "fortiip": {
          "type": "ip"
        },
        "xlarge": {
          "type": "integer"
        },
        "geoip.location": {
          "type": "geo_point"
        }
      }
    }
  }
}

// Hans