GeoIP Pipeline and Geopoint

Elastic Stack Elasticsearch
1

Hi I have a pipeline that remaps an IP address to location points using GeoIP. When I import data via the bulk API and pass it through the pipeline I can see the added fields in Kibana.

The issue is Lon and Lat are separate fields and not set as GeoPoint and therefore I cannot map these? - The index mapping is automatic and has put them as a number.

I am an ELK stack n00b so any help would be much appreciated. I am just running a local instance on my Mac and installed via brew.

Question - 2 is there anything I can add to an index mapping that imports the rest of the data if a field is missing. I.e I have a field with a date that only exists 50% of the time however if that field is missing then when indexing it errors and does not import.

Thanks

2

You need to define the mapping first.

3

When I tried adding an index mapping due to the data not being in the JSON as it is added by the pipeline it errors.

Do I map each Lon and Lat field individually as a Geo point?

Thanks for the help!

4

No. The location field must be a geopoint.

5

I tried a simple mapping:

PUT /waf/_mapping/location?pretty
{
      "properties": {
        "location": {
          "type":   "geo_point"
        }
      }
    }

I run the import and get

{"index":{"_index":"waf","_type":"logs","_id":"3215102fa4eb0c9b","status":400,"error":{"type":"illegal_argument_exception","reason":"[geoip.location] is defined as an object in mapping [logs] but this name is already used for a field in other types"}}}]

I am sure I am missing something simple.

Thanks again!

6

You probably indexed already documents or have a mapping.
You need to start from scratch as you can't change the existing mapping .

7

comment - deleted

8

Run GET yourindexname/_mapping to see what your mapping is.

9

Sorry David you were correct this was in my mapping. However I am struggling to map lon and lat to string and the location to a geo_point it keeps treating them as separate and not the correct method for a geo point. The closest I got was lat and lon were string but also managed to create a new field called geopoing.location.loacation which saved as a geo point but stored no data as there was not matches

      },
  "geoip": {
    "properties": {
      "location": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          }
        }
      }
    }
10

Hard to tell without knowing what you are exactly doing.

Could you share a script which reproduce what you are doing?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.