Geohash graph

graph

(Ahmed HADDAD) #1

I m planning to create a graph to visualize the relation between different cabs pickups location in Nyc city.
so i choosed the index named "green" (for green cabs) and the field pickup_location (which is type is geohash), the result is 500 server error !?


(Mark Harwood) #2

Can you share the mapping and the stacktrace?

I expect you want to map the value as a keyword type if you are using it as a location identifier - geo fields are optimised for spatial range queries rather than ID-type lookups.


(Ahmed HADDAD) #3

this is the stacktrace i took a screenshot


what you mean by mapping ? template mapping for geopoint ?


(Mark Harwood) #4

Mappings are effectively the schema for your index - see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

Here's an example of trips from JFK to Brooklyn and Queens which you can paste into Kibana's console:

DELETE test
PUT test
{
  "mappings": {
	"trip": {
	  "properties": {
		"geo": {
		  "type": "geo_point",
		  "fields":{
			"keyword":{
			  "type":"keyword"
			}
		  }
		}
	  }
	}
  }
}

POST test/trip/_bulk
{"index":{}}
{ "geo":["dr5x2bn", "dr5x8v1"] }
{"index":{}}
{ "geo":["dr5x2bn", "dr5rmks"]}

GET test/_search
{
  "size": 0,
  "aggs": {
	"terms": {
	  "terms": {
		"field": "geo.keyword"
	  }
	}
  }
}

Note that I mapped the JSON value as both a geo_point field and a keyword field - one is for spatial queries, the other for ID-based queries of the sort Graph uses.


(Ahmed HADDAD) #5

Ok, understood i this is the mapping of the index green

{
  "green": {
"mappings": {
  "traj": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Dropoff_location": {
        "type": "geo_point"
      },
      "Extra": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Fare_amount": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Lpep_dropoff_datetime": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "MTA_tax": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Passenger_count": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Payment_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "PickupDate": {
        "type": "date"
      },
      "Pickup_location": {
        "type": "geo_point"
      },
      "RateCodeID": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Store_and_fwd_flag": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Tip_amount": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Tolls_amount": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Total_amount": {
        "type": "float"
      },
      "Trip_distance": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "Trip_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "VendorID": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "lpep_pickup_datetime": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "message": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "path": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  },
  "_default_": {
    "properties": {
      "Dropoff_location": {
        "type": "geo_point"
      },
      "Pickup_location": {
           "type": "geo_point"
         }
       }
     }
   }
}
}

(Ahmed HADDAD) #6

so ?


(Mark Harwood) #7

I suggest you create a new field called "my_location_keywords" mapped as a keyword type.
You should pass docs, as in my example, where the pickup and dropoff points are expressed as 2 truncated geohashes in the "my_location_keywords" array. Why truncated? Full-resolution geohashes will likely lead to a graph with millions of nodes each containing exactly one doc per connection - every full-precision pickup/dropoff location may be unique. If you truncate the geohash according to this table you'll "round up" to a higher level view of the data.

If your original data is lat/lon you can find geohash implementations for a variety of languages.


(Ahmed HADDAD) #8

Ok, I did as you told me:
i changed my mapping for the index like this :

PUT _template/test
{
  "order":0,
  "index_patterns":"test",
"mappings":{
  "_default_":{
    "properties":{
      "Pickup_location":{
        "type":"geo_point"
        ,"fields":{
          "keyword":{
            "type":"keyword"
          }
        }
      },
      "Dropoff_location":{
        "type":"geo_point"
        ,"fields":{
          "keyword":{
            "type":"keyword"
             }
           }
         }
       }
     }
   }
}

then i indexed with logstash the test index.
the result i still don't see the graph created when i choice test as index and pickup_location.keyword as a field, but the error is gone.
i don't undesrstand the truncated part of geohash since the ingestion is done by logstash


(Mark Harwood) #9

You need to write some code or dig deeper into what logstash can do for you in terms of document preparation.


(Ahmed HADDAD) #10

ok , but even your example dosen't work for me.
i chosed the pickup_location as a field


(Mark Harwood) #11

Working here with these GUI settings on 6.2.3


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.