Geospatial search between polylines[multiple geo_point] and points

Hi
Sorry if it is too long, I'll try to keep the content short.

My scenario is I have many polylines (which basically are array of geo_points with directions, and I can store the index of each point) indexed as my documents for search. And I need to query out with polyline is the closest one give two geo_points with direction(say start_point and end_point).
For that, in theory what I need to do is

  1. calculate score using start_point for all points in polyline, and get the max. also the index of the geo_points in the polyline say index_one
  2. do the same thing using end_point and get a max_score plus another index of the geo_point say index_two
  3. Sum the score from step 1 and 2.
  4. exclude the polylines where index_one is bigger than index_two(which means the polyline's direction is different from the points direction).

I've tried a few approach, but no luck so far.

  1. Simple treat the polyline as array of geo_points works great for calculating score, but I can't get the index of the scored geo_point of that polyLine.
  2. I've tried to index the points as nested documents but the problem is seems I can't query one nested document field twice (or at least can't see the hit). Also I've heard that nested document cost a lot of resource and likely will cause performance problem.

Just wondering did any one done something similar? What are the possible approaches to try with?
And what would be the possible performance issue?

Thanks for your time in advance