Geo_shape query processing not thread-safe which causes issues with parallel shard requests

[robqliu]

We're on ES 6.8.x and we have an integration test that spins up an in-process ES node and it appears to be processed non-deterministically. It creates a geo_shape query and occasionally we trigger this assert in JtsGeometry:

private static Geometry cutUnwrappedGeomInto360(Geometry geom) {
    Envelope geomEnv = geom.getEnvelopeInternal();
    if (geomEnv.getMinX() >= -180.0D && geomEnv.getMaxX() <= 180.0D) {
        return geom;
    } else {
        assert geom.isValid() : "geom";

But the reason why it's happening is actually because of a check-then-act race in PolygonBuilder:

/**
 * Transforms coordinates in the eastern hemisphere (-180:0) to a (180:360) range
 */
private static void translate(Coordinate[] points) {
    for (Coordinate c : points) {
        if (c.x < 0) {
            c.x += 2*DATELINE;
        }
    }
}

And the only reason this is even happening concurrently on the same Coordinate is because both shards in our single node are concurrently processing the same exact *Query instance.

I was wondering if this is an issue only with our tests, but tracing up the stack in ES code, it doesn't seem like it is (i.e. we could see this issue in the field). Happy to be wrong. Right now, it seems like the only way to remediate this is to set maxConcurrentShardRequests on the request client-side (OR changing the # of shards in our test to 1).