Scroll/Scan API, pockets of slow responses while scrolling

Elastic Stack Elasticsearch
1

Using ES 1.6.0

I have the following query using Transport Client

client.prepareSearch("my-index").setTypes("my-type")
				.setQuery(query.getBytes())
				.setSearchType(SearchType.SCAN)
				.addFields(
						"my-field-1",
						"my-field-2"
						)
						.setScroll(new TimeValue(60000))
						.setSize(500)
						.execute().actionGet();

The query...

{
	"filtered": {
      "filter": {
        "bool": {
          "must": [
            {
              "term": {
                "my-user": "100000"
              }
            },
            {
              "range": {
                "my-date": {
                  "gte": "2015-03-01",
                  "lte": "2015-03-31"
                }
              }
            },
            {
              "bool": {
                "should": [
                  {
                    "term": {
                      "shape": "triangle"
                    }
                  },
                  {
                    "term": {
                      "shape": "circle"
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    }
}

The query has been redacted to protect the innocent :smile: (It's the actual query just field names/values where renamed)

The index is 260 million documents, the query produces 30 million hits.

So on each scroll request I print the took ms using, scrollResp.getTookInMillis(). I see pockets of slowness. So some scroll requests will burst with took time of 50ms and other times each scroll will take 4-5 seconds and then it will burst again. And some times each scroll request will humm around 300ms.

2

It could be many things! What are you monitoring ES with?

3

Windows perfmon, fusion io manager and marvel

4

I also find it very spiky. For instance all the io stats are very spiky. With fusion io monitoring I see lots of io spike but nothing consistant. I seen it go all the way to 1GB per second reads some time almost idle sometimes it hums along around 50mb per second reads. Other time it spikes up to 500mb then drops then again spikes and drops. For the same scroll operation

5

Iops.png721×357 157 KB

As you can see it's not very consistent.