Scan and Scroll - Memory Usage 2.2.0

We just upgraded to ES 2.2.0 and are using the Scroll method to read bunches of documents in C# and doing something to them and then updating them in another linked system.

We clear the scroll ID's after they are used, we use _doc and we have tried cleaning the cache but our System Memory spikes by 10-20% each time we run the queries which yes read every document in a 5 million doc index.

Any other ideas to help us force memory back down?