Hi @SanderP! Sorry for the late response.
Please ensure you're on 8.3.3 version; there was a performance bug fixed for that release that was present on 8.3.0 - 8.3.2. As you already experienced this on 8.2, I don't believe you're impacted by this, but please make sure you're on 8.3.3 
Hello @Carlos_D , would this still help next to the above debug flag from within the Appsearch call itself? Thanks for your reply?
I've checked your query, and you have already checked its execution time directly on Elasticsearch, so we can rule out the query side.
Enterprise Search does more communication with Elasticsearch under the hood before actually performing the actual query. That could be part of the extra time you are experiencing versus doing the raw Elasticsearch query.
One way we can check where time is actually spent is using Application Performance Monitoring (APM). Enterprise Search supports APM integration, which you can use to understand performance issues.
I suggest checking APM for diving deeper into this issue.