App Search Rate limiting search requests?

Elastic Search
1

Hi,

I currently have a cloud instance of elastic and an application, the application sends a lot of search request to App Search, around 2-5/s however some of these search requests return an empty body*, if I then wait for a bit and resend the exact same request I get a perfect result set.

With empty body I mean:

{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 8,
    "engine": {
      "name": "engine-name-qa",
      "type": "default"
    },
    "page": {
      "current": 1,
      "total_pages": 0,
      "total_results": 0,
      "size": 10
    },
    "request_id": "1234"
  },
  "results": []
}

Note: No alerts or warnings!

I was thinking this was because of rate limiting however I am using a Private key so I would think app search would take that into account and not rate limit requests with a private key?

Another reason I was thinking of would be because we do not have enough resources? (I am starting to think it is because of ES resources however still weird that app search just returns an empty search response instead of telling me something is wrong.)

This QA deployment has the following resources:
ES 1 zone, 1GB ram, 8vCPU's
KIB 1 zone, 1GB ram, 8vCPU's
ENT Search, 1 zone 2GB ram & 8 vCPU's

Any help would be appreciated.

2

Hi @Chenko ,

App Search does not have some rate limiting behaviors. You can read about the limits it does have here: Limits | App Search documentation [8.13] | Elastic

When App Search does encounter a limit, or under the hood Elasticsearch hits a limit, you won't receive an empty result set, but an error response.

I can think of two things that might be causing the behavior you're seeing:

  1. Your Elasticsearch is unhealthy, and having shard errors. This means that depending on which node services the App Search request, you're getting different/no results. Check your Elasticsearch health.
  2. Your App Search engine is having a lot of underlying write operations (documents ingested/deleted) while these queries are running. So the queries are actually accurate, in that the available document matches change from one moment to the next.
3

Hi @Sean_Story Thanks for the swift response!

  1. The cluster is healthy except for only having one zone which couses its health to be Healthy, with warnings.
  1. No writes are being done when this process is happening.

Also may I add that when the app first launches the first +-100 requests are super quick with perfect results, however after the first time getting no results it happens every second, 2 seconds sleeping after encountering no documents, getting 2 documents, and it happens again, in a loop..

Also I forgot to add We are using v8.11, might be an issue that is not on the latest version(8.13)?

4

Hey there @Chenko along with @Sean_Story 's suggestions I would like to know if the Enterprise Search logs are reporting any warnings or errors when you get back the empty result sets? Also, when empty responses are returned what is the HTTP status code? That may help better diagnose exactly what's happening.

5

Hi @Kathleen_DeRusso ,

Do you mean the App Search API Logs? if so, none.
If you mean the logs stored in the logs-enterprise_search-api... I also do not see any errors here.

Both the App Search API logs and logs index tell me it returns a 200, without errors.

6

Thanks. What about the Elasticsearch logs? Are there any errors logged there?

7

I am not sure if I can see these as the metrics & logging is not enabled in the cloud. Is there still a way to see this? We did not enable this as we would most likily need more resources in our QA deployment.

8

I'd missed that this is on a cloud instance.

I suggest reaching out to support directly. As an Elastic Cloud customer (even during a trial period), you're entitled to support services through http://support.elastic.co

They'd be able to look at your Elasticsearch and Enterprise Search logs and help you debug the issue more effectively than we can through this forum.

1 Like
9

Thanks, I did not realize this, I thought it was only for platinum and above!

10

I am looking at the support options however I see that technical support is unavailable for this account. We do however currently have an active deployment with ECU's (Maybe because we are using ECU's? However we still should have a gold subscription).

Screenshot:

image
image795×427 25.8 KB

11

Hmm, IDK why that's greyed out for you. Can you email support@elastic.co and ask if they can tell you why that's not available? I don't want to have you provide any identifying information in this thread, which is why I'm suggesting you shoot them an email.

1 Like
12

Yes, indeed thanks for the help!