Rate limit exceeded

Elastic Search
1

I currently have a demo account. I am trying to make a POC using SwifType for my employer. We have a very big database that is indexec every 1h and creates a JSON file. I thought the integration with Elastic would be very easy considering it's only a mater of sending the string when it's generated. I used PHP Curl and got a connection to the API. The code sends out part of the data and then freaks out with a "Rate limit exceeded" error.

How can I manage around that error and get the full JSON indexed?

My code looks like this at the moment:

// SENDING DATA TO ELASTIC SEARCH
$arr = array_change_key_case($arr, CASE_LOWER); // Keys to lower case
$arrlist = array_chunk($arr,100); // Split to chunks of 100
foreach($arrlist as $key=>$arr){
    $json = json_encode($arr); // Making the JSON string from the array
    $ch = curl_init('https://host-***.api.swiftype.com/api/as/v1/engines/***/documents');                                                                      
    curl_setopt($ch, CURLOPT_POST, 1);                                                                     
    curl_setopt($ch, CURLOPT_POSTFIELDS, $json);                                                                  
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);      
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);                                                                
    $headers = array();
    $headers[] = 'Content-Type: application/json';
    $headers[] = 'Authorization: Bearer private-***';
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    
    $result = curl_exec($ch);
    if (curl_errno($ch)) {
        echo 'Error:' . curl_error($ch);
    }
    curl_close($ch);
    
    echo $result."<hr>";
}

Also, considering this code is going to be indexed every hour, if I am sending the same data over and over, will it UPDATE the previous one or will it duplicate it? If so, how can I manage that?

2

Hey @Patrick_Simard,

Sorry that you are having trouble getting your content indexed. Here is a knowledge base article on Rate Limiting: https://swiftype.com/documentation/site-search/rate_limiting.

Also, check out the documentation on documents, you can use the create_or_update endpoint: https://swiftype.com/documentation/site-search/indexing#add-document.

Jason

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.