Curl timeout settings - PHP-api

Hi everyone!

I am pretty new to Elasticsearch, sorry if this is too dumb or newbie.
I am having problems with Curl Timeout exceeding 60 seconds and its configuration.

Here is the error message:

FatalErrorException in CurlFactory.php line 169:
Maximum execution time of 60 seconds exceeded

I have tried:

Issue 259 on github
direct setting as params

and a couple more stuff found on stackoverflow:

Stackoverflow - set connect timeout of elasticsearch-php-client

I am integrating elasticsearch with Laravel, in a very simple way. So, here is the class I have developed for this integration. Sorry the mess there is 5.000 character limit, so I had to clean comments up:

    // all variables was declared

    public function __construct() 
    {
        $this->params = array();
        $this->params['hosts'] = $this->host;
        $this->params['guzzleOptions']['command.request_options']['connect_timeout'] = 240; // Standard value 2.0
        $this->params['guzzleOptions']['command.request_options']['timeout'] = 240; // Standard value 2.0
        $this->client = \Elasticsearch\ClientBuilder::create($this->params)
                        ->setHosts([$this->host])
                        ->build();
    }
    
    public function index()
    {
        $this->destroy();
        $this->create();
        $this->map();
        $this->indexFile('File1.pdf');
        $this->indexFile('File2.pdf');
        $this->indexFile('File3.pdf');
        $this->indexFile('File4.pdf');
    }
    
    public function create()
    {
        $this->params = [
            'index' => $this->index
//            ,'client' => [
//               'timeout'           => 0,        // ten second timeout
//                'connect_timeout'   => 0
//             ],            
        ];
       $this->client->indices()->create($this->params);
    }

    public function destroy()
    {
        $this->params = [
            'index' => $this->index
        ];
       $this->client->indices()->delete($this->params);
    }

    public function map()
    {
           $this->params = [
                'index' => $this->index,
                'type'  => $this->type,
                'body'  => [
                    $this->type => [
                        'properties'    => [
                            'file'      => [
                                'type'      => 'attachment',
                                'fields'    => [
                                    'content'   => [
                                        'type'          => 'string',
                                        'term_vector'   => 'with_positions_offsets',
                                        'store'         => true
                                    ]
                                ]
                            ]
                        ]
                    ]
                ]
            ];
            $this->client->indices()->putMapping($this->params);
    }
    
    public function indexFile($file = null)
    {
        if ( empty($file) ){
            return false;
        }
        $file = storage_path('app/' . $this->file_path . $file);
        if ( is_file($file) ) {
            $this->params = [
                'index' => $this->index,
                'type'  => $this->type,
                'timeout' => '5m',
                // Curl timeoute Option 4
//                'client' => [
//                    'timeout'           => 0,        // ten second timeout
//                    'connect_timeout'   => 0
//                ],            
                'body'  => [
                    'file'    => [
                        '_content_type'     => 'application/pdf',
                        '_name'             => $file,
                        '_language'         => 'en',
                        '_indexed_chars'    => -1,
                        '_content'          => base64_encode(file_get_contents($file))
                    ]
                ]
            ];
                $this->client->index($this->params);
                echo 'File "' . $file . '" was indexed. <br>';
        } else {
            return false;
        }
    }
}

Same error using bulk indexing.

Has anyone faced this issue?

Thanks in advance!

1 Like

Heya, maintainer of the PHP client here.

I believe that exception is actually being generated by PHP itself, not the client. By default, most PHP installations only allow scripts to run for 30-60 seconds before it is forcefully terminated, as a protection against long-running or abusive scripts.

You can change the timeout value in your php.ini file, under the max_execution_time setting. You can also set these values dynamically in the script (set_time_limit(...) and ini_set('max_execution_time',...), see http://us2.php.net/manual/en/function.set-time-limit.php). But they may be limited by the max value in the php.ini file, I'm not sure.

Hi polyfractal,

Thank you very much for your attention and your reply!

Unfortunatelly, that was not the case... I set the values, in my php.ini, as below:

max_execution_time = 240
max_input_time = 240
default_socket_timeout = 240

And I am still getting the same error:

FatalErrorException in CurlFactory.php line 169: 
Maximum execution time of 60 seconds exceeded

Would you recommend any other place I could look for?

Best regards,

Hmm. Ok, three things:

  • Do you know what version of Elasticsearch client is being used? The timeout parameters are different, depending on if you are using the 1.x or 2.x branch of Elasticsearch-php

  • What version of PHP and libcurl?

  • It's possible your system has a different php.ini which needs to be changed. E.g. often times, the cli will use one php.ini while Apache/fpm uses a different one. The easiest way to verify is to print out phpinfo() in your browser and verify the php.ini you modified is the one being used by your server.

I'm still leaning towards a configuration problem with PHP, since that FatalErrorException isn't something that is thrown by the client (or Guzzle), it's generated from PHP itself.

Elasticsearch:

"elasticsearch/elasticsearch": "2.1.*"

PHP Version:

PHP 7.0.3 (cli) (built: Feb  4 2016 20:50:17) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies

Curl:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

The only timeout that has 60 seconds is the apache2handler, which I could not get it updated.

Would that be the problem? Any suggestions on how to update that one? Because it is not in the php.ini as I sent the link to you by message.

Thanks again!

Ok, since you are using 2.x, the correct timeout parameters are the ones you have commented out. So you can go ahead and revert back to those. E.g.

$this->params = [
    'index' => $this->index,
    'type'  => $this->type,
    'client' => [
        'timeout'           => 240,       
        'connect_timeout'   => 240
    ],            
    'body'  => [
        'file'    => [
            '_content_type'     => 'application/pdf',
            '_name'             => $file,
            '_language'         => 'en',
            '_indexed_chars'    => -1,
            '_content'          => base64_encode(file_get_contents($file))
        ]
    ]
];
$this->client->index($this->params);

To be honest, I'm not sure about Apache. It's been years since I've used Apache. It very well could be that timeout...if Apache decides to kill the process because it has been running too long, that'd look very similar to PHP killing it for the same reason. Perhaps there is a way you can override it in a .htaccess file?

Does this timeout only happen when indexing the PDFs? E.g. if you run through all your code, but omit the PDF indexing, does it work?

I checked the error source line (169 in CurlFactory of RingPHP), and it's assigning an anonymous function to a value in an array. Which makes me believe this really is a forceful termination, since that's an odd place for an exception.

I have tested with this configuration as well, with no success.

The code runs perfectly even when indexing file, the problem is when we index multiple files, as the sample. If the whole function ('index() function') takes more that 60 seconds it brings the error. So, in that case, if I index one file of a time, it will work fine. But I have a couple more than 500 pdf files to index (average of 2Mb each).

I will try with .htaccess...

Thanks again. Whenever I find a way out I will post it here for future references.

Best regards,

Yeah, I unfortunately expected that :disappointed:

I'll keep digging on my end and see if there is something I can uncover regarding apache.

1 Like

Well, still not working...

I have settled the apache TimeOut to 240 and through .htaccess and still getting this error.

I keep it informed.

Thanks

Hello Polyfractal!

Have you had any good results on that? Am I the only one having this problem?

I forgot to say.. I am using Laravel (and Lumen for the API) framework... shouldn't be the issue though. Only for reference.

Thanks

Sorry, I never discovered anything outside the usual suspects (apache config, firewall, php time limit, etc).

If you want, you can open a ticket at the repo and we can keep exploring the issue there. I keep a closer eye on issues there than the forum (where it's easy to lose individual threads due to volume)

1 Like

Thanks @polyfractal!

I will make some extensive tests before posting an issue. We keep in touch.

Best regards!

Hi @polyfractal,

I have tested many options and found out what the error was... in php.ini has a file size upload limit, besides some other configurations as default_socket_timeout which was the seconds on the php error message was coming from. Only configuration I hadn´t change since our last talk. No sure if that would really change anything.

So, the error was on the file size configuration in php.ini, even if php does not show the correct error.

Thanks again and best regards!

1 Like

Ah, interesting. Good sleuthing to find that. Wasn't aware that value would have an affect on the curl connections. Good to know :slight_smile: