Elasticsearch-PHP [8.8] - Search for field in date-range, Client Helpers SearchResponseIterator & SearchHitIterator

The poor documentation of Elasticsearch continues to hamper expanding my usage, and even poorer vagueness in the PHP API documentation.

This seems like a simple example. Search for field (it is a tag field, it can have multiple values ) in this cases 'event_source' (*) over a date range, say last 24 hours. Something simple, that is like the basic operation of Kibana's discover area.

Here is exactly what I want to do :

Which also says, my fields are there, data is there, index, etc, ... this seems to be purely some sort of undocumented abstraction issue with the PHP API.

As far as I can see, this is the way to do so ... and it returns nothing:

$json='{
     "query": {
            "range" : {
                "date" : {
                    "gte" : "now-24h",
                    "lt" :  "now"
                }
            }
        }
}
';
$search_params = [
    'scroll'      => '1m', // period to retain the search context
    'index'       => 'filebeat-*', // here the index name
    'size'        => 10, // 100 results per page
    'body'        => json_decode($json,false),
];
$url = $opt['es'];
$client = Elastic\Elasticsearch\ClientBuilder::create()
    ->setHosts($url)
    ->setBasicAuthentication('elastic', 'xxxx')
    ->setCABundle('le-ca.pem')
    ->build();

$pages = new SearchResponseIterator($client, $search_params);
$hits = new SearchHitIterator($pages);
foreach($hits as $hit) {
    echo $hit['_id'] . " " .  $hit['_source']['@timestamp'] . " " .  $hit['_source']['message'], PHP_EOL;
}

Any other kind of query ... changing query with ranges or match ... (this is only having the date range) as to just match on a single single field ... will either return nothing, or a syntax error:

... "parsing_exception","reason":"[match] malformed query, expected [END_OBJECT] but found [FIELD_NAME]", ... 

Yes, changing the query to, as the example says ....

    'body'        => [
        'query' => [
            'match_all' => new StdClass // {} in JSON
        ]
    ]

Does in fact work ... but of course, this is returning ... everything?

I have exhausted all the examples, in both the PHP and raw API documentation, reached the end of google, and searches here in the forum turned up just about nothing.

Have the latest ...

  • elasticsearch/elasticsearch v8.8.2
  • php 8.2.x

... well a new day ... and things suddenly work ...

I did find some of my old php code before using the helpers, and I located a syntax that works ... what is exactly different, it's still not clear ... but for record ... (in php array syntax) :

$search_params = [
    'scroll' => '5s',
    'size'   => 10,
    'index'  => 'filebeat*',
    'body'   => [
        'query' => [
            'bool' => [
                'must' => [
                    [ 'match' => [ 'event_source' => 'apache_access' ] ],
                ],
                'filter' => [
                    'range' => [
                         '@timestamp' => [
                            "gte" => "now-1h",
                            "lt" => "now"
                         ],
                        ],
                    ],
            ],
        ],
    ],
];

And because the schema of the page (return of SearchResponseIterator()) object/class is not readily available ....

Array
(
    [_scroll_id] => xxxx
    [took] => 1390
    [timed_out] =>
    [_shards] => Array
        (
            [total] => 2798
            [successful] => 2798
            [skipped] => 2774
            [failed] => 0
        )

    [hits] => Array
        (
            [total] => Array
                (
                    [value] => 4257
                    [relation] => eq
                )

            [max_score] => 11.303201
            [hits] => Array
                (
                    ....

Likely because pages is a an Iterator over scrolls ... you can't directly access data in pages, so ... to get the total hits from your search ... (the example on the web counts the hits pre scroll-page):

foreach($pages as $page) {
    print_r ($page);
    echo "Total Hits: " . $page['hits']['total']['value'] . "\n\n";
    break;
}

Then you can Iterate over the hits ....

$hits = new SearchHitIterator($pages);
foreach($hits as $hit) {
    echo $hit['_id'] . " " .  $hit['_source']['@timestamp'] . " " .  $hit['_source']['message'], PHP_EOL;
}

Also note, that your retrieved document is actually in $hit['_source'] ... the wrapper of this object is:

                    [8] => Array
                        (
                            [_index] => filebeat-8.2.3-2023.06.30
                            [_id] => aI8CDYkBmKV2FGzRFNTd
                            [_score] => 6.284789
                            [_source] => Array
                                (
                               ....

So the client helpers are nice ... just wish there were some more helpful & usable examples at the top level documents.

Elastic: I release this into public domain ... please update your examples ! :slight_smile:

Hopefully this saves someone else a lost of day of work.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.