Sending multiple _count requests in one go

Yuval_Khalifa · March 8, 2019, 8:43am

Is it possible to send multiple requests to the _count API in one go? I saw this post Does count API support searching by multiple TERMs and return multiple COUNTs? but from what I can understand, the recommendation to use the multi-search API and set the search_type to count won't work since that it can only be set to either "dfs_query_then_fetch" or "query_then_fetch". Is that correct?

What I was hoping for is to be able to send a request like this to Elastic:
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.10""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.172""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.141""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.153""}}}

and receive as a response the counts for each query in an array or something similar.

Is this doable?

dadoonet · March 8, 2019, 8:59am

You can run multiple filter aggregations or you can use the multisearch API.

Yuval_Khalifa · March 8, 2019, 9:16am

Thank you for your quick reply...

How can I get only the counts with the multisearch API? I couldn't find any example of that? For example, how would my example above would be written if I wanted to get only the counts from the multisearch API?

dadoonet · March 8, 2019, 10:53am

Reading the doc here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html

$ cat requests

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/logstash-*/_msearch --data-binary "@requests"; echo

That should work I think.

Yuval_Khalifa · March 8, 2019, 11:20am

Thanks a lot. It worked, although I had to escape the quotes like this:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Thanks a LOT!!
Yuval

Yuval_Khalifa · March 8, 2019, 11:25am

Ehmmm.... I tried it now and I noticed that I only received responses for two of my queries and I'm wondering where are the rest of them:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 
{
  "responses" : [
    {
      "took" : 1992,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 2259,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 1987,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 36,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}

dadoonet · March 8, 2019, 11:39am

The blank lines I added in my example are super important. Don't skip them.

Yuval_Khalifa · March 8, 2019, 11:43am

Well if add them back like so:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

The result is this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}

dadoonet · March 8, 2019, 11:44am

Read my example again. To be clear:

$ cat requests

Gives:


{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Yuval_Khalifa · March 8, 2019, 11:50am

I'm sorry if I'm missing something stupid here... but I tried again using exactly your example (the only thing that I changed was the filename and the fact that I added '?pretty' to the URL)

$ cat tmp_bulk_queries_example 

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

The result is this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}

Am I missing something stupid here?

Yuval_Khalifa · March 8, 2019, 11:59am

Ok, I think I figured it out with your help... When I use this syntax it seems to work perfectly:

$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Then, the result looks like this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "responses" : [
    {
      "took" : 6897,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 788182,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6904,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 2259,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6844,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 612,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6848,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 36,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}

and if I use jq it looks fine:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null | jq '.responses[]["hits"]["total"]'
788182
2259
612
36

Thanks a LOT!

dadoonet · March 8, 2019, 12:07pm

Yeah sorry. It probably requires at least a non blank line.

Could you do me a favor? Could you try with this syntax?

$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Yuval_Khalifa · March 8, 2019, 12:19pm

Well, this input:

smartonion@smart-onion:~$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}


$

resulted in this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}

dadoonet · March 8, 2019, 1:35pm

What version are you using? I can't reproduce that problem locally on 6.6.1.

DELETE /test
PUT /test/_doc/1
{
  "foo": "bar"
}

$ cat requests

{}
{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

curl -H "Content-Type: application/x-ndjson" -XGET "http://127.0.0.1:9200/test/_msearch?pretty" --data-binary "@requests"; echo

Gives:

{
  "responses" : [
    {
      "took" : 6,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 5,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    }
  ]
}

Yuval_Khalifa · March 8, 2019, 1:39pm

Well it's only 6.6.0 here:

$ curl 10.253.0.153:9200
{
  "name" : "N4g8_lf",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "vTGsAaa9T7C8BbQNV0iaiw",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Yuval_Khalifa · March 8, 2019, 1:42pm

Perhaps it caused by the copy-pasting to/from the forum. Maybe if you'll attach the queries file to the message it would work correctly.

dadoonet · March 8, 2019, 1:43pm

Ok. FYI I sent a doc fix here:

system · April 5, 2019, 1:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
No MultiCountRequest API Object in Java Elasticsearch	2	473	December 19, 2019
Is there a "Multi Count API" in ES Elasticsearch	3	2833	December 26, 2017
Is there any way to fire multiple query against a index in one request Elasticsearch	3	495	May 27, 2019
Most efficient way to do multiple query counts Elasticsearch	2	605	July 5, 2017
Multiple requests in one query Elasticsearch	3	7970	July 6, 2017

Sending multiple _count requests in one go

Related Topics