Sending multiple _count requests in one go

Is it possible to send multiple requests to the _count API in one go? I saw this post Does count API support searching by multiple TERMs and return multiple COUNTs? but from what I can understand, the recommendation to use the multi-search API and set the search_type to count won't work since that it can only be set to either "dfs_query_then_fetch" or "query_then_fetch". Is that correct?

What I was hoping for is to be able to send a request like this to Elastic:
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.10""}}}
{"index": "logstash-
"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.172""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.141""}}}
{"index": "logstash-
"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.153""}}}

and receive as a response the counts for each query in an array or something similar.

Is this doable?

You can run multiple filter aggregations or you can use the multisearch API.

Thank you for your quick reply...

How can I get only the counts with the multisearch API? I couldn't find any example of that? For example, how would my example above would be written if I wanted to get only the counts from the multisearch API?

Reading the doc here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html

$ cat requests

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}
$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/logstash-*/_msearch --data-binary "@requests"; echo

That should work I think.

Thanks a lot. It worked, although I had to escape the quotes like this:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Thanks a LOT!!
Yuval

Ehmmm.... I tried it now and I noticed that I only received responses for two of my queries and I'm wondering where are the rest of them:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 
{
  "responses" : [
    {
      "took" : 1992,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 2259,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 1987,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 36,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}

The blank lines I added in my example are super important. Don't skip them.

Well if add them back like so:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

The result is this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}

Read my example again. To be clear:

$ cat requests

Gives:


{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

I'm sorry if I'm missing something stupid here... but I tried again using exactly your example (the only thing that I changed was the filename and the fact that I added '?pretty' to the URL)

$ cat tmp_bulk_queries_example 

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

The result is this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}

Am I missing something stupid here?

Ok, I think I figured it out with your help... When I use this syntax it seems to work perfectly:

$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Then, the result looks like this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "responses" : [
    {
      "took" : 6897,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 788182,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6904,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 2259,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6844,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 612,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6848,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 36,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}

and if I use jq it looks fine:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null | jq '.responses[]["hits"]["total"]'
788182
2259
612
36

Thanks a LOT!

Yeah sorry. It probably requires at least a non blank line.

Could you do me a favor? Could you try with this syntax?

$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Well, this input:

smartonion@smart-onion:~$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}


$ 

resulted in this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}
1 Like

What version are you using? I can't reproduce that problem locally on 6.6.1.

DELETE /test
PUT /test/_doc/1
{
  "foo": "bar"
}
$ cat requests
{}
{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}
curl -H "Content-Type: application/x-ndjson" -XGET "http://127.0.0.1:9200/test/_msearch?pretty" --data-binary "@requests"; echo

Gives:

{
  "responses" : [
    {
      "took" : 6,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 5,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    }
  ]
}

Well it's only 6.6.0 here:

$ curl 10.253.0.153:9200
{
  "name" : "N4g8_lf",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "vTGsAaa9T7C8BbQNV0iaiw",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Perhaps it caused by the copy-pasting to/from the forum. Maybe if you'll attach the queries file to the message it would work correctly.

Ok. FYI I sent a doc fix here:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.