Sending multiple _count requests in one go

Elastic Stack Elasticsearch
1

Is it possible to send multiple requests to the _count API in one go? I saw this post Does count API support searching by multiple TERMs and return multiple COUNTs? but from what I can understand, the recommendation to use the multi-search API and set the search_type to count won't work since that it can only be set to either "dfs_query_then_fetch" or "query_then_fetch". Is that correct?

What I was hoping for is to be able to send a request like this to Elastic:
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.10""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.172""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.141""}}}
{"index": "logstash-"}
{"query": {"query_string": {"query": "destination_ip:"10.253.0.153""}}}

and receive as a response the counts for each query in an array or something similar.

Is this doable?

2

You can run multiple filter aggregations or you can use the multisearch API.

3

Thank you for your quick reply...

How can I get only the counts with the multisearch API? I couldn't find any example of that? For example, how would my example above would be written if I wanted to get only the counts from the multisearch API?

4

Reading the doc here: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html

$ cat requests

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/logstash-*/_msearch --data-binary "@requests"; echo

That should work I think.

5

Thanks a lot. It worked, although I had to escape the quotes like this:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Thanks a LOT!!
Yuval

6

Ehmmm.... I tried it now and I noticed that I only received responses for two of my queries and I'm wondering where are the rest of them:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 
{
  "responses" : [
    {
      "took" : 1992,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 2259,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 1987,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 36,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}
7

The blank lines I added in my example are super important. Don't skip them.

8

Well if add them back like so:

$ cat tmp_bulk_queries_example 
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

The result is this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}
9

Read my example again. To be clear:

$ cat requests

Gives:


{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}
10

I'm sorry if I'm missing something stupid here... but I tried again using exactly your example (the only thing that I changed was the filename and the fact that I added '?pretty' to the URL)

$ cat tmp_bulk_queries_example 

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

The result is this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}

Am I missing something stupid here?

11

Ok, I think I figured it out with your help... When I use this syntax it seems to work perfectly:

$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}

Then, the result looks like this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "responses" : [
    {
      "took" : 6897,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 788182,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6904,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 2259,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6844,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 612,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    },
    {
      "took" : 6848,
      "timed_out" : false,
      "_shards" : {
        "total" : 95,
        "successful" : 95,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 36,
        "max_score" : 0.0,
        "hits" : [ ]
      },
      "status" : 200
    }
  ]
}

and if I use jq it looks fine:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null | jq '.responses[]["hits"]["total"]'
788182
2259
612
36

Thanks a LOT!

12

Yeah sorry. It probably requires at least a non blank line.

Could you do me a favor? Could you try with this syntax?

$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}
13

Well, this input:

smartonion@smart-onion:~$ cat tmp_bulk_queries_example 
{}
{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.10\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.172\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.141\""}}}

{"size": 0, "query": {"query_string": {"query": "destination_ip:\"10.253.0.153\""}}}


$

resulted in this:

$ curl -H "Content-Type: application/x-ndjson" -XGET 'http://10.253.0.153:9200/logstash-*/_msearch?pretty' --data-binary "@tmp_bulk_queries_example" 2>/dev/null
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Expected [START_OBJECT] but found [null]",
        "line" : 1,
        "col" : 0
      }
    ],
    "type" : "parsing_exception",
    "reason" : "Expected [START_OBJECT] but found [null]",
    "line" : 1,
    "col" : 0
  },
  "status" : 400
}
1 Like
14

What version are you using? I can't reproduce that problem locally on 6.6.1.

DELETE /test
PUT /test/_doc/1
{
  "foo": "bar"
}

$ cat requests

{}
{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

{ "query": {"match_all": {}}}

curl -H "Content-Type: application/x-ndjson" -XGET "http://127.0.0.1:9200/test/_msearch?pretty" --data-binary "@requests"; echo

Gives:

{
  "responses" : [
    {
      "took" : 6,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 5,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 4,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 1,
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : 1.0,
            "_source" : {
              "foo" : "bar"
            }
          }
        ]
      },
      "status" : 200
    }
  ]
}
15

Well it's only 6.6.0 here:

$ curl 10.253.0.153:9200
{
  "name" : "N4g8_lf",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "vTGsAaa9T7C8BbQNV0iaiw",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
16

Perhaps it caused by the copy-pasting to/from the forum. Maybe if you'll attach the queries file to the message it would work correctly.

17

Ok. FYI I sent a doc fix here:

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.