Elasticsearch Input Seems To Not Use Provided Query - It Just Dumps All docs in Index

Elastic Stack Logstash
1

Hello, I am trying to use the Elasticsearch input but keep running into trouble. It seems like the input does not care about the query I give it. Instead, I just see logstash spew out every document in the index.

This is my logstash config -

    input {
  # Returns all campaigns in a day with >100,000 emails sent and stores the output in a file.
  elasticsearch {
    hosts => [localhost]
    query => {
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*",
            "analyze_wildcard": true
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-2d/d",
              "lte": "now-1d/d",
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "_source": {
    "excludes": []
  },
  "aggs": {
    "3": {
      "terms": {
        "field": "Campaign_ID.keyword",
        "size": 1000,
        "min_doc_count": 100000,
        "order": {
          "_count": "desc"
        }
      }
    }
  }
}
  	index => "powermta-*"
  }
}
output {stdout { codec => rubydebug }}

The output when I run this query is

  "took": 531,
  "timed_out": false,
  "_shards": {
"total": 385,
"successful": 385,
"failed": 0
  },
  "hits": {
    "total": 7705389,
"max_score": 0,
"hits": []
  },
  "aggregations": {
"3": {
  "doc_count_error_upper_bound": 0,
  "sum_other_doc_count": 0,
  "buckets": [
    {
      "key": "34748",
      "doc_count": 893660
    },
    {
      "key": "50059",
      "doc_count": 770699
    },...

Which is what I want.

Could someone help me figure out why the Elasticsearch input is not giving me the data I want?

2

I'm very surprised that your configuration doesn't upset the config file syntax checker. AFAIK the query option is supposed to contain a (JSON) string. See examples in the docs.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.