Logstash query template

Elastic Stack Logstash
1

Change from original question

{
  "query": {
    "bool": {
      "must": {
        "query_string": {
          "query": "id=%{id}"
        }
      },
      "filter": {
        "range" : {
            "timestamp" : {
              "gt" : "now-1h",
              "lt" : "now"
            }
        }
      }
    }
  }
}

I paste this into a template.json file and changed my elasticsearch to

elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "${input}"
                query_template => "template.json"
}

And this results in an error. reason=>"Something is wrong with your configuration."
Not sure how to work around this? I looked at samples on elastic.co and looks as simple as this. I tried copying the whole path, but did not solve the problem.

2

Why not just use a date range filter in your query? I believe reasonably recent versions of ES are quite efficient in skipping indexes that obviously don't contain any documents in range (i.e. using logstash-* isn't significantly slower)

3

Wasn't sure if applying a date range would have applied to looking for the correct index. Am I suppose to write it within parameters filter of elasticsearch? I came up with a solution by parsing the timestamp and aggregate it to my index query name instead of using a 2-days date range. Not sure about the solution tho, do you have any opinion/suggestion?

4

Wasn’t sure if applying a date range would have applied to looking for the correct index.

You don't need to figure out which indexes to query. Just use a wildcard.

Am I suppose to write it within parameters filter of elasticsearch?

What do you mean by "parameters filter"?

5

If I wanted to apply a date range in the query, how would I use it within elasticsearch plug in? This ElasticSearch Parameters Plugin

Say give or take you have 90 indexes, would wildcard vs specific index contribute to speed? I thought it would since took's average value is less than that of a wildcard search

6

If I wanted to apply a date range in the query, how would I use it within elasticsearch plug in? This Elasticsearch Parameters Plugin

That's the elasticsearch output plugin, but you seem to be asking about the elasticsearch input or filter plugin.

Say give or take you have 90 indexes, would wildcard vs specific index contribute to speed? I thought it would since took's average value is less than that of a wildcard search

Then I'm surprised. I was under the impression that the different would be far less noticeable.

1 Like
7

I am trying to test the wildcard with date range, to see if it'll make any difference

GET logstash-*/_search
{
  "query": {
    "bool": {
      "must": {
        "query_string": {
          "query": "id=%{id}"
        }
      },
      "filter": {
        "range" : {
            "timestamp" : {
              "gt" : "now-1h",
              "lt" : "now"
            }
        }
      }
    }
  }
}

I paste this into a template.json and changed my elasticsearch to

elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "${input}"
                query_template => "template.json"
}

And this results in an error. reason=>"Something is wrong with your configuration." Not sure how to work around this?

Nevermind. Found the error.

Unknown setting 'query_template' for elasticsearch {:level=>:error}

Version error?

8

Yes, your elasticsearch filter plugin is most likely too old. Try upgrading it, or upgrade Logstash altogether.

9

Ahh... I cant really just upgrade logstash/elasticsearch... do you have any alternative suggestions? Can I somehow squeeze all of the template into query in elasticsearch filter plugin?

10

Can I somehow squeeze all of the template into query in elasticsearch filter plugin?

Sure, just stuff it into a single string like in the examples in the documentation. It won't look nice but it'll work.

11

Thanks for the response Magnus. I have tried following this documentation Elasticsearch filter plugin and squeeze all of the query on one line, but I am obtaining a fetched an invalid config file error

Possibly due to quotes within query => " "bool" : ... "?

I tried doing something like query => " 'bool': ..." and now the resulting error message is
"reason":"Failed to parse query
"reason":{"type":"query_shard_exception","reason":"Failed to parse query
"reason":"Encountered \" <RANGE_GOOP>

My problem is very similar to the issue v0it has

Nvm. Solved my own question. Had to use lucene format.

query => '"id":%{id} AND @timestamp:[now-5h/d TO now/d]'

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.