Logstash query template


(Bao Thai) #1

Change from original question

{
  "query": {
    "bool": {
      "must": {
        "query_string": {
          "query": "id=%{id}"
        }
      },
      "filter": {
        "range" : {
            "timestamp" : {
              "gt" : "now-1h",
              "lt" : "now"
            }
        }
      }
    }
  }
}

I paste this into a template.json file and changed my elasticsearch to

elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "${input}"
                query_template => "template.json"
}

And this results in an error. reason=>"Something is wrong with your configuration."
Not sure how to work around this? I looked at samples on elastic.co and looks as simple as this. I tried copying the whole path, but did not solve the problem.


(Magnus Bäck) #2

Why not just use a date range filter in your query? I believe reasonably recent versions of ES are quite efficient in skipping indexes that obviously don't contain any documents in range (i.e. using logstash-* isn't significantly slower)


(Bao Thai) #3

Wasn't sure if applying a date range would have applied to looking for the correct index. Am I suppose to write it within parameters filter of elasticsearch? I came up with a solution by parsing the timestamp and aggregate it to my index query name instead of using a 2-days date range. Not sure about the solution tho, do you have any opinion/suggestion?


(Magnus Bäck) #4

Wasn’t sure if applying a date range would have applied to looking for the correct index.

You don't need to figure out which indexes to query. Just use a wildcard.

Am I suppose to write it within parameters filter of elasticsearch?

What do you mean by "parameters filter"?


(Bao Thai) #5

If I wanted to apply a date range in the query, how would I use it within elasticsearch plug in? This ElasticSearch Parameters Plugin

Say give or take you have 90 indexes, would wildcard vs specific index contribute to speed? I thought it would since took's average value is less than that of a wildcard search


(Magnus Bäck) #6

If I wanted to apply a date range in the query, how would I use it within elasticsearch plug in? This ElasticSearch Parameters Plugin

That's the elasticsearch output plugin, but you seem to be asking about the elasticsearch input or filter plugin.

Say give or take you have 90 indexes, would wildcard vs specific index contribute to speed? I thought it would since took's average value is less than that of a wildcard search

Then I'm surprised. I was under the impression that the different would be far less noticeable.


(Bao Thai) #7

I am trying to test the wildcard with date range, to see if it'll make any difference

GET logstash-*/_search
{
  "query": {
    "bool": {
      "must": {
        "query_string": {
          "query": "id=%{id}"
        }
      },
      "filter": {
        "range" : {
            "timestamp" : {
              "gt" : "now-1h",
              "lt" : "now"
            }
        }
      }
    }
  }
}

I paste this into a template.json and changed my elasticsearch to

elasticsearch {
                hosts => [ "localhost:9200" ]
                index => "${input}"
                query_template => "template.json"
}

And this results in an error. reason=>"Something is wrong with your configuration." Not sure how to work around this?

Nevermind. Found the error.

Unknown setting 'query_template' for elasticsearch {:level=>:error}

Version error?


(Magnus Bäck) #8

Yes, your elasticsearch filter plugin is most likely too old. Try upgrading it, or upgrade Logstash altogether.


(Bao Thai) #9

Ahh... I cant really just upgrade logstash/elasticsearch... do you have any alternative suggestions? Can I somehow squeeze all of the template into query in elasticsearch filter plugin?


(Magnus Bäck) #10

Can I somehow squeeze all of the template into query in elasticsearch filter plugin?

Sure, just stuff it into a single string like in the examples in the documentation. It won't look nice but it'll work.


(Bao Thai) #11

Thanks for the response Magnus. I have tried following this documentation Elasticsearch filter plugin and squeeze all of the query on one line, but I am obtaining a fetched an invalid config file error

Possibly due to quotes within query => " "bool" : ... "?

I tried doing something like query => " 'bool': ..." and now the resulting error message is
"reason":"Failed to parse query
"reason":{"type":"query_shard_exception","reason":"Failed to parse query
"reason":"Encountered \" <RANGE_GOOP>

My problem is very similar to the issue v0it has

Nvm. Solved my own question. Had to use lucene format.

query => '"id":%{id} AND @timestamp:[now-5h/d TO now/d]'


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.