# Use of variables in a query in logstash **URL:** \ **Category:** Logstash\ **Created:** [May 21, 2020, 9:16am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706 "2020-05-21T09:16:57Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 21, 2020, 9:16am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/1 "2020-05-21T09:16:58Z")
Hi, I'm using ELK 7.4.0. I want to use dynamic variables in logstash configuration file. Plan A: There is an input plugin called elasticsearch for querying elasticsearch, can I use input from a file and then put its output as input to the elasticsearch input plugin? How? OR Plan B: Can I use input from file and then in filter query elasticsearch? The only problem in this plan is how do I use a date range along with term filter in that query? Please help me execute either of those plans or suggest a new one thanks! ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [May 21, 2020, 11:36am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/2 "2020-05-21T11:36:08Z")
> [@Shubhangi](#): > > Can I use input from file and then in filter query elasticsearch? The only problem in this plan is how do I use a date range along with term filter in that query? This would work well. The elasticsearch filter has a [query](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-query) parameter which lets you define query string and [range](https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-query-string-query.html#_ranges) filtering is supported in the query strings. The other option is to instead use the [query\_template](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-query_template) parameter which allows full fuse of Elasticsearch's [query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html). ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [May 21, 2020, 12:06pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/3 "2020-05-21T12:06:10Z")
plan A won’t work. you use elasticsearch input plugin to search for events in elasticsearch instance. if you want to read from a file then use file input plugin and elasticsearch filter as in plan B. ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 21, 2020, 12:15pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/4 "2020-05-21T12:15:14Z")
Hi Can you show me an example executing a query with a query template? This document from Elastic shows them independently [https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-query](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-query) ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 22, 2020, 1:29pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/6 "2020-05-22T13:29:12Z")
Hi all, I tried plan B and elasticsearch filter does not quiet work for me. I'm thinking of a plan C: What if I use file input along with Elasticsearch query input and then in filter i drop the Elasticsearch query results that don't quite match my file input? Now here is my problem: I can't seem to access and compare fields from both inputs simultaneously. Here is my conf: ```auto input { file{ path => "/path/to/input" start_position => "beginning" type => "file_input" } elasticsearch{ hosts => ["localhost:9200"] index => "index*" query => '{ "query": { "range": { "@timestamp": { "gt": "now-1d/d", "lt": "now" } } }}' type => "es" } #stdin{} } filter{ if [type] == "file_input"{ grok{ match => ["message","(?.*)"] } } if [type] == "es" { if [Clientip] != [ip] { drop{} } } } output { csv { fields => ["@timestamp","server", "Solution"] path => "/path/to/output.csv" } #stdout{} } ``` I am not able to drop logs because I can't access fields from two different inputs. Can someone help, please! Thanks! ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [May 23, 2020, 7:19pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/7 "2020-05-23T19:19:19Z")
> [@Shubhangi](#): > > What if I use file input along with Elasticsearch query input and then in filter i drop the Elasticsearch query results that don't quite match my file input? Plan C is not going to work because two inputs can never share events with each other. You have to use one input plugin and then access its events in the filter plugin. What is not working in the Elasticsearch filter when you try plan B? ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 24, 2020, 2:58pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/8 "2020-05-24T14:58:44Z")
No matter what input plugin I use, or even if I don't use an input plugin, I don't get any output for elasticsearch filter. That's why I was asking about it's compatibility with an input plugin. Here's my conf: ```auto input{} filter{ elasticsearch{ hosts => ["localhost:9200"] index => ["index*"] query => "ip:1.2.3.4" } } output { stdout{} } ``` For the above conf, I get following output: ```auto [logstash.agent] Successfully started Logstash API endpoint {:port=>9601} [logstash.runner] Logstash shut down. ``` If I use an stdin{} plugin I get: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/2055814bf6eaad8b7b308ab3fbd975d82268f540.png) And i get desired output for: ```auto GET index*/_search?q=ip:1.2.3.4 ``` The output for above query is company sensitive, so please don't ask for it. Once I can use elasticsearch filter, I think I can achieve my objective. Please help me out. Thanks! ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [May 24, 2020, 6:46pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/9 "2020-05-24T18:46:41Z")
Please try this: 1. The `file` goes in the input filter. Update the `path` to whatever is valid. 2. Since I do not know what the messages in your `file` look like, I used a basic grok pattern to pull out the `ip` field, so update this grok pattern to whatever is applicable otherwise you will get `grokparsefailure` and your Elasticsearch filter in step below will not return results again. 3. Reference that `ip` field available in step 2 in the query of the Elaticsearch filter. If you change the name of the field from ip to something else, please change the reference too. Update the `host` if your Elasticsearch is not running on localhost. 4. See what it outputs in `stdout` before using the elasticsearch output plugin . ```auto input { file{ path => "/path/to/input" start_position => "beginning" } } filter{ grok{ match => { "message" => "(?%{IP})%{GREEDYDATA}" } } elasticsearch { hosts => ["localhost:9200"] index => ["index*"] query => "ip:%{ip}" } } output { stdout{} } ``` ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 24, 2020, 7:00pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/10 "2020-05-24T19:00:02Z")
Um the value(1.2.3.4) is in the file, not the field(ip). I'm filtering the field on the value in file. So, I believe the query should be something like: ```auto query=> "ip:%{ip}" ``` You see > [@Shubhangi](#): > > ```auto > filter{ > elasticsearch{ > hosts => ["localhost:9200"] > index => ["index*"] > query => "ip:1.2.3.4" > } > > ``` here I've tried to filter it on a value. I think if it works with a static value then I'll definitely work with dynamic value(with the value extracted from file). Thanks for bearing with me. ---
**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Post date:** [May 24, 2020, 7:02pm UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/11 "2020-05-24T19:02:31Z")
> [@Shubhangi](#): > > `query=> "ip:%{ip}"` Good catch. Yes, the ip filter should be how you set it. I corrected it in my response. ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 25, 2020, 6:53am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/12 "2020-05-25T06:53:26Z")
@Rahul_Kumar4 Your way also does not work. Here's my config: ```auto input{ stdin{} } filter{ grok{ match => { "message" => "(?%{IP})%{GREEDYDATA}" } } elasticsearch { hosts => ["localhost:9200"] index => ["index*"] query => "xforwarded1:%{ip}" } } output { stdout{} } ``` Here's my output: ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/8/b8c2a7c134ae2b9e17e110822849491efed12620.png) And I'd get same output if my config didn't have elasticsearch filter. I don't think I'm supposed to expect this output from elasticsearch filter. Conclusion: elasticsearch filter does not work, not for me. If anyone has got a working elasticsearch filter conf then please help me out. Thanks. ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 25, 2020, 7:50am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/13 "2020-05-25T07:50:27Z")
I'm getting some output, still not right. Here's my config: ```auto input{ stdin{} } filter{ grok{ match => { "message" => "(?%{IP})%{GREEDYDATA}" } } elasticsearch { hosts => ["localhost:9200"] index => ["index*"] query => "xforwarded1:%{ip} AND @timestamp:[now-1d/d TO now]" fields => { "message" => "log" } } } output { stdout{} } ``` field was the catch! But I can only see one hit, while my console shows 16 hits. Any remedy? Thanks ---
**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Post date:** [May 25, 2020, 8:20am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/14 "2020-05-25T08:20:57Z")
Here's the solution: ```auto input{ stdin{} } filter{ grok{ match => { "message" => "(?%{IP})%{GREEDYDATA}" } } elasticsearch { hosts => ["localhost:9200"] index => ["index*"] result_size => 100 query => "xforwarded1:%{ip} AND @timestamp:[now-1d/d TO now]" fields => { "message" => "log" } } split { field => "log" } } output { stdout{} } ``` Use `result_size` and `split` to make the output complete and presentable. Target achieved. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2020, 8:21am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706/15 "2020-06-22T08:21:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.