I'm using ELK 7.4.0.
I want to use dynamic variables in logstash configuration file.
Plan A: There is an input plugin called elasticsearch for querying elasticsearch, can I use input from a file and then put its output as input to the elasticsearch input plugin? How?
Plan B: Can I use input from file and then in filter query elasticsearch? The only problem in this plan is how do I use a date range along with term filter in that query?
Please help me execute either of those plans or suggest a new one thanks!
I tried plan B and elasticsearch filter does not quiet work for me.
I'm thinking of a plan C:
What if I use file input along with Elasticsearch query input and then in filter i drop the Elasticsearch query results that don't quite match my file input?
Now here is my problem:
I can't seem to access and compare fields from both inputs simultaneously.
Here is my conf:
Plan C is not going to work because two inputs can never share events with each other. You have to use one input plugin and then access its events in the filter plugin. What is not working in the Elasticsearch filter when you try plan B?
The file goes in the input filter. Update the path to whatever is valid.
Since I do not know what the messages in your file look like, I used a basic grok pattern to pull out the ip field, so update this grok pattern to whatever is applicable otherwise you will get grokparsefailure and your Elasticsearch filter in step below will not return results again.
Reference that ip field available in step 2 in the query of the Elaticsearch filter. If you change the name of the field from ip to something else, please change the reference too. Update the host if your Elasticsearch is not running on localhost.
See what it outputs in stdout before using the elasticsearch output plugin .