I want to use file input along with Elasticsearch query input and then in filter I want to drop the Elasticsearch query results that don't quite match my file input.
Now here is my problem:
I can't seem to access and compare fields from both inputs simultaneously.
There are neither any errors nor an output.
Logstash stops at:
[INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9601}
That is correct. The events from the two inputs are processed independently. You cannot reference fields from an event produced by the file input when processing an event produced by the elasticsearch input.
can both inputs combined to a common output pipeline using the collector pattern in pipeline-to-pipeline? if so, will fields from each inputs accessible to a filter in the common output pipeline ?
Thanks for the reply!
How do you suggest I accomplish my task:
I've to use a make a dynamic elasticsearch query filter in logstash, which filters on values of a field i provide, hence the file input and then give out an output file.
Example:
query => "ip:<dynamic_value> AND @timestamp:[now-1d/d TO now]"
Thanks for replying @Badger
I tried using elasticsearch filter, I didn't get any output. I'll try once again and post about it.
But is there a pre requisite to use elasticsearch input plugin along with elasticsearch filter plugin? I mean does it filter the output from input plugin?
The Logstash document of elasticsearch filter plugin does not show the input part of it at all. It'd be great if you could shed some light on it.
No matter what input plugin I use, or even if I don't use an input plugin, I don't get any output for elasticsearch filter. That's why I was asking about it's compatibility with an input plugin.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.