# Accessing/comparing fields from two different input plugins in filter, simultaneously

**URL:** <https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917>\
**Category:** Logstash\
**Created:** [May 22, 2020, 2:39pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917 "2020-05-22T14:39:37Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [May 22, 2020, 2:39pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/1 "2020-05-22T14:39:37Z")

</div>

I want to use file input along with Elasticsearch query input and then in filter I want to drop the Elasticsearch query results that don't quite match my file input.

Now here is my problem:  
I can't seem to access and compare fields from both inputs simultaneously.

There are neither any errors nor an output.

Logstash stops at:

```auto
[INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"}
[logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO][filewatch.observingtail][main] START, creating Discoverer, Watch with file and sincedb collections
[INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9601}

```

Here is my conf:

```auto
input {
 file{
 path => "/path/to/input" 
 start_position => "beginning"
 type => "file_input"
}
elasticsearch{
	 hosts => ["localhost:9200"]
	index => "index*"
	query => '{ "query": {

                "range": {
                    "@timestamp": {
                        "gt": "now-1d/d",
                        "lt": "now"
                    }
                }
              
 
  }}'
	type => "es"
}
#stdin{}
}
filter{

	if [type] == "file_input"{
		grok{
			match => ["message","(?<ip>.*)"]
		}
		
	}
if [type] == "es" {
			if [Clientip] != [ip] {
				drop{}
			}
		}
		
}
output {
  csv {
    fields => ["@timestamp","server", "Solution"]
    
path => "/path/to/output.csv"
  }
stdout{}
}

```

My input file has an ip like this:

```auto
"1.2.3.4"

```

Can someone help, please! Thanks!

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 22, 2020, 2:45pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/2 "2020-05-22T14:45:16Z")

</div>

do both inputs produce outputs? what kind of outputs are produced by each output, and which fields do you want to compare ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 22, 2020, 3:20pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/3 "2020-05-22T15:20:36Z")

</div>

> [@Shubhangi](#):
>
> I can't seem to access and compare fields from both inputs simultaneously.

That is correct. The events from the two inputs are processed independently. You cannot reference fields from an event produced by the file input when processing an event produced by the elasticsearch input.

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 22, 2020, 3:42pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/4 "2020-05-22T15:42:00Z")

</div>

can both inputs combined to a common output pipeline using the collector pattern in pipeline-to-pipeline? if so, will fields from each inputs accessible to a filter in the common output pipeline ?

---

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [May 22, 2020, 4:22pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/5 "2020-05-22T16:22:35Z")

</div>

Thanks for the reply!  
How do you suggest I accomplish my task:  
I've to use a make a dynamic elasticsearch query filter in logstash, which filters on values of a field i provide, hence the file input and then give out an output file.

Example:

```auto
query => "ip:<dynamic_value> AND @timestamp:[now-1d/d TO now]"

```

Please help.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 22, 2020, 5:24pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/6 "2020-05-22T17:24:32Z")

</div>

Using an elasticsearch filter sounds like a promising approach, but I do not run elasticsearch, so I cannot advise you on configuring it.

---

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [May 22, 2020, 5:34pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/7 "2020-05-22T17:34:34Z")

</div>

Thanks for replying @Badger  
I tried using elasticsearch filter, I didn't get any output. I'll try once again and post about it.

But is there a pre requisite to use elasticsearch input plugin along with elasticsearch filter plugin? I mean does it filter the output from input plugin?

The Logstash document of elasticsearch filter plugin does not show the input part of it at all. It'd be great if you could shed some light on it.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 22, 2020, 7:27pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/8 "2020-05-22T19:27:55Z")

</div>

> [@Shubhangi](#):
>
> But is there a pre requisite to use elasticsearch input plugin along with elasticsearch filter plugin?

No, they are independent.

---

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [May 24, 2020, 2:56pm UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/9 "2020-05-24T14:56:32Z")

</div>

No matter what input plugin I use, or even if I don't use an input plugin, I don't get any output for elasticsearch filter. That's why I was asking about it's compatibility with an input plugin.

Here's my conf:

```auto
input{}
filter{
	elasticsearch{
		hosts => ["localhost:9200"]
	    index => ["index*"]
	    query => "ip:1.2.3.4"
}
		
}
output {
 
stdout{}
}

```

For the above conf, I get following output:

```auto
[logstash.agent] Successfully started Logstash API endpoint {:port=>9601}
[logstash.runner] Logstash shut down.

```

If I use an stdin{} plugin I get:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/2055814bf6eaad8b7b308ab3fbd975d82268f540.png)

And i get desired output for:

```auto
GET index*/_search?q=ip:1.2.3.4

```

The output for above query is company sensitive, so please don't ask for it.  
Please help me out. Thanks!

---

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [May 25, 2020, 7:51am UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/10 "2020-05-25T07:51:02Z")

</div>

@Badger  
I'm getting some output, still not right.  
Here's my config:

```auto
input{
 stdin{}
}
filter{
	grok{
        match => {
            "message" => "(?<ip>%{IP})%{GREEDYDATA}"
        }
    }
    elasticsearch {
        hosts => ["localhost:9200"]
        index => ["index*"]
        query => "xforwarded1:%{ip} AND @timestamp:[now-1d/d TO now]"
	fields => {
  "message" => "log"
}
    }
		
}
output {
  stdout{}
}

```

field was the catch!

But I can only see one hit, while my console shows 16 hits. Any remedy?

Thanks

---

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [May 25, 2020, 8:21am UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/11 "2020-05-25T08:21:37Z")

</div>

Here's the solution:

```auto
input{
 stdin{}
}
filter{
	grok{
        match => {
            "message" => "(?<ip>%{IP})%{GREEDYDATA}"
        }
    }
    elasticsearch {
        hosts => ["localhost:9200"]
        index => ["index*"]
	result_size => 100
        query => "xforwarded1:%{ip} AND @timestamp:[now-1d/d TO now]"
	fields => {
  		"message" => "log"
	}
	
    }
	split {
   field => "log"
 }	
}
output {
   stdout{}
}

```

Use `result_size` and `split` to make the output complete and presentable. Target achieved. Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2020, 8:21am UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917/12 "2020-06-22T08:21:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.