# Elasticsearch filter plugin does not work as expected

**URL:** <https://discuss.elastic.co/t/elasticsearch-filter-plugin-does-not-work-as-expected/174108>\
**Category:** Logstash\
**Created:** [March 27, 2019, 11:34am UTC](https://discuss.elastic.co/t/elasticsearch-filter-plugin-does-not-work-as-expected/174108 "2019-03-27T11:34:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![takeno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/takeno/32/28524_2.png) [@takeno](https://discuss.elastic.co/u/takeno)\
**Post date:** [March 27, 2019, 11:34am UTC](https://discuss.elastic.co/t/elasticsearch-filter-plugin-does-not-work-as-expected/174108/1 "2019-03-27T11:34:44Z")

</div>

I'm trying to calculate job duration time using elasticsearch filter.  
The problem is that I can not get job.duration when I input the whole json.

 ![elk_err](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9f3413305beecca92b6c4d7897751eb768735b6f.png)

logstash conf:

```
input {
  stdin { codec => "json" }
}

filter {
          date {
            match => ["[job][log_date]" , "yyyy/MM/dd HH:mm:ss.SSS"]
            target => "[job][log_date]"
          }

      if [job][message_id] == "JC00000002" {
        elasticsearch {
          hosts => ["srv03:9200"]
          query => 'tags:startjobnet AND job.jobnet_id:"%{[job][jobnet_id]}"'
          index => "tmp-*"
          enable_sort => true
          sort => "job.log_date:desc"
          fields => { "[job][log_date]" => "started" }
        }

        date {
          match => ["[started]", "ISO8601"]
          target => "[started]"
        }

        if [started] {
          ruby {
            code => "event.set('[job][duration]', (event.get('[job][log_date]') - event.get('started')) )"
          }
        }
      }

      if [job][message_id] == "JC00000001" {
        mutate { add_tag => ["startjobnet"] }
      }
}

output {
    elasticsearch {
      hosts => ["srv03:9200"]
      index => "tmp-%{+yyyy.MM.dd}"
    }
}

```

json input:

```auto
{ "job": { "log_date": "2019/03/27 17:06:03.108", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000001", "message": "start jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } },
{ "job": { "log_date": "2019/03/27 17:06:04.008", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000002", "message": "end jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } }
{ "job": { "log_date": "2019/03/27 17:10:02.216", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000001", "message": "start jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } },
{ "job": { "log_date": "2019/03/27 17:10:03.901", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000002", "message": "end jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } }
{ "job": { "log_date": "2019/03/27 17:14:02.216", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000001", "message": "start jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } },
{ "job": { "log_date": "2019/03/27 17:16:03.901", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000002", "message": "end jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } }
{ "job": { "log_date": "2019/03/27 17:26:03.108", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000001", "message": "start jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } },
{ "job": { "log_date": "2019/03/27 17:26:04.008", "inner_jobnet_main_id": 922556, "inner_jobnet_id": 922556, "run_type": 0, "public_flag": 0, "jobnet_id": "hoge", "job_id": "", "message_id": "JC00000002", "message": "end jobnet", "jobnet_name": "hoge-mon", "job_name": "", "user_name": "user1", "update_date": "2019/03/12 14:15:14", "return_code": "" } }

```

But I cat get job.duration by inputting json **one by one** manually.  
 ![elk_ok](https://us1.discourse-cdn.com/elastic/original/3X/0/8/086599ba1573abadc275f5760ac9a2d2b70ba61c.png)

It seems logstash filter pass to Output after the whole event completely processed in Filter area.  
Is there any way to query between events that occur at almost same time by elasticsearch filter plugin?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2019, 11:34am UTC](https://discuss.elastic.co/t/elasticsearch-filter-plugin-does-not-work-as-expected/174108/2 "2019-04-24T11:34:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.