# Filter and Output plugin not working

**URL:** <https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941>\
**Category:** Logstash\
**Created:** [May 29, 2020, 2:57pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941 "2020-05-29T14:57:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Xophis](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@Xophis](https://discuss.elastic.co/u/Xophis)\
**Post date:** [May 29, 2020, 2:57pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941/1 "2020-05-29T14:57:07Z")

</div>

Hi,

I need some help regarding JDBC input plugin.

The documentation says : \* You can periodically schedule ingestion using a cron syntax (see schedule setting) or run the query one time to load data into Logstash\*

I would like to run the query only once to make a huge insert in ES.  
But in the below configuration, if i remove the schedule configuration, filter and output plugin don't work.. If i let the schedule configuration, it will work correctly but insert several times the same jdbc rows.

Here is the configuration :

```auto
input{
  jdbc{
    jdbc_driver_library => 'postgresql-42.2.12.jar'
    jdbc_driver_class => 'org.postgresql.Driver'
    jdbc_connection_string => 'jdbc:postgresql://host:port/instanceName' 
    jdbc_user => 'Username'
    jdbc_password => 'PasswordMegaSecured'
    schedule => "* * * * *"
    statement => 'select * from "Schema"."dblogs"'
  }
}
filter{
  aggregate{
    task_id => "%{id}"
    code => "
      map['id'] = event.get('id')
      map['source'] = event.get('source')
      map['target'] = event.get('target')
      map['logs'] ||= []
      map['logs'] << {
        'stacktrace' => event.get('stacktrace')
      }
    "
    push_map_as_event_on_timeout => true
    timeout => 10
    timeout_code => "
      event.set('[@aggregated]', 1)
    "
  }
  if [@aggregated] != 1 { drop {} }
  mutate{
    remove_field => ['@timestamp','@version','@aggregated']
  }
}
output{
  elasticsearch {
    hosts => ["192.168.119.136:9200"]
    index => "myindex"
  }
}

```

Can you help me ?  
Thanks you.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 29, 2020, 4:11pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941/2 "2020-05-29T16:11:06Z")

</div>

> [@Xophis](#):
>
> if i remove the schedule configuration, filter and output plugin don't work

In what way do they not work?

---

<div class="post-metadata">

**Author:** ![Xophis](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@Xophis](https://discuss.elastic.co/u/Xophis)\
**Post date:** [May 29, 2020, 4:58pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941/3 "2020-05-29T16:58:42Z")

</div>

Hi Badger,

Thanks for your reply.

I don't have any output in ES. I tried to output with stdout and after the sql request, i got logstash shutdown.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 29, 2020, 7:15pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941/4 "2020-05-29T19:15:05Z")

</div>

Try removing the timeout\_code and instead adding

```
event.cancel()

```

to the end of the code option.

---

<div class="post-metadata">

**Author:** ![Xophis](https://avatars.discourse-cdn.com/v4/letter/x/65b543/32.png) [@Xophis](https://discuss.elastic.co/u/Xophis)\
**Post date:** [June 2, 2020, 1:27pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941/5 "2020-06-02T13:27:55Z")

</div>

Hi Badger,

I updated my configuration and still the same result..

Here is the conf :

```auto
filter{
  aggregate{
    task_id => "%{id}"
    code => "
      map['id'] = event.get('id')
      map['source'] = event.get('source')
      map['target'] = event.get('target')
      map['logs'] ||= []
      map['logs'] << {
        'stacktrace' => event.get('stacktrace')
      }
      event.cancel()
    "
    push_map_as_event_on_timeout => true
    timeout => 10
# timeout_code => "
# event.set('[@aggregated]', 1)
# "
  }
# if [@aggregated] != 1 { drop {} }
# mutate{
# remove_field => ['@timestamp','@version','@aggregated']
# }
}
output{
  elasticsearch {
    hosts => ["192.168.119.136:9200"]
    index => "nested3"
  }
}

```

Thanks for helping !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2020, 1:29pm UTC](https://discuss.elastic.co/t/filter-and-output-plugin-not-working/234941/6 "2020-06-30T13:29:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.