# Not all logs are forwarded to elasticsearch

**URL:** <https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282>\
**Category:** Logstash\
**Created:** [January 16, 2020, 9:38am UTC](https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282 "2020-01-16T09:38:30Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tennaen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tennaen/32/87293_2.png) [@tennaen](https://discuss.elastic.co/u/tennaen)\
**Post date:** [January 16, 2020, 9:38am UTC](https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282/1 "2020-01-16T09:38:30Z")

</div>

Good morning,  
i have an issue with logstash. I use jdbc input to collect data from postgres and then forward them to elastic. I've noticed that there is a difference between what is forwarded to elastic (i use kibana to display indicies) and what is in database.  
This is how my configuration looks like:  
pipeline.yml

- pipeline.id: dev  
path.config: "/u01/app/logstash/logstash-7.4.2/pipelines-config/focus\_dev.conf"  
pipeline.workers: 1  
pipeline.batch.size: 125  
pipeline.batch.delay: 50  
queue.type: persisted  
queue.page\_capacity: 1mb  
queue.max\_events: 100  
queue.max\_bytes: 16mb

pipeline config file:

```
 input {
    jdbc {
            jdbc_driver_class => "Java::org.postgresql.Driver"
            jdbc_connection_string => "jdbc:postgresql://db_host:db_port/db_name"
            jdbc_user => "focus_dev"
            jdbc_password => "${devpass}"
            #jdbc_default_timezone => "CET"
            #plugin_timezone => "utc"
            statement => "select * from T_AUDIT_LOG_EVENT where id > :sql_last_value"
            use_column_value => true
            tracking_column => id
            tracking_column_type => "numeric"
            #clean_run => true
            schedule => "* * * * *"
            last_run_metadata_path => "/u01/app/logstash/logstash_logs/dev/.logstash_jdbc_last_run"
    }
 }
 filter {
    json {
            source => "value"
            target => "event"
    }
    mutate {
            add_field => {
                    "[@metadata][event_type]" => "%{[event][event_type]}"
            }
    }
    mutate {
            lowercase => ["[@metadata][event_type]"]
    }
    date {
            timezone => "UTC"
            match => ["[event][event_date]", "YYYY-MM-dd HH:mm:ss.SSS", "ISO8601"]
            target => "@timestamp"
    }
 }
 output {
    elasticsearch {
            hosts => "elastic_host:port"
            user => focusaudit
            password => "${focauditpass}"
            index => "dev-focus-audit-log--%{[@metadata][event_type]}"
            #document_type => "audit-log-events"
            document_id => "%{id}"
    }
    stdout {
            codec => "rubydebug"
    }
 }

```

I will appreciate all the help.

Best Regards,  
Norbert

---

<div class="post-metadata">

**Author:** ![tennaen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tennaen/32/87293_2.png) [@tennaen](https://discuss.elastic.co/u/tennaen)\
**Post date:** [January 16, 2020, 9:59am UTC](https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282/2 "2020-01-16T09:59:43Z")

</div>

Additionaly:  
select count(\*) from table where id \> '1721569' (last id in Kibana)  
shows 174 entries  
and:  
cat .logstash\_jdbc\_last\_run  
--- 1721741

---

<div class="post-metadata">

**Author:** ![tennaen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tennaen/32/87293_2.png) [@tennaen](https://discuss.elastic.co/u/tennaen)\
**Post date:** [January 17, 2020, 6:40am UTC](https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282/3 "2020-01-17T06:40:28Z")

</div>

I know now why i cannot see all logs in kibana. I just need to set filter +1h from now. Anyone can help out with proper TZ settings? I'm in UTC +1.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2020, 2:47pm UTC](https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282/4 "2020-01-17T14:47:57Z")

</div>

> [@tennaen](#):
>
> timezone =\> "UTC"

The timezone option should be set to the timezone in which the logfile timestamp are. A date filter always converts timestamps to UTC.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2020, 2:48pm UTC](https://discuss.elastic.co/t/not-all-logs-are-forwarded-to-elasticsearch/215282/5 "2020-02-14T14:48:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.