Logstash Reading Same Data

Elastic Stack Logstash
1

Hi,

I have a logstash .conf:

input {
    jdbc {
        jdbc_driver_library => "${LOGSTASH_JDBC_DRIVER_JAR_LOCATION}"
        jdbc_driver_class => "${LOGSTASH_JDBC_DRIVER}"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        schedule => "* * * * *"
        statement => "select * from testtable"
    }
}

filter {
  mutate {
  	add_field => { "message" => "%{time}" }
          convert => [ "time", "string" ]
  }
  date {
      timezone => "Etc/GMT+3"
      match => ["time" , "ISO8601", "yyyy-MM-dd HH:mm:ss.SSS"]
      target => "@timestamp"
      remove_field => [ "time", "timestamp" ]
  }
  fingerprint {
    source => ["testid", "programid", "unitid"]
    target => "[@metadata][fingerprint]"
    method => "MD5"
    key => "${LOGSTASH_JDBC_PASSWORD}"
  }
  ruby {
    code => "event.set('[@metadata][tsprefix]', event.get('@timestamp').to_i.to_s(16))"
  }
}

output {
    elasticsearch {
        hosts => ["${LOGSTASH_ELASTICSEARCH_HOST}"]
        user => "${ELASTIC_USER}"
        password => "${ELASTIC_PASSWORD}"
        index => "heartbeat"
        document_id => "%{[@metadata][tsprefix]}%{[@metadata][fingerprint]}"
    }
    stdout { codec => json_lines }
}

And Logstash keeps reading same data over and over again, Index Count does not increase in Elasticsearch but size of it does. (Database: Postgresql 14.5, JDBC Driver: 42.4.1)
Can you help me understand the reason?

2

Also tried:

use_column_value => true
tracking_column => "time"

But did not resolve my problem.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.