Logstash run pipeline but not send data to output

Elastic Stack Elasticsearch
1

I'm trying to use logstash Jdbc plugin to synchronize data from a Postgres database to some ouput (elastic, rabbitMq).

The problem is that logstash is running well every hour (I configured the cronJob like that).
Logstash is successfully reading the database but for some reason doesn't send the messages right away.
However, after a few hours, messages are successfully sent.

I first thought it was a perfs issue because of a high volume of messages but even with just one message the behavior is the same.

I tested locally on docker with docker-compose, I don't have the issue but in a Kubernetes pod it doesn't work.

pipeline.conf

input {
      jdbc {
        jdbc_driver_class => "${JDBC_DRIVER_CLASS}"
        jdbc_connection_string => "${JDBC_CONNECTION_STRING}"
        jdbc_user => "${JDBC_USER}"
        jdbc_password => "${JDBC_PASSWORD}"
        jdbc_paging_enabled => false
        codec => "json"
        tracking_column => "sync_unix_ts"
        use_column_value => true
        tracking_column_type => "numeric"
        schedule => "${LOGSTASH_CRONEXPRESSION}"
        statement_filepath => "/usr/share/logstash/query/elasticsearch-query.sql"
      }
    }
    filter {
      json {
        source => "fieldjson"
      }
      mutate {  
        remove_field => ["fieldjson","sync_unix_ts"]
      }
    }
    
    output {
      elasticsearch {
          index => "index_name"
          document_id => "%{id}"
          hosts => ["${ELASTICSEARCH_HOST}"]
      }
    }

logstash.yaml

http.host: "0.0.0.0"
xpack.monitoring.enabled: false

I don't know if it is a buffer issue or not...

Thanks for your answers

2

Where are you seeing the messages? In Kibana? After how many hours do you see them? It could be related to time zone configuration.

Can you run your pipeline using the stdout output, and see if the messages appears?

3

thanks for your answer.

I see the messages directly on 'elastic' with request on index (ex. 'kibana').
this is my cron expression : 0 * * * *.

For the tests situations, i modify my cron expression to '*/15 * * * *' (At every 15th minute) and using stdout but messages not appears.

In logstash log, he run every 15th minute and execute a query in input and get 10 messages but not send.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.