Logstash pipeline not inserting all data at first time

[vikramdayma (Vikram Singh)]

Hi,

I setup a logstash conf file for data from sql to elasticsearch index.
When I run conf file first time, it insert less data, every time with new index. Inserted data is less than from total data, randomly.
But second time when I run logstash conf file it insert all data.
Only first time whole data is not inserting.

Thanks
Vikram Singh

[venkatesh_prasanth]

hi could you post your conf file

[TheSumitBanik (Sumit Banik)]

Please post your config file to understand the issue better.

[vikramdayma (Vikram Singh)]

Thanks Sumbit for reply.
My config file below.

input {
    jdbc {
           jdbc_connection_string => "jdbc:postgresql://DB_IP_ADDRESS:5444/mydb"
           jdbc_user => "DB_USER"
           jdbc_password => "DB_PASSWORD"
           jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.0.0.jar"
           jdbc_driver_class => "org.postgresql.Driver"
           statement => "SELECT * FROM data_table"
           jdbc_paging_enabled => "true"
           jdbc_page_size => "100"
           jdbc_validation_timeout => "300"
         }
      }




filter {
        mutate {
               }
       }



output {

         stdout { codec => "rubydebug" }

         elasticsearch {
                         hosts => "http://IP_ADDRESS:9200"
                         index => "mydata"
                         user => "xxxxx"
                         password => "yyyyy"
                         
                       }
       }

[vikramdayma (Vikram Singh)]

Hi Venkatesh,
Thanks for reply.
Here is my conf file.

input {
    jdbc {
           jdbc_connection_string => "jdbc:postgresql://DB_IP_ADDRESS:5444/mydb"
           jdbc_user => "DB_USER"
           jdbc_password => "DB_PASSWORD"
           jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.0.0.jar"
           jdbc_driver_class => "org.postgresql.Driver"
           statement => "SELECT * FROM data_table"
           jdbc_paging_enabled => "true"
           jdbc_page_size => "100"
           jdbc_validation_timeout => "300"
         }
      }




filter {
        mutate {
               }
       }



output {

         stdout { codec => "rubydebug" }

         elasticsearch {
                         hosts => "http://IP_ADDRESS:9200"
                         index => "mydata"
                         user => "xxxxx"
                         password => "yyyyy"
                         
                       }
       }