# Logstash pipeline not inserting all data at first time

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347>\
**Category:** Logstash\
**Created:** [June 18, 2021, 7:51am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347 "2021-06-18T07:51:13Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vikramdayma](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@vikramdayma](https://discuss.elastic.co/u/vikramdayma)\
**Post date:** [June 18, 2021, 7:51am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347/1 "2021-06-18T07:51:13Z")

</div>

Hi,

I setup a logstash conf file for data from sql to elasticsearch index.  
When I run conf file first time, it insert less data, every time with new index. Inserted data is less than from total data, randomly.  
But second time when I run logstash conf file it insert all data.  
Only first time whole data is not inserting.

Thanks  
Vikram Singh

---

<div class="post-metadata">

**Author:** ![venkatesh\_prasanth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkatesh_prasanth/32/89319_2.png) [@venkatesh\_prasanth](https://discuss.elastic.co/u/venkatesh_prasanth)\
**Post date:** [June 18, 2021, 8:41am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347/2 "2021-06-18T08:41:42Z")

</div>

hi could you post your conf file

---

<div class="post-metadata">

**Author:** ![TheSumitBanik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thesumitbanik/32/89748_2.png) [@TheSumitBanik](https://discuss.elastic.co/u/TheSumitBanik)\
**Post date:** [June 19, 2021, 8:28am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347/3 "2021-06-19T08:28:37Z")

</div>

Please post your config file to understand the issue better.

---

<div class="post-metadata">

**Author:** ![vikramdayma](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@vikramdayma](https://discuss.elastic.co/u/vikramdayma)\
**Post date:** [June 21, 2021, 4:27am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347/4 "2021-06-21T04:27:06Z")

</div>

Thanks Sumbit for reply.  
My config file below.

```auto
input {
    jdbc {
           jdbc_connection_string => "jdbc:postgresql://DB_IP_ADDRESS:5444/mydb"
           jdbc_user => "DB_USER"
           jdbc_password => "DB_PASSWORD"
           jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.0.0.jar"
           jdbc_driver_class => "org.postgresql.Driver"
           statement => "SELECT * FROM data_table"
           jdbc_paging_enabled => "true"
           jdbc_page_size => "100"
           jdbc_validation_timeout => "300"
         }
      }

filter {
        mutate {
               }
       }

output {

         stdout { codec => "rubydebug" }

         elasticsearch {
                         hosts => "http://IP_ADDRESS:9200"
                         index => "mydata"
                         user => "xxxxx"
                         password => "yyyyy"
                         
                       }
       }

```

---

<div class="post-metadata">

**Author:** ![vikramdayma](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@vikramdayma](https://discuss.elastic.co/u/vikramdayma)\
**Post date:** [June 21, 2021, 4:28am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347/5 "2021-06-21T04:28:27Z")

</div>

Hi Venkatesh,  
Thanks for reply.  
Here is my conf file.

```auto
input {
    jdbc {
           jdbc_connection_string => "jdbc:postgresql://DB_IP_ADDRESS:5444/mydb"
           jdbc_user => "DB_USER"
           jdbc_password => "DB_PASSWORD"
           jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-42.0.0.jar"
           jdbc_driver_class => "org.postgresql.Driver"
           statement => "SELECT * FROM data_table"
           jdbc_paging_enabled => "true"
           jdbc_page_size => "100"
           jdbc_validation_timeout => "300"
         }
      }

filter {
        mutate {
               }
       }

output {

         stdout { codec => "rubydebug" }

         elasticsearch {
                         hosts => "http://IP_ADDRESS:9200"
                         index => "mydata"
                         user => "xxxxx"
                         password => "yyyyy"
                         
                       }
       }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 19, 2021, 4:28am UTC](https://discuss.elastic.co/t/logstash-pipeline-not-inserting-all-data-at-first-time/276347/6 "2021-07-19T04:28:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.