# Logstash is not getting whole table data

**URL:** <https://discuss.elastic.co/t/logstash-is-not-getting-whole-table-data/341945>\
**Category:** Logstash\
**Created:** [August 30, 2023, 7:24am UTC](https://discuss.elastic.co/t/logstash-is-not-getting-whole-table-data/341945 "2023-08-30T07:24:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![uma\_parvathy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/uma_parvathy/32/123696_2.png) [@uma\_parvathy](https://discuss.elastic.co/u/uma_parvathy)\
**Post date:** [August 30, 2023, 7:24am UTC](https://discuss.elastic.co/t/logstash-is-not-getting-whole-table-data/341945/1 "2023-08-30T07:24:49Z")

</div>

Hi All,

i've been working on Elasticsearch recently. the logstash pipeline pulls the data only 10k records from the db table.

How shall i make it to pull whole table data ?

i tried both jdbc\_page\_size and jdbc\_fetch\_size , nothing works.

```auto
input {
    jdbc {
        jdbc_driver_library => "/usr/local/Cellar/logstash/8.9.0/libexec/logstash-core/lib/jars/postgresql-jdbc.jar"
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/pl_itsm_stg"
        jdbc_user => "postgres"
        jdbc_password => "root"
        jdbc_driver_class => "org.postgresql.Driver"
        tracking_column => "incident_number"
        schedule => "0 * * * *" # cronjob schedule format (see "Helpful Links")
        statement => "SELECT incident_number, site_id, sub_site_id, account_id, sub_account_id, closed_at, state, short_description, description, incident_parent_id from customerdata_incident"
        jdbc_fetch_size => 100000
    }
}

filter {
	if [incident_parent_id] {
  		jdbc_streaming {
        		jdbc_driver_library => "/usr/local/Cellar/logstash/8.9.0/libexec/logstash-core/lib/jars/postgresql-jdbc.jar"
        		jdbc_connection_string => "jdbc:postgresql://localhost:5432/pl_itsm_stg"
        		jdbc_user => "postgres"
        		jdbc_password => "root"
        		jdbc_driver_class => "org.postgresql.Driver"
    			statement => "select incident_number, company, sub_site_id, site_id, account_id, sub_account_id, issue_type, incident_state, resolved_at, resolved_by, impact, state, urgency from customerdata_incident WHERE incident_number = :parent_id"
    			parameters => { "parent_id" => "incident_parent_id"}
    			target => "incident_parent_id"
  		}
 
        }
        ruby {
			code => '
    					incident_parent = event.get("incident_parent_id")
    					if incident_parent.is_a? Array
						event.set("parent_id", incident_parent[0])
                                        end
                                '

        }	
        if [parent_id] {
		mutate {
                        rename => {"parent_id" => "incident_parent_id"}
                }
        }
	mutate {
		copy => {"id" => "[@metadata][_id]"}
                
	}
}
output {
	stdout { codec => "json" }
	elasticsearch {
		hosts => ["https://localhost:9200"]
                ssl => true
                ssl_certificate_verification => false
                cacert => "/Users/umaparvathykaliappan/ca_logstash.cer"
		user => "elastic"
		password => "+JYA1pqFMlbl+ZjH9WaK"
		index => "logstash_itsm_incidents_parent"
                ilm_enabled => true
	}
}

```

1. My db table has 26K record. how shall i get all 26K.
2. if have to paginate my sql query to get all records, how shall i achieve it.

---

<div class="post-metadata">

**Author:** ![uma\_parvathy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/uma_parvathy/32/123696_2.png) [@uma\_parvathy](https://discuss.elastic.co/u/uma_parvathy)\
**Post date:** [September 4, 2023, 5:11am UTC](https://discuss.elastic.co/t/logstash-is-not-getting-whole-table-data/341945/2 "2023-09-04T05:11:40Z")

</div>

if i use django-elasticsearch-dsl search\_build feature, it uses Elasticsearch builk api and pulled all 26K records.  
Only when i use logstash, I can see only 10K records as count . What is that missing here in my above configuration .

Please guide me .

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![uma\_parvathy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/uma_parvathy/32/123696_2.png) [@uma\_parvathy](https://discuss.elastic.co/u/uma_parvathy)\
**Post date:** [September 4, 2023, 10:28am UTC](https://discuss.elastic.co/t/logstash-is-not-getting-whole-table-data/341945/3 "2023-09-04T10:28:42Z")

</div>

my bad.

GET /\_search  
shows count as 10,000

but when i queried with track\_total\_hits: true, i got the whole table count.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 2, 2023, 10:28am UTC](https://discuss.elastic.co/t/logstash-is-not-getting-whole-table-data/341945/4 "2023-10-02T10:28:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.