# Not getting all Records imported from Logstash to Elasticsearch

**URL:** <https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697>\
**Category:** Logstash\
**Created:** [January 8, 2016, 7:41am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697 "2016-01-08T07:41:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![trushad](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@trushad](https://discuss.elastic.co/u/trushad)\
**Post date:** [January 8, 2016, 7:41am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697/1 "2016-01-08T07:41:22Z")

</div>

My config file is as below:

> input {  
> jdbc {  
> jdbc\_driver\_library =\> "D:\Microsoft JDBC Driver 6.0 for SQL Server\sqljdbc\_6.0\enu\sqljdbc42.jar"  
> jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"  
> jdbc\_connection\_string =\> "jdbc:sqlserver://V-STAGING\SQL2012;databaseName=CartrackDetailsInfoForAzure;user=sa;password=database@1;autoReconnect=true;"  
> jdbc\_user =\> "sa"  
> jdbc\_password =\> "database@1"  
> statement =\> "select top 10 \* from dbo.CartrackDetailsInfo"  
> jdbc\_paging\_enabled =\> "true"  
> jdbc\_page\_size =\> "200000"  
> }  
> }

> output {  
> elasticsearch {  
> hosts =\> "localhost:9200"  
> index =\> "newjdbc"  
> document\_type =\> "jdbccon"  
> document\_id =\> "%{iCartrackDetailsID}"
> 
> }  
> stdout { codec =\> rubydebug }  
> }

Here I'm importing 10 Records to Elasticsearch, but When hitting GET count request at Elasticsearch their I get to see only 1 Record.

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [January 8, 2016, 1:46pm UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697/2 "2016-01-08T13:46:21Z")

</div>

Could you paste the output from running Logstash?

---

<div class="post-metadata">

**Author:** ![trushad](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@trushad](https://discuss.elastic.co/u/trushad)\
**Post date:** [January 11, 2016, 5:27am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697/3 "2016-01-11T05:27:55Z")

</div>

> D:\logstash-2.1.0\bin\>logstash -f log.conf  
> io/console not supported; tty will not be manipulated  
> Settings: Default filter workers: 1  
> Logstash startup completed  
> {  
> "icartrackdetailsid\_1" =\> 178790318,  
> "cout\_driver" =\> nil,  
> "cout\_event\_description" =\> "Kynoch Rd, Durban City, Durban, KwaZulu-Natal, South Africa",  
> "cout\_event\_odo" =\> 578676,  
> "dout\_event\_ts" =\> "2014-06-07T18:37:29.000Z",  
> "cout\_ignition" =\> "ON",  
> "fout\_latitude" =\> -30.0178,  
> "fout\_longitude" =\> 30.906525,  
> "cout\_registration" =\> "ND343265",  
> "fout\_speed" =\> 9.0,  
> "cout\_vehicleid" =\> nil,  
> "dtimestamp" =\> "2014-06-07T18:37:29.000Z",  
> "acqaccountnumber" =\> "030676:043801",  
> "wdate" =\> #\<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)\>,  
> "cdate" =\> #\<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)\>,  
> "ccountry" =\> nil,  
> "cprovience" =\> nil,  
> "caddress" =\> nil,  
> "bgps" =\> true,  
> "synctimestamp" =\> "2014-06-07T22:14:12.403Z",  
> "icartrackdetailsid" =\> 1,  
> "@version" =\> "1",  
> "@timestamp" =\> "2016-01-11T05:19:55.335Z"  
> }  
> {  
> "icartrackdetailsid\_1" =\> 178790319,  
> "cout\_driver" =\> nil,  
> "cout\_event\_description" =\> "Lyndhurst Rd, John Dube Village, East London, Eastern Cape, South Africa",  
> "cout\_event\_odo" =\> 367210,  
> "dout\_event\_ts" =\> "2014-06-07T18:37:29.000Z",  
> "cout\_ignition" =\> "ON",  
> "fout\_latitude" =\> -33.00025,  
> "fout\_longitude" =\> 27.841779,  
> "cout\_registration" =\> "TEMP-CT959623",  
> "fout\_speed" =\> 19.0,  
> "cout\_vehicleid" =\> nil,  
> "dtimestamp" =\> "2014-06-07T18:37:29.000Z",  
> "acqaccountnumber" =\> "029295:033937",  
> "wdate" =\> #\<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)\>,  
> "cdate" =\> #\<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)\>,  
> "ccountry" =\> nil,  
> "cprovience" =\> nil,  
> "caddress" =\> nil,  
> "bgps" =\> true,  
> "synctimestamp" =\> "2014-06-07T22:14:12.403Z",  
> "icartrackdetailsid" =\> 2,  
> "@version" =\> "1",  
> "@timestamp" =\> "2016-01-11T05:19:55.348Z"  
> }  
> {  
> "icartrackdetailsid\_1" =\> 178790327,  
> "cout\_driver" =\> nil,  
> "cout\_event\_description" =\> "1st St, Salvokop, Pretoria, Gauteng, South Africa",  
> "cout\_event\_odo" =\> 159668,  
> "dout\_event\_ts" =\> "2014-06-07T18:37:28.000Z",  
> "cout\_ignition" =\> "ON",  
> "fout\_latitude" =\> -25.76023,  
> "fout\_longitude" =\> 28.184337,  
> "cout\_registration" =\> "TEMP-CT843590",  
> "fout\_speed" =\> 0.0,  
> "cout\_vehicleid" =\> nil,  
> "dtimestamp" =\> "2014-06-07T18:37:28.000Z",  
> "acqaccountnumber" =\> "041962:048292",  
> "wdate" =\> #\<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)\>,  
> "cdate" =\> #\<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)\>,  
> "ccountry" =\> nil,  
> "cprovience" =\> nil,  
> "caddress" =\> nil,  
> "bgps" =\> true,  
> "synctimestamp" =\> "2014-06-07T22:14:12.403Z",  
> "icartrackdetailsid" =\> 10,  
> "@version" =\> "1",  
> "@timestamp" =\> "2016-01-11T05:19:55.446Z"  
> }  
> Logstash shutdown completed

In Logstash running window I m able to see all the records. but In Elasticsearch only last record( "icartrackdetailsid\_1" =\> 178790327,) is seen

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 11, 2016, 8:12am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697/4 "2016-01-11T08:12:18Z")

</div>

> [@trushad](#):
>
> "%{iCartrackDetailsID}"

You are specifying the field to be used as ID here in the configuration, but from the output you provided it looks like the field name is all lowercase:

> [@trushad](#):
>
> "icartrackdetailsid" =\> 1,

This means that all records most likely get the exact string you provided and therefore the same ID. Make sure that the field names match and try again.

---

<div class="post-metadata">

**Author:** ![trushad](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@trushad](https://discuss.elastic.co/u/trushad)\
**Post date:** [January 11, 2016, 9:20am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697/5 "2016-01-11T09:20:40Z")

</div>

Thank you! It works!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:16am UTC](https://discuss.elastic.co/t/not-getting-all-records-imported-from-logstash-to-elasticsearch/38697/6 "2017-07-06T05:16:09Z")

</div>