Not getting all Records imported from Logstash to Elasticsearch

Elastic Stack Logstash
1

My config file is as below:

input {
jdbc {
jdbc_driver_library => "D:\Microsoft JDBC Driver 6.0 for SQL Server\sqljdbc_6.0\enu\sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://V-STAGING\SQL2012;databaseName=CartrackDetailsInfoForAzure;user=sa;password=database@1;autoReconnect=true;"
jdbc_user => "sa"
jdbc_password => "database@1"
statement => "select top 10 * from dbo.CartrackDetailsInfo"
jdbc_paging_enabled => "true"
jdbc_page_size => "200000"
}
}

output {
elasticsearch {
hosts => "localhost:9200"
index => "newjdbc"
document_type => "jdbccon"
document_id => "%{iCartrackDetailsID}"

}
stdout { codec => rubydebug }
}

Here I'm importing 10 Records to Elasticsearch, but When hitting GET count request at Elasticsearch their I get to see only 1 Record.

2

Could you paste the output from running Logstash?

3

D:\logstash-2.1.0\bin>logstash -f log.conf
io/console not supported; tty will not be manipulated
Settings: Default filter workers: 1
Logstash startup completed
{
"icartrackdetailsid_1" => 178790318,
"cout_driver" => nil,
"cout_event_description" => "Kynoch Rd, Durban City, Durban, KwaZulu-Natal, South Africa",
"cout_event_odo" => 578676,
"dout_event_ts" => "2014-06-07T18:37:29.000Z",
"cout_ignition" => "ON",
"fout_latitude" => -30.0178,
"fout_longitude" => 30.906525,
"cout_registration" => "ND343265",
"fout_speed" => 9.0,
"cout_vehicleid" => nil,
"dtimestamp" => "2014-06-07T18:37:29.000Z",
"acqaccountnumber" => "030676:043801",
"wdate" => #<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)>,
"cdate" => #<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)>,
"ccountry" => nil,
"cprovience" => nil,
"caddress" => nil,
"bgps" => true,
"synctimestamp" => "2014-06-07T22:14:12.403Z",
"icartrackdetailsid" => 1,
"@version" => "1",
"@timestamp" => "2016-01-11T05:19:55.335Z"
}
{
"icartrackdetailsid_1" => 178790319,
"cout_driver" => nil,
"cout_event_description" => "Lyndhurst Rd, John Dube Village, East London, Eastern Cape, South Africa",
"cout_event_odo" => 367210,
"dout_event_ts" => "2014-06-07T18:37:29.000Z",
"cout_ignition" => "ON",
"fout_latitude" => -33.00025,
"fout_longitude" => 27.841779,
"cout_registration" => "TEMP-CT959623",
"fout_speed" => 19.0,
"cout_vehicleid" => nil,
"dtimestamp" => "2014-06-07T18:37:29.000Z",
"acqaccountnumber" => "029295:033937",
"wdate" => #<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)>,
"cdate" => #<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)>,
"ccountry" => nil,
"cprovience" => nil,
"caddress" => nil,
"bgps" => true,
"synctimestamp" => "2014-06-07T22:14:12.403Z",
"icartrackdetailsid" => 2,
"@version" => "1",
"@timestamp" => "2016-01-11T05:19:55.348Z"
}
{
"icartrackdetailsid_1" => 178790327,
"cout_driver" => nil,
"cout_event_description" => "1st St, Salvokop, Pretoria, Gauteng, South Africa",
"cout_event_odo" => 159668,
"dout_event_ts" => "2014-06-07T18:37:28.000Z",
"cout_ignition" => "ON",
"fout_latitude" => -25.76023,
"fout_longitude" => 28.184337,
"cout_registration" => "TEMP-CT843590",
"fout_speed" => 0.0,
"cout_vehicleid" => nil,
"dtimestamp" => "2014-06-07T18:37:28.000Z",
"acqaccountnumber" => "041962:048292",
"wdate" => #<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)>,
"cdate" => #<Date: 2014-06-08 ((2456817j,0s,0n),+0s,2299161j)>,
"ccountry" => nil,
"cprovience" => nil,
"caddress" => nil,
"bgps" => true,
"synctimestamp" => "2014-06-07T22:14:12.403Z",
"icartrackdetailsid" => 10,
"@version" => "1",
"@timestamp" => "2016-01-11T05:19:55.446Z"
}
Logstash shutdown completed

In Logstash running window I m able to see all the records. but In Elasticsearch only last record( "icartrackdetailsid_1" => 178790327,) is seen

4

You are specifying the field to be used as ID here in the configuration, but from the output you provided it looks like the field name is all lowercase:

This means that all records most likely get the exact string you provided and therefore the same ID. Make sure that the field names match and try again.

5

Thank you! It works!