# Logstash Elasticsearch output does not add the first document of a collection **URL:** \ **Category:** Logstash\ **Created:** [November 7, 2017, 11:30am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675 "2017-11-07T11:30:41Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![MariSpirandelli](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@MariSpirandelli](https://discuss.elastic.co/u/MariSpirandelli)\ **Post date:** [November 7, 2017, 11:30am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675/1 "2017-11-07T11:30:41Z")
I'm trying to use Logstash to get data from Mongodb and insert into Elasticsearch. I'm using Logstash input plugin for Mongodb from [here](https://github.com/phutchins/logstash-input-mongodb) My logstash.conf is as follow: ``` input { mongodb { uri => 'mongodb_host' placeholder_db_dir => '/opt/logstash-mongodb/' placeholder_db_name => 'logstash-db.db' collection => 'myCollection' batch_size => 5000 unpack_mongo_id => true } } filter{ mutate { remove_field => "_id" } } output { elasticsearch { hosts => ["host"] index => "myIndex" doc_as_upsert => true document_id => "%{mongo_id}" document_type => "myDocumentType" } stdout { codec => rubydebug } } ``` Any clue? ps: each document has its own id, collections with only one document is not inserted as well. Thanks in advance ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 9, 2017, 10:00pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675/2 "2017-11-09T22:00:53Z")
Is the first document of the collection appearing in the stdout plugin output? ---
**Author:** ![MariSpirandelli](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@MariSpirandelli](https://discuss.elastic.co/u/MariSpirandelli)\ **Post date:** [November 10, 2017, 6:06pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675/3 "2017-11-10T18:06:26Z")
Yes... the output prints the first document properties correctly, but then it inserts starting from the second document. tks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 13, 2017, 6:24am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675/4 "2017-11-13T06:24:12Z")
Hmm. Maybe the elasticsearch output drops the event because ES rejects it. Are there any clues in the Logstash or ES logs? ---
**Author:** ![MariSpirandelli](https://avatars.discourse-cdn.com/v4/letter/m/b9bd4f/32.png) [@MariSpirandelli](https://discuss.elastic.co/u/MariSpirandelli)\ **Post date:** [November 29, 2017, 10:11am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675/5 "2017-11-29T10:11:04Z")
Thanks for replying and I'm sorry I've disappeared for while.. There is no clue about what happens., no error, nothing. As you can see at the example below, it reads all data from the first document and start to add from the second: ``` [2017-11-29T08:04:30,861][INFO][logstash.inputs.mongodb] init placeholder for logstash_since_segmentation: {"_id"=>BSON::ObjectId('59eb49f0ec0019ce0be8854d'), "_class"=>"br.com.algartech.cdi.core.data.model.Segmentation", "name"=>"[ENR] - Regra de enriquecimento", "description"=>"[ENR] - Regra dos clientes que devem ir para enriquecimento", "status"=>true, "segmentationType"=>"ENRICHMENT", "mongodbQuery"=>"{\"$and\":[{\"clientRetail\":true},{\"$and\":[{\"older_aging\":{\"$gte\":7}},{\"older_aging\":{\"$lte\":30}}]},{\"$or\":[{\"has_mobile\":false},{\"$and\":[{\"has_mobile\":true},{\"$nor\":[{\"$or\":[{\"telephone.sentDate\":{\"$exists\":false}},{\"telephone.sentLteCpc\":true}]}]}]}]},{\"$and\":[{\"$or\":[{\"claimPaymentDate\":null},{\"claimPaymentAging\":{\"$gt\":7}}]},{\"$or\":[{\"promiseDate\":null},{\"promiseAging\":{\"$gte\":5}}]},{\"$or\":[{\"unknownDebtDate\":null},{\"unknownDebtAging\":{\"$gt\":30}}]}]},{\"$or\":[{\"$and\":[{\"clientRetail\":true},{\"has_mobile\":false},{\"has_homephone\":true},{\"older_aging\":{\"$in\":[8,13,25,30]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":6}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":-1},{\"debt_sum\":{\"$gte\":30}},{\"debt_sum\":{\"$lte\":60}},{\"older_aging\":{\"$in\":[13,18,30]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":-1},{\"debt_sum\":{\"$gt\":60}},{\"telephone.sentLteCpc\":false},{\"older_aging\":{\"$in\":[8,13,18,25]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":-1},{\"debt_sum\":{\"$gt\":60}},{\"telephone.sentLteCpc\":true},{\"older_aging\":{\"$in\":[8,13,18,25]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":{\"$ne\":-1}},{\"VAR1\":{\"$lt\":0.8}},{\"VAR3\":{\"$lte\":35}},{\"telephone.sentLteCpc\":false},{\"older_aging\":{\"$in\":[8,13,18,30]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":{\"$ne\":-1}},{\"VAR1\":{\"$lt\":0.8}},{\"VAR3\":{\"$lte\":35}},{\"telephone.sentLteCpc\":true},{\"older_aging\":{\"$in\":[8,13,18,30]}}]}]}]}", "noReached"=>false, "wallet"=>{"$ref"=>"wallets", "$id"=>BSON::ObjectId('59a46c9fea8a9e4af1051041')}} [2017-11-29T08:04:30,863][INFO][logstash.pipeline] Pipeline main started [2017-11-29T08:04:30,977][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} D, [2017-11-29T08:04:31.229000 #15240] DEBUG -- : MONGODB | QUERY | namespace=cdi.segmentation selector={:_id=>{:$gt=>BSON::ObjectId('59eb49f0ec0019ce0be8854d')}} flags=[:slave_ok] limit=5000 skip=0 project=nil | runtime: 347.9998ms D, [2017-11-29T08:04:31.725000 #15240] DEBUG -- : MONGODB | COMMAND | namespace=cdi.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system\.|\$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 198.0000ms D, [2017-11-29T08:04:32.033000 #15240] DEBUG -- : MONGODB | QUERY | namespace=cdi.segmentation selector={:_id=>{:$gt=>BSON::ObjectId('59f8d6c51843b3f9beb53fe7')}} flags=[:slave_ok] limit=5000 skip=0 project=nil | runtime: 190.0001ms D, [2017-11-29T08:04:32.712000 #15240] DEBUG -- : MONGODB | COMMAND | namespace=cdi.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system\.|\$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 189.9998ms ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 27, 2017, 10:11am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-does-not-add-the-first-document-of-a-collection/106675/6 "2017-12-27T10:11:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.