Logstash Elasticsearch output does not add the first document of a collection

Elastic Stack Logstash
1

I'm trying to use Logstash to get data from Mongodb and insert into Elasticsearch.
I'm using Logstash input plugin for Mongodb from here

My logstash.conf is as follow:

input {
    mongodb {
       uri => 'mongodb_host'
       placeholder_db_dir => '/opt/logstash-mongodb/'
       placeholder_db_name => 'logstash-db.db'
       collection => 'myCollection'
       batch_size => 5000
       unpack_mongo_id => true
    }
}

filter{
    mutate { remove_field => "_id" }
}

output {
    elasticsearch { 
         hosts => ["host"]
         index => "myIndex"
         doc_as_upsert => true
         document_id => "%{mongo_id}"
         document_type => "myDocumentType"
    }
    stdout { codec => rubydebug }
}

Any clue?

ps: each document has its own id, collections with only one document is not inserted as well.

Thanks in advance

2

Is the first document of the collection appearing in the stdout plugin output?

3

Yes... the output prints the first document properties correctly, but then it inserts starting from the second document.

tks

4

Hmm. Maybe the elasticsearch output drops the event because ES rejects it. Are there any clues in the Logstash or ES logs?

5

Thanks for replying and I'm sorry I've disappeared for while..

There is no clue about what happens., no error, nothing. As you can see at the example below, it reads all data from the first document and start to add from the second:

[2017-11-29T08:04:30,861][INFO ][logstash.inputs.mongodb  ] init placeholder for 
logstash_since_segmentation: {"_id"=>BSON::ObjectId('59eb49f0ec0019ce0be8854d'), 
"_class"=>"br.com.algartech.cdi.core.data.model.Segmentation", "name"=>"[ENR] - Regra de 
enriquecimento", "description"=>"[ENR] - Regra dos clientes que devem ir para enriquecimento",     "status"=>true, "segmentationType"=>"ENRICHMENT", "mongodbQuery"=>"{\"$and\":[{\"clientRetail\":true},{\"$and\":[{\"older_aging\":{\"$gte\":7}},{\"older_aging\":{\"$lte\":30}}]},{\"$or\":[{\"has_mobile\":false},{\"$and\":[{\"has_mobile\":true},{\"$nor\":[{\"$or\":[{\"telephone.sentDate\":{\"$exists\":false}},{\"telephone.sentLteCpc\":true}]}]}]}]},{\"$and\":[{\"$or\":[{\"claimPaymentDate\":null},{\"claimPaymentAging\":{\"$gt\":7}}]},{\"$or\":[{\"promiseDate\":null},{\"promiseAging\":{\"$gte\":5}}]},{\"$or\":[{\"unknownDebtDate\":null},{\"unknownDebtAging\":{\"$gt\":30}}]}]},{\"$or\":[{\"$and\":[{\"clientRetail\":true},{\"has_mobile\":false},{\"has_homephone\":true},{\"older_aging\":{\"$in\":[8,13,25,30]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":6}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":-1},{\"debt_sum\":{\"$gte\":30}},{\"debt_sum\":{\"$lte\":60}},{\"older_aging\":{\"$in\":[13,18,30]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":-1},{\"debt_sum\":{\"$gt\":60}},{\"telephone.sentLteCpc\":false},{\"older_aging\":{\"$in\":[8,13,18,25]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":-1},{\"debt_sum\":{\"$gt\":60}},{\"telephone.sentLteCpc\":true},{\"older_aging\":{\"$in\":[8,13,18,25]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":{\"$ne\":-1}},{\"VAR1\":{\"$lt\":0.8}},{\"VAR3\":{\"$lte\":35}},{\"telephone.sentLteCpc\":false},{\"older_aging\":{\"$in\":[8,13,18,30]}}]},{\"$and\":[{\"has_mobile\":true},{\"promiseAging\":{\"$ne\":6}},{\"clientRetail\":true},{\"VAR1\":{\"$ne\":-1}},{\"VAR1\":{\"$lt\":0.8}},{\"VAR3\":{\"$lte\":35}},{\"telephone.sentLteCpc\":true},{\"older_aging\":{\"$in\":[8,13,18,30]}}]}]}]}", "noReached"=>false, "wallet"=>{"$ref"=>"wallets", "$id"=>BSON::ObjectId('59a46c9fea8a9e4af1051041')}}
[2017-11-29T08:04:30,863][INFO ][logstash.pipeline        ] Pipeline main started
[2017-11-29T08:04:30,977][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
D, [2017-11-29T08:04:31.229000 #15240] DEBUG -- : MONGODB | QUERY | namespace=cdi.segmentation selector={:_id=>{:$gt=>BSON::ObjectId('59eb49f0ec0019ce0be8854d')}} flags=[:slave_ok] limit=5000 skip=0 project=nil | runtime: 347.9998ms
D, [2017-11-29T08:04:31.725000 #15240] DEBUG -- : MONGODB | COMMAND | namespace=cdi.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system\.|\$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 198.0000ms
D, [2017-11-29T08:04:32.033000 #15240] DEBUG -- : MONGODB | QUERY | namespace=cdi.segmentation selector={:_id=>{:$gt=>BSON::ObjectId('59f8d6c51843b3f9beb53fe7')}} flags=[:slave_ok] limit=5000 skip=0 project=nil | runtime: 190.0001ms
D, [2017-11-29T08:04:32.712000 #15240] DEBUG -- : MONGODB | COMMAND | namespace=cdi.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system\.|\$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 189.9998ms
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.