Getting this error while ingesting data from Mongodb into Elasticsearch

Elastic Stack Logstash
1

Hii.I am trying to ingest data from Mongodb into Elasticsearch using logstash pipeline.I am using "logstash-input-mongodb" plugin to ingest data from Mongodb into Elasticsearch.This is my configuration pipeline.

input {

    mongodb {

        uri => 'mongodb://localhost:27017/mydata'

        placeholder_db_dir => '/root/logstash-mongodb'

        placeholder_db_name => 'logstash_sqlite.db'

        collection => 'mylappy'

        batch_size => 50

    }

}

filter {

    mutate {

        remove_field => ["_id"]

    }

}

output {

    stdout { 

        codec => rubydebug 

    }

    elasticsearch {

        hosts => ["http://localhost:9200"]

        user => "elastic"

        password => "password"

        index => "mongodocs"

    }

}

I am getting this Error after running Logstash Pipeline:-

D, [2021-07-26T20:54:19.438557 #3607] DEBUG -- : MONGODB | QUERY | namespace=mydata.mylappy selector={:_id=>{:$gt=>BSON::ObjectId('600599437bd4eecfee0db02a')}} flags=[:slave_ok] limit=50 skip=0 project=nil | runtime: 2.2900ms
D, [2021-07-26T20:54:19.450360 #3607] DEBUG -- : MONGODB | COMMAND | namespace=mydata.$cmd selector={:listCollections=>1, :cursor=>{}, :filter=>{:name=>{"$not"=>/system.|$/}}} flags=[:slave_ok] limit=-1 skip=0 project=nil | runtime: 2.5320ms

D, [2021-07-26T20:54:22.511500 #3607] DEBUG -- : MONGODB | COMMAND | namespace=admin.$cmd selector={:ismaster=>1} flags= limit=-1 skip=0 project=nil | runtime: 2.8701ms

2

Where is the error? Those are just DEBUG messages.

3

@Badger Thanks for the reply.I am trying to put documents in elasticsearch from mongodb,but out of 3 documents only 1 document is getting stored in elasticsearch.Rest are not.

4

If there are three documents in the MongoDB collection then you should only expect two of them to get indexed. When the code initializes the placeholder that it uses to track progress it sorts the documents in the collection and saves the oldest value (of _id by default). When it runs a query to fetch data it fetches anything greater than that value, so the first document is never indexed.

Showing all of the DEBUG and INFO messages from the input might tell us more.

5

Hii @Badger .Thanks Alot for explaining this in detail.Is there any way that we can retrieve all the documents from Mongodb?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.