# Insert old MongoDB data for ES

**URL:** <https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626>\
**Category:** Logstash\
**Created:** [June 3, 2020, 7:34pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626 "2020-06-03T19:34:56Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 3, 2020, 7:34pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/1 "2020-06-03T19:34:57Z")

</div>

My logstash configuration for inserting data in Elasticsearch inserts only new data, it does not insert the old DB documents.

My .conf file is as follows:

```
input {

  mongodb {

        uri => 'mongodb://localhost:27017/empresa'

        placeholder_db_dir => 'C:/Users/Administrator/Documents/Elastic/versao7.7/logstash-7.7.0/data/opt/logstash-mongo'

        placeholder_db_name => 'novo.db'

        #database =>'br'        

        #unpack_mongo_id => true

        generateId => 'false'

        collection => 'novo'

        isodate => 'true'

        batch_size => 5000

  }

}

filter {

mutate {

remove_field => ["_id" ,"@timestamp"]

}

}

output {

  elasticsearch {

    hosts => ["http://localhost:9200"]

    index => "tentandoindex1"

  }

  stdout { codec => json }

}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2020, 7:56pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/2 "2020-06-03T19:56:57Z")

</div>

I suggest you start over with a new placeholder\_db, run logstash, then use sqlite to dump the placeholder\_db. See if the value of \_id in the table makes sense.

---

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 3, 2020, 8:12pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/3 "2020-06-03T20:12:39Z")

</div>

Thanks for answering me! I made the change but he just inserted a document, the file change was like this:  
input {

```
  mongodb {

        uri => 'mongodb://localhost:27017/empresa'

        placeholder_db_dir => 'C:/Users/Administrator/Documents/Elastic/versao7.7/logstash-7.7.0/data/opt/logstash-mongo'

        placeholder_db_name => 'logstash_sqlite.db'

        #database =>'br'        

        #unpack_mongo_id => true

        generateId => 'false'

        collection => 'novo'

        isodate => 'true'

        batch_size => 5000

  }

}

filter {

mutate {

remove_field => ["_id" ,"@timestamp"]

}

}

output {

  elasticsearch {

    hosts => ["http://localhost:9200"]

    index => "tentandoindex2"

  }

  stdout { codec => json }

}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 3, 2020, 8:57pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/4 "2020-06-03T20:57:53Z")

</div>

Using sqlite, dump the contents of logstash\_sqlite.db

See if the value of \_id makes sense.

---

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 4, 2020, 2:16pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/5 "2020-06-04T14:16:46Z")

</div>

Thanks for answering me, but I still haven't managed to insert the old data from ES, any command that I should be forgetting?

---

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 4, 2020, 2:37pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/6 "2020-06-04T14:37:23Z")

</div>

As an example this line in sql,  
use column value =\> false, do you know if there would be something like that, any parameters for full and incremental?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 4, 2020, 2:56pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/7 "2020-06-04T14:56:21Z")

</div>

Enable log.level debug. Read the log messages about first entry and the placeholder db. You need to figure out what value of \_id it is starting with and whether that value makes sense.

---

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 4, 2020, 3:18pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/8 "2020-06-04T15:18:46Z")

</div>

How do I do that?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 4, 2020, 3:45pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/9 "2020-06-04T15:45:57Z")

</div>

Either edit log.level in logstash.yml or add '--log.level debug' to the command line.

---

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 4, 2020, 7:28pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/10 "2020-06-04T19:28:01Z")

</div>

It shows me the id of an id object only and does not include the old data, when neither inserted nor the index is created

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 4, 2020, 7:54pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/11 "2020-06-04T19:54:05Z")

</div>

Right, but when you first connect to the db you should get these messages

```
"init placeholder for #{since_table}_#{mongo_collection_name}"
"init placeholder for #{since_table}_#{mongo_collection_name}: #{first_entry}"
"FIRST ENTRY ID for #{mongo_collection_name} is #{first_entry_id}"

```

which tell you what \_id it is starting with. On subsequent passes you should get

```
placeholder already exists, it is #{x[:place]}

```

What \_id does it log, and what are the \_id values on the older data you say it is not reading?

---

<div class="post-metadata">

**Author:** ![FernandoMOTA](https://avatars.discourse-cdn.com/v4/letter/f/c67d28/32.png) [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\
**Post date:** [June 4, 2020, 8:03pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/12 "2020-06-04T20:03:32Z")

</div>

My DEBUG return:

```
D, [2020-06-04T13:02:44.142000 #5860] DEBUG -- : MONGODB | [3985] localhost:27017 #1 | super_loja.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"super_loja", "lsid"=>{"id"=><BSON::Binary:0x2604 type=uuid data=0x6e1fb66148e34eab...>}}
D, [2020-06-04T13:02:44.165000 #5860] DEBUG -- : MONGODB | [3985] localhost:27017 | super_loja.listCollections | SUCCEEDED | 0.019s
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 4, 2020, 8:26pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/13 "2020-06-04T20:26:03Z")

</div>

Look at the logstash logs, not the mongo logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 2, 2020, 8:26pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626/14 "2020-07-02T20:26:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.