Insert old MongoDB data for ES

Elastic Stack Logstash
1

My logstash configuration for inserting data in Elasticsearch inserts only new data, it does not insert the old DB documents.

My .conf file is as follows:

input {

  mongodb {

        uri => 'mongodb://localhost:27017/empresa'

        placeholder_db_dir => 'C:/Users/Administrator/Documents/Elastic/versao7.7/logstash-7.7.0/data/opt/logstash-mongo'

        placeholder_db_name => 'novo.db'

        #database =>'br'        

        #unpack_mongo_id => true

        generateId => 'false'

        collection => 'novo'

        isodate => 'true'

        batch_size => 5000

  }

}

filter {

mutate {

remove_field => [ "_id" ,"@timestamp"]

}

}

output {

  elasticsearch {

    hosts => ["http://localhost:9200"]

    index => "tentandoindex1"

  }

  stdout { codec => json }

}
2

I suggest you start over with a new placeholder_db, run logstash, then use sqlite to dump the placeholder_db. See if the value of _id in the table makes sense.

3

Thanks for answering me! I made the change but he just inserted a document, the file change was like this:
input {

  mongodb {

        uri => 'mongodb://localhost:27017/empresa'

        placeholder_db_dir => 'C:/Users/Administrator/Documents/Elastic/versao7.7/logstash-7.7.0/data/opt/logstash-mongo'

        placeholder_db_name => 'logstash_sqlite.db'

        #database =>'br'        

        #unpack_mongo_id => true

        generateId => 'false'

        collection => 'novo'

        isodate => 'true'

        batch_size => 5000

  }

}

filter {

mutate {

remove_field => [ "_id" ,"@timestamp"]

}

}

output {

  elasticsearch {

    hosts => ["http://localhost:9200"]

    index => "tentandoindex2"

  }

  stdout { codec => json }

}
4

Using sqlite, dump the contents of logstash_sqlite.db

See if the value of _id makes sense.

5

Thanks for answering me, but I still haven't managed to insert the old data from ES, any command that I should be forgetting?

6

As an example this line in sql,
use column value => false, do you know if there would be something like that, any parameters for full and incremental?

7

Enable log.level debug. Read the log messages about first entry and the placeholder db. You need to figure out what value of _id it is starting with and whether that value makes sense.

1 Like
8

How do I do that?

9

Either edit log.level in logstash.yml or add '--log.level debug' to the command line.

10

It shows me the id of an id object only and does not include the old data, when neither inserted nor the index is created

11

Right, but when you first connect to the db you should get these messages

"init placeholder for #{since_table}_#{mongo_collection_name}"
"init placeholder for #{since_table}_#{mongo_collection_name}: #{first_entry}"
"FIRST ENTRY ID for #{mongo_collection_name} is #{first_entry_id}"

which tell you what _id it is starting with. On subsequent passes you should get

placeholder already exists, it is #{x[:place]}

What _id does it log, and what are the _id values on the older data you say it is not reading?

12

My DEBUG return:

D, [2020-06-04T13:02:44.142000 #5860] DEBUG -- : MONGODB | [3985] localhost:27017 #1 | super_loja.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"super_loja", "lsid"=>{"id"=><BSON::Binary:0x2604 type=uuid data=0x6e1fb66148e34eab...>}}
D, [2020-06-04T13:02:44.165000 #5860] DEBUG -- : MONGODB | [3985] localhost:27017 | super_loja.listCollections | SUCCEEDED | 0.019s
13

Look at the logstash logs, not the mongo logs.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.