I can't send MongoDB data to Elasticseach via logstash

Elastic Stack Logstash
1

My configuration file is as follows:

    input {
  mongodb {
        uri => 'mongodb://localhost:27017/exemplo'
        placeholder_db_dir => 'C:/Users/Administrator/Documents/Elastic/logstash-7.6.1/data/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        #database =>'br'        
        #unpack_mongo_id => true
        collection => 'movie'
        batch_size => 5000

  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "aminhaindexcaraisxsxsx"
  }
}

The following appears in the terminal, but does not create the index, much less receive the data

D, [2020-05-29T12:37:57.903000 #3572] DEBUG -- : MONGODB | [141] localhost:27017 #1 | exemplo.find | STARTED | {"find"=>"movie", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5ed11391474a8677a9bf3882')}}, "limit"=>5000, "$db"=>"exemplo", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x9f9103ab597f4342...>}}
D, [2020-05-29T12:37:57.907000 #3572] DEBUG -- : MONGODB | [141] localhost:27017 | exemplo.find | SUCCEEDED | 0.002s
D, [2020-05-29T12:37:58.025000 #3572] DEBUG -- : MONGODB | [142] localhost:27017 #1 | exemplo.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"exemplo", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0x9f9103ab597f4342...>}}
D, [2020-05-29T12:37:58.029000 #3572] DEBUG -- : MONGODB | [142] localhost:27017 | exemplo.listCollections | SUCCEEDED | 0.002s
2

Can someone help me?

3

Do you see any logs in Logstash that show any error as it tries to index your mongodb data in Elasticsearch?

It appears that your host variable may not be set correctly. See the examples here.

You can also set up a stdout in your output plugin to see if you get any data from Mongodb.

4

Thank you very much for answering me.

I changed my file to the following:

input {

  mongodb {

        uri => 'mongodb://localhost:27017/exemplo'

        placeholder_db_dir => 'C:/Users/Administrator/Documents/Elastic/logstash-7.6.1/data/opt/logstash-mongodb/'

        placeholder_db_name => 'logstash_sqlite.db'

        #database =>'br'        

        #unpack_mongo_id => true

        collection => 'movie'

        batch_size => 5000

  }

}

output {

  elasticsearch {

    hosts => ["http://localhost:9200"]

    index => "aminhaindexcaraisxsxsx"

  }

  stdout { codec => json }

}

Logstash displays the following response, without an error:

D, [2020-06-01T07:33:35.814000 #276] DEBUG -- : MONGODB | [202] localhost:27017 #1 | exemplo.find | STARTED | {"find"=>"movie", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5ed11391474a8677a9bf3882')}}, "limit"=>5000, "$db"=>"exemplo", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0xc8d0b628481c4ec2...>}}
D, [2020-06-01T07:33:35.838000 #276] DEBUG -- : MONGODB | [202] localhost:27017 | exemplo.find | SUCCEEDED | 0.015s
D, [2020-06-01T07:33:35.947000 #276] DEBUG -- : MONGODB | [203] localhost:27017 #1 | exemplo.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"exemplo", "lsid"=>{"id"=><BSON::Binary:0x2064 type=uuid data=0xc8d0b628481c4ec2...>}}
D, [2020-06-01T07:33:35.955000 #276] DEBUG -- : MONGODB | [203] localhost:27017 | exemplo.listCollections | SUCCEEDED | 0.003s

But he doesn't create the index when I check Kibana, in the logstash response he even takes the ID of an object from the DB document.

Could you help me?

5

Those seem to be logs from your Mongodb. What do you see in the logstash logs?

6

In the logs it returns like this:

[2020-06-01T08:00:54,265][INFO ][logstash.inputs.mongodb  ][main] Registering MongoDB input
[2020-06-01T08:00:57,828][INFO ][logstash.inputs.mongodb  ][main] init placeholder for logstash_since_movie: {"_id"=>BSON::ObjectId('5ed11391474a8677a9bf3882'), "name"=>"tutorials point"}
[2020-06-01T08:00:57,849][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-06-01T08:00:58,357][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-01T08:00:59,014][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.