How to Ingest data from MongoDB database into ElasticSearch using Logstash

Elastic Stack Logstash
1

Hello Logstash/ES Gurus,

I am new to ES and logstash, I need help on ingesting a user collection data from a mongo database into elastic search.

I have used logstash-input-mongodb plugin and started the logstash and it failed with following error though the error out line in config file looks good.

C:>logstash-6.7.0\bin\logstash.bat -f logstash-6.7.0\config\logstash-mongo.conf
Sending Logstash logs to C:/logstash-6.7.0/logs which is now configured via log4j2.properties
[2020-01-23T18:14:06,538][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-01-23T18:14:06,557][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.0"}
[2020-01-23T18:14:07,479][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 2, column 7 (byte 16) after input {\r\n uri ", :backtrace=>["C:/logstash-6.7.0/logstash-core/lib/logstash/compiler.rb:4

Below is how my config file looks
input {
uri => 'mongodb://:@<mongo_host>:/?replicaSet=vncdev?ssl=true'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'testcases'
batch_size => 5000
}
filter {

}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "mongo_vncdev_vncmgr"
hosts => ["http://localhost:9200"]
}
}

Any help in fixing is highly appreciated

Thanks
Kishore

2

Try

input {
    mongodb {
        uri => 'mongodb://:@<mongo_host>:/?replicaSet=vncdev?ssl=true'
        placeholder_db_dir => '/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'testcases'
        batch_size => 5000
    }
}
3

Hi Badger,

Thanks for the input, I am able to bypass the error after placing mongodb keyword after input as you mentioned.

I encountered additional error due to non-existance path /opt/logstash-mongodb.
My mongodb is on a RHEL 7 VM, what path should I mention for placeholder_db_dir and placeholder_db_name

[2020-01-24T12:45:41,971][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::MongoDB placeholder_db_name=>"logstash_sqlite.db", collection=>"testcases", batch_size=>5000, id=>"acf2b106554f69e4310e9b934c64f14f396eedf391b6f6d201ad56c89de37da2", uri=>"mongodb://vncmgr:vnc22019@xsj-dvdbvnc02:27060/vncreg?replicaSet=vncdev", placeholder_db_dir=>"/opt/logstash-mongodb/", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_47a8a380-777a-4cec-86af-5099c36f6b89", enable_metric=>true, charset=>"UTF-8">, since_table=>"logstash_since", since_column=>"_id", since_type=>"id", parse_method=>"flatten", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>"Default message...", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': 'C:\opt' does not exist", :thread=>"#<Thread:0x3bcff4ef run>"}

Thanks
Kishore

4

You need to set the placeholder path to a directory that exists on your computer.

5

I am able to resolve the issue after placing the value of placeholder_db_dir to my local directory. Thanks for your help.

6

i am using logstash 7.5.2 and trying to connect mongodb using JDBC driver but cant find the suitable driver across the unity, progress (third party drivers)

7

Go to logstash/bin folder and list the plugins using the command (

Example on windows:

logstash-plugin.bat list

logstash-plugin.bat install logstash-input-mongodb

8

i done with a logstash-input-mongodb plugin method but i cant connnect via JDBC driver(logstash-input-jdbc)

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.