Logstash from Mongo To Elastic - Array Problem

Elastic Stack Logstash
1

Hello guys, I'm running Logstash to sync my ElasticSearch with my MongoDB operation.
I used the mongdb plugin to read from mongo, this is my initial config file:

input {
mongodb {
uri => 'mongodb://localhost:27017/db'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'catalog'
}
}
output {
elasticsearch {
hosts => ['localhost:9200']
user => 'elastic'
password => 'xxxxx'
index => 'catalog'
}
}

I have a problem on syncronizind the json containing arrays.
This is my inserto on mongo:

db.getCollection('catalog').insert({
"schema" : {
"first" : {
"name" : "name",
"type" : "type",
"namespace" : "namespace",
"fields" : [
{
"name" : "first_field",
"type" : "string"
},
{
"name" : "second_field",
"type" : "string"
}
]
}
}
})

and this is the result on elasticsearch:

{
"_index": "catalog",
"_type": "doc",
"_id": "Jd_lL2IBwDdr4Pihholf",
"_score": 1,
"_source": {
"@version": "1",
"@timestamp": "2018-03-16T17:39:52.899Z",
"logdate": "2018-03-16T17:35:10+00:00",
"schema_first_name": "name",
"schema_first_namespace": "namespace",
"log_entry": """{"_id"=>BSON::ObjectId('5aac004edf7f789a5cbf6fe5'), "schema"=>{"first"=>{"name"=>"name", "type"=>"type", "namespace"=>"namespace", "fields"=>[{"name"=>"first_field", "type"=>"string"}, {"name"=>"second_field", "type"=>"string"}]}}}""",
"host": "",
"schema_first_type": "type",
"mongo_id": "5aac004edf7f789a5cbf6fe5"
}
}

i've lost the "fields" object.
I tried different solution, for example:

filter {
split {
field => "fields"
}
}

but I always received the follow error:

[logstash.filters.split ] Only String and Array types are splittable. field:fields is of type = NilClass_

Any suggestion?
Thanks in advance.

PPL

2

It looks like you're probably using Phillip Hutchins' community-contributed plugin, which was designed for a specific use-case and may or may not fit your needs.

From what I can tell, by default the input "flatten"s the structured object; it can also be configured to "dig" out specific bits, and has a "simple" option that does surprising things like discarding the sign of negative numbers.

It's also unclear to me what exactly the local database does, or how that database is kept from growing unbounded.

3

Thanks yaauie, you are right, but about you wich is the best way to sync MongoDB and ES?
(I need for a runtime sync) I tried also via jdbc plugin but I didn't find a good jdbc driver for mongodb.

4

If you need a runtime sync, you'll need to dual-write in your application since MongoDB doesn't expose a transaction log -- there is no way to get the creates, updates, and deletes in the order that they happened so that we can keep them in sync other than hooking into the application that is performing those actions.

5

And what about running mongo in replica-set mode? I read in this article that is possible to sync via the mongo oplog:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.