How to have a boolean based field migrated from MongoDB to Elasticsearch using Logstash?

Elastic Stack Logstash
1

I have banged my head on so many internet forums but failed to accomplish this.

Here are my sample documents present in my MongoDB in test collection:

{
    "_id" : ObjectId("59a41f63dba6488b33268377"),
    "EntityID": "abc123"
    "IsRestrictedView" : false
},
{
    "_id" : ObjectId("59a41f63dba6488b33268388"),
    "EntityID" : "abc456",
    "IsRestrictedView" : true
},
{
    "_id" : ObjectId("59a41f63dba6488b33268399"),
    "EntityID" : "abc789",
    "IsRestrictedView" : false
 }

Here is content of my Logstash config file:

input {
    mongodb {
        uri => 'mongodb://localhost/TestDB'
        placeholder_db_dir => 'E:/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'test'
        batch_size => 5000
        generateId => false
    }
}
filter {
    mutate {
        rename => { "_id" => "mongo_id" }
    }
    mutate {
        convert => {"IsRestrictedView" => "boolean"}
    }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "test"
        doc_as_upsert => true
        document_id => "%{mongo_id}"
    }
}

The indexed data in Elasticsearch contains all other fields except "IsRestrictedView".

So,
1. How can I get this boolean field correctly into Elasticsearch?
2. Also the first document is always missing in indexed data. Why so?

* Logstash Version: 6.5.4
* Windows 10
2

How do you know it is not a boolean already?

What is the output from stdout { codec => { rubydebug } }? with the mutate/convert and without?

The mutate/convert (boolean) is expecting a string representation of some boolean like value.

  def convert_boolean(value)
    return true if value =~ TRUE_REGEX
    return false if value.empty? || value =~ FALSE_REGEX
    @logger.warn("Failed to convert #{value} into boolean.")
    value
  end

where

  TRUE_REGEX = (/^(true|t|yes|y|1)$/i).freeze
  FALSE_REGEX = (/^(false|f|no|n|0)$/i).freeze

Following the code snippet, you should have that WARN line logged or a trapped error message logged if the value is not a string.

3

With mutate convert, output from stdout { codec => {rubydebug} }
Some output omitted for brevity.

E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T14:55:56,160][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T14:55:56,178][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T14:56:03,175][INFO ][logstash.inputs.mongodb  ] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T14:56:05,097][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T14:56:05.934000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T14:56:05.960000 #5644] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T14:56:06.005000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T14:56:06.009000 #5644] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T14:56:06.294000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T14:56:06.302000 #5644] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T14:56:06.315000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
D, [2019-01-17T14:56:06.337000 #5644] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T14:56:06.360000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T14:56:06.363000 #5644] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T14:56:06,397][INFO ][logstash.inputs.mongodb  ] Registering MongoDB input
D, [2019-01-17T14:56:07.433000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:07.569000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.123s
D, [2019-01-17T14:56:08.495000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:08.508000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.010s
[2019-01-17T14:56:08,721][INFO ][logstash.inputs.mongodb  ] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false}
[2019-01-17T14:56:08,749][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6e0e8c71 run>"}
[2019-01-17T14:56:08,804][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
D, [2019-01-17T14:56:08.816000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:08.832000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.012s
[2019-01-17T14:56:09,184][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T14:56:09.604000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:09.661000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.045s
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true}",
      "@version" => "1",
      "EntityID" => "abc456",
    "@timestamp" => 2019-01-17T09:56:09.001Z,
          "host" => "faisal-LP",
       "logdate" => "2017-08-28T13:49:23+00:00",
      "mongo_id" => "59a41f63dba6488b33268388"
}
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false}",
      "@version" => "1",
      "EntityID" => "abc789",
    "@timestamp" => 2019-01-17T09:56:09.063Z,
          "host" => "faisal-LP",
       "logdate" => "2017-08-28T13:49:23+00:00",
      "mongo_id" => "59a41f63dba6488b33268399"
}
4

Without mutate convert output from the stdout { codec => {rubydebug} }
Some output omitted for brevity.

E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T15:00:09,721][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T15:00:09,741][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T15:00:15,093][INFO ][logstash.inputs.mongodb  ] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T15:00:17,123][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T15:00:17.725000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T15:00:17.759000 #2552] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T15:00:17.806000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T15:00:17.810000 #2552] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T15:00:18.130000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T15:00:18.138000 #2552] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T15:00:18.153000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
D, [2019-01-17T15:00:18.181000 #2552] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T15:00:18.203000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T15:00:18.206000 #2552] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T15:00:18,241][INFO ][logstash.inputs.mongodb  ] Registering MongoDB input
D, [2019-01-17T15:00:19.155000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:19.297000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.126s
D, [2019-01-17T15:00:20.355000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:20.368000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.011s
[2019-01-17T15:00:20,643][INFO ][logstash.inputs.mongodb  ] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false}
[2019-01-17T15:00:20,686][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xee4536b run>"}
[2019-01-17T15:00:20,736][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
D, [2019-01-17T15:00:20.750000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:20.764000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.011s
[2019-01-17T15:00:21,076][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T15:00:21.125000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:21.131000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.006s
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false}",
          "host" => "faisal-LP",
      "@version" => "1",
      "EntityID" => "abc789",
    "@timestamp" => 2019-01-17T10:00:20.985Z,
      "mongo_id" => "59a41f63dba6488b33268399",
       "logdate" => "2017-08-28T13:49:23+00:00"
}
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true}",
          "host" => "faisal-LP",
      "@version" => "1",
      "EntityID" => "abc456",
    "@timestamp" => 2019-01-17T10:00:20.925Z,
      "mongo_id" => "59a41f63dba6488b33268388",
       "logdate" => "2017-08-28T13:49:23+00:00"
}
5

It looks like the mongodb input is not adding the IsRestrictedView field to the event - I'm not sure why is adding EntityID only. I wonder if there is an off-by-one bug. To test this, fetch a doc that has more than two fields (excluding _id) and we can see if the last one, as seen in log_entry, is missing.

Looking at the mongodb input code I think you need to set parse_method => "simple".

Try that with and without as you just did.

6

I am new to Elasticsearch, ruby. Please can you explain in detail how can I accomplish it? Thanks

7

It is not ruby, I'm saying to add another setting to the mongodb input settings.

input {
    mongodb {
        uri => 'mongodb://localhost/TestDB'
        placeholder_db_dir => 'E:/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'test'
        batch_size => 5000
        generateId => false
        # added setting below
        parse_method => "simple"
    }
}
8

I edited my post - you may not have seen my additions.

I wonder if there is an off-by-one bug. To test this, fetch a doc that has more than two fields (excluding _id ) and we can see if the last one, as seen in log_entry , is missing.

9

**With mutate convert stdoutput**

E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T15:30:16,821][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T15:30:16,838][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T15:30:22,180][INFO ][logstash.inputs.mongodb ] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T15:30:24,092][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T15:30:24.757000 #1308] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T15:30:24.796000 #1308] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T15:30:24.847000 #1308] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T15:30:24.853000 #1308] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T15:30:25.211000 #1308] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T15:30:25.222000 #1308] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T15:30:25.239000 #1308] DEBUG -- : MONGODB | EVENT: #Mongo::Monitoring::Event::ServerDescriptionChanged
D, [2019-01-17T15:30:25.267000 #1308] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T15:30:25.303000 #1308] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T15:30:25.307000 #1308] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T15:30:25,353][INFO ][logstash.inputs.mongodb ] Registering MongoDB input
D, [2019-01-17T15:30:26.303000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system.|$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...>}}
D, [2019-01-17T15:30:26.439000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.124s
D, [2019-01-17T15:30:27.289000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...>}}
D, [2019-01-17T15:30:27.304000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.013s
[2019-01-17T15:30:27,592][INFO ][logstash.inputs.mongodb ] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false, "Description"=>"Here is description of 377"}
[2019-01-17T15:30:27,614][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1411c0ef run>"}
[2019-01-17T15:30:27,669][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
D, [2019-01-17T15:30:27.679000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...>}}
D, [2019-01-17T15:30:27.694000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.012s
[2019-01-17T15:30:27,998][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T15:30:28.431000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system.|$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...>}}
D, [2019-01-17T15:30:28.482000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.047s
{
"@version" => "1",
"host" => "faisal-LP",
"@timestamp" => 2019-01-17T10:30:27.897Z,
"mongo_id" => "59a41f63dba6488b33268399",
"IsRestrictedView" => false,
"Description" => "Here is description of 399",
"EntityID" => "abc789",
"logdate" => "2017-08-28T13:49:23+00:00",
"log_entry" => "{"_id"=>BSON::ObjectId('59a41f63dba6488b33268399'), "EntityID"=>"abc789", "IsRestrictedView"=>false, "Description"=>"Here is description of 399"}"
}
{
"@version" => "1",
"host" => "faisal-LP",
"@timestamp" => 2019-01-17T10:30:27.851Z,
"mongo_id" => "59a41f63dba6488b33268388",
"IsRestrictedView" => true,
"Description" => "Here is description of 388",
"EntityID" => "abc456",
"logdate" => "2017-08-28T13:49:23+00:00",
"log_entry" => "{"_id"=>BSON::ObjectId('59a41f63dba6488b33268388'), "EntityID"=>"abc456", "IsRestrictedView"=>true, "Description"=>"Here is description of 388"}"
}

10

Without mutate convert stdout. Done after you parse => simple suggestion

E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T15:33:40,851][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T15:33:40,872][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T15:33:46,149][INFO ][logstash.inputs.mongodb  ] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T15:33:48,262][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T15:33:48.922000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T15:33:48.947000 #17152] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T15:33:48.989000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T15:33:48.993000 #17152] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T15:33:49.331000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T15:33:49.338000 #17152] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T15:33:49.349000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
D, [2019-01-17T15:33:49.373000 #17152] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T15:33:49.397000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T15:33:49.401000 #17152] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T15:33:49,433][INFO ][logstash.inputs.mongodb  ] Registering MongoDB input
D, [2019-01-17T15:33:50.415000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:50.558000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.130s
D, [2019-01-17T15:33:51.608000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:51.622000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.012s
[2019-01-17T15:33:51,897][INFO ][logstash.inputs.mongodb  ] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false, "Description"=>"Here is description of 377"}
[2019-01-17T15:33:51,931][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7272d95b sleep>"}
[2019-01-17T15:33:51,978][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
D, [2019-01-17T15:33:51.987000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:52.001000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.011s
[2019-01-17T15:33:52,337][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T15:33:52.785000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:52.841000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.045s
{
         "Description" => "Here is description of 388",
                "host" => "faisal-LP",
          "@timestamp" => 2019-01-17T10:33:52.171Z,
            "mongo_id" => "59a41f63dba6488b33268388",
            "@version" => "1",
             "logdate" => "2017-08-28T13:49:23+00:00",
           "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true, \"Description\"=>\"Here is description of 388\"}",
            "EntityID" => "abc456",
    "IsRestrictedView" => "true"
}
{
         "Description" => "Here is description of 399",
                "host" => "faisal-LP",
          "@timestamp" => 2019-01-17T10:33:52.230Z,
            "mongo_id" => "59a41f63dba6488b33268399",
            "@version" => "1",
             "logdate" => "2017-08-28T13:49:23+00:00",
           "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false, \"Description\"=>\"Here is description of 399\"}",
            "EntityID" => "abc789",
    "IsRestrictedView" => "false"
}
11

Ah.... I guess first one of my recent replies now showing IsRectrictedView
Great

1 Like
12

Great. So the setting parse_method => "simple" works and it converts the mongodb values to strings if they are scalars - meaning you need the mutate/convert.

It seems as if the parse_method => "flatten", used by default has an off-by-one bug.

Can you test if hypothesis?

If so temporarily switch back to not setting parse_method, you can just comment it out with a leading # character, and then fetch a bigger document - lets see if the last field (key) is missing.

13

I don't understand, in the last run you now have a Description field in your mongodb doc - it was not there before.

14

Yeah just added it before my last run to docs.

15

I see.

BTW I spotted this in the mongodb input source repo https://github.com/phutchins/logstash-input-mongodb/issues/85
It seems to be referring to the flatten method.

16

Here is my larger document in MongoDB:

{
    "_id" : ObjectId("59a41f63dba6488b332683ee"),
    "VideoID" : "DVR_One_Backseat_1_3132017164853",
    "StartDate" : ISODate("2017-03-13T11:48:53.310Z"),
    "EndDate" : ISODate("2017-03-13T12:48:53.857Z"),
    "ClientID" : 1,
    "UnitID" : "DVR_One",
    "UserID" : "guest ",
    "UserIDList" : [ 
        "0"
    ],
    "StationID" : 1,
    "Duration" : NumberLong(3605099),
    "FileSize" : NumberLong(552412952),
    "FileType" : "mp4",
    "ClientStatus" : 3,
    "MediaStatus" : 6,
    "MediaStatusDesc" : "Available",
    "MediaState" : 4,
    "MediaStateDesc" : "Deleted",
    "IsRestrictedView" : false,
    "ClientExpiryDate" : ISODate("2017-03-23T13:50:01.547Z"),
    "MediaTag" : "",
    "Descriptions" : "",
    "VideoData" : "DVR_One_Backseat_1_3132017164853 DVR_One guest  Deleted Available Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr  Incident 1 | Incident 2 mp4  Video In Car PIW_Station 37,test cont videos of dvr,test cont videos of dvr ",
    "Position" : {
        "lon" : 67.0487,
        "lat" : 24.859
    },
    "IncidentTypes" : [ 
        "Incident 1", 
        "Incident 2"
    ],
    "EGDetail" : [ 
        {
            "Count" : 3,
            "EvidenceGroupId" : 53
        }
    ],
    "ACDetail" : [ 
        {
            "CaseId" : "test cont videos of dvr",
            "SysSerial" : "37",
            "title" : "test cont videos of dvr"
        }
    ],
    "SensorsData" : [ 
        {
            "sensorEventId" : 1,
            "lat" : 24.859,
            "lon" : 67.0487,
            "logTime" : ISODate("2017-03-13T11:48:55.000Z")
        }, 
        {
            "sensorEventId" : 2,
            "lat" : 24.8595,
            "lon" : 67.0482,
            "logTime" : ISODate("2017-03-13T12:48:59.000Z")
        }, 
        {
            "sensorEventId" : 1,
            "lat" : 24.859,
            "lon" : 67.0487,
            "logTime" : ISODate("2017-03-13T11:48:55.000Z")
        }, 
        {
            "sensorEventId" : 2,
            "lat" : 24.8595,
            "lon" : 67.0482,
            "logTime" : ISODate("2017-03-13T12:48:59.000Z")
        }, 
        {
            "sensorEventId" : 1,
            "lat" : 24.859,
            "lon" : 67.0487,
            "logTime" : ISODate("2017-03-13T11:48:55.000Z")
        }, 
        {
            "sensorEventId" : 2,
            "lat" : 24.8595,
            "lon" : 67.0482,
            "logTime" : ISODate("2017-03-13T12:48:59.000Z")
        }
    ],
    "Forms" : "Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr"
}

Output from stdout after running logstash

{
        "Position_lat" => 24.859,
          "@timestamp" => 2019-01-17T10:49:40.299Z,
            "MediaTag" => "",
              "UnitID" => "DVR_One",
    "ClientExpiryDate" => "2017-03-23T13:50:01Z",
            "@version" => "1",
         "MediaStatus" => 6,
        "Position_lon" => 67.0487,
            "FileSize" => 552412952,
        "Descriptions" => "",
                "host" => "faisal-LP",
           "StartDate" => "2017-03-13T11:48:53Z",
             "EndDate" => "2017-03-13T12:48:53Z",
            "ClientID" => 1,
            "FileType" => "mp4",
            "Duration" => 3605099,
             "logdate" => "2017-08-28T13:49:23+00:00",
           "VideoData" => "DVR_One_Backseat_1_3132017164853 DVR_One guest  Deleted Available Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr  Incident 1 | Incident 2 mp4  Video In Car PIW_Station 37,test cont videos of dvr,test cont videos of dvr ",
            "location" => {
        "lon" => 67.0487,
        "lat" => 24.859
    },
     "MediaStatusDesc" => "Available",
      "MediaStateDesc" => "Deleted",
        "ClientStatus" => 3,
           "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b332683ee'), \"VideoID\"=>\"DVR_One_Backseat_1_3132017164853\", \"StartDate\"=>2017-03-13 11:48:53 UTC, \"EndDate\"=>2017-03-13 12:48:53 UTC, \"ClientID\"=>1, \"UnitID\"=>\"DVR_One\", \"UserID\"=>\"guest \", \"UserIDList\"=>[\"0\"], \"StationID\"=>1, \"Duration\"=>3605099, \"FileSize\"=>552412952, \"FileType\"=>\"mp4\", \"ClientStatus\"=>3, \"MediaStatus\"=>6, \"MediaStatusDesc\"=>\"Available\", \"MediaState\"=>4, \"MediaStateDesc\"=>\"Deleted\", \"IsRestrictedView\"=>false, \"ClientExpiryDate\"=>2017-03-23 13:50:01 UTC, \"MediaTag\"=>\"\", \"Descriptions\"=>\"\", \"VideoData\"=>\"DVR_One_Backseat_1_3132017164853 DVR_One guest  Deleted Available Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr  Incident 1 | Incident 2 mp4  Video In Car PIW_Station 37,test cont videos of dvr,test cont videos of dvr \", \"Position\"=>{\"lon\"=>67.0487, \"lat\"=>24.859}, \"IncidentTypes\"=>[\"Incident 1\", \"Incident 2\"], \"EGDetail\"=>[{\"Count\"=>3, \"EvidenceGroupId\"=>53}], \"ACDetail\"=>[{\"CaseId\"=>\"test cont videos of dvr\", \"SysSerial\"=>\"37\", \"title\"=>\"test cont videos of dvr\"}], \"SensorsData\"=>[{\"sensorEventId\"=>1, \"lat\"=>24.859, \"lon\"=>67.0487, \"logTime\"=>2017-03-13 11:48:55 UTC}, {\"sensorEventId\"=>2, \"lat\"=>24.8595, \"lon\"=>67.0482, \"logTime\"=>2017-03-13 12:48:59 UTC}, {\"sensorEventId\"=>1, \"lat\"=>24.859, \"lon\"=>67.0487, \"logTime\"=>2017-03-13 11:48:55 UTC}, {\"sensorEventId\"=>2, \"lat\"=>24.8595, \"lon\"=>67.0482, \"logTime\"=>2017-03-13 12:48:59 UTC}, {\"sensorEventId\"=>1, \"lat\"=>24.859, \"lon\"=>67.0487, \"logTime\"=>2017-03-13 11:48:55 UTC}, {\"sensorEventId\"=>2, \"lat\"=>24.8595, \"lon\"=>67.0482, \"logTime\"=>2017-03-13 12:48:59 UTC}], \"Forms\"=>\"Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr\"}",
              "UserID" => "guest ",
               "Forms" => [
        [0] "Form 1 ",
        [1] " Form 2 : test cont videos of dvr ",
        [2] " test cont videos of dvr ",
        [3] " test cont videos of dvr"
    ],
          "MediaState" => 4,
           "StationID" => 1,
            "mongo_id" => "59a41f63dba6488b332683ee",
             "VideoID" => "DVR_One_Backseat_1_3132017164853"
}

You can see IsRestrictedView not coming in indexed data if I don't set
parse_method => "simple"

17

The issue that I linked to disproves the off-by-one bug theory - the flatten code path simply ignores boolean values.

18

Here is my input file for previous output

input {
	mongodb {
		uri => 'mongodb://localhost/TestDB'
		placeholder_db_dir => 'E:/logstash-mongodb/'
		placeholder_db_name => 'logstash_sqlite.db'
		collection => 'MediaSummary'
		batch_size => 5000
		generateId => false
		# parse_method => "simple"
    }
}
filter {
	mutate {
        rename => { "_id" => "mongo_id" }
	}
	date {
		match => ["StartDate", "yyyy-MM-dd HH:mm:ss.SSSZ"]
		timezone => "UTC"
	}
	date {
		match => ["EndDate", "yyyy-MM-dd HH:mm:ss.SSSZ"]
		timezone => "UTC"
	}
	date {
		match => ["ClientExpiryDate", "yyyy-MM-dd HH:mm:ss.SSSZ"]
		timezone => "UTC"
	}
	mutate {
		convert => { "Position_lat" => "float" }
		convert => { "Position_lon" => "float" }
    }
	mutate {
		 add_field => [ "[location][lat]", "%{Position_lat}" ]
		 add_field => [ "[location][lon]", "%{Position_lon}" ]
	}
	mutate {
		convert => {"[location][lat]" => "float"}
		convert => {"[location][lon]" => "float"}
	}
	mutate {
		convert => {"IsRestrictedView" => "boolean"}
	}
	mutate {
		split => {"Forms" => ","}
	}
	mutate {
		remove_field => ["path","tags"]
	}
	
}
output {
	stdout {
		codec => rubydebug
	}
}
19

Can you please also guide me how location field showing in my input config file be set correctly to geo_point type? So far its showing it only as an object with float members.

20

We first have to fix the boolean problem because you have compound (nested) documents in MongoDB and so you need to use flatten or dig parse_method.