# How to have a boolean based field migrated from MongoDB to Elasticsearch using Logstash?

**URL:** <https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589>\
**Category:** Logstash\
**Created:** [January 17, 2019, 8:34am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589 "2019-01-17T08:34:58Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 8:34am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/1 "2019-01-17T08:34:58Z")

</div>

I have banged my head on so many internet forums but failed to accomplish this.

Here are my sample documents present in my MongoDB in _test_ collection:

```
{
    "_id" : ObjectId("59a41f63dba6488b33268377"),
    "EntityID": "abc123"
    "IsRestrictedView" : false
},
{
    "_id" : ObjectId("59a41f63dba6488b33268388"),
    "EntityID" : "abc456",
    "IsRestrictedView" : true
},
{
    "_id" : ObjectId("59a41f63dba6488b33268399"),
    "EntityID" : "abc789",
    "IsRestrictedView" : false
 }

```

Here is content of my Logstash config file:

```
input {
    mongodb {
        uri => 'mongodb://localhost/TestDB'
        placeholder_db_dir => 'E:/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'test'
        batch_size => 5000
        generateId => false
    }
}
filter {
    mutate {
        rename => { "_id" => "mongo_id" }
    }
    mutate {
        convert => {"IsRestrictedView" => "boolean"}
    }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "test"
        doc_as_upsert => true
        document_id => "%{mongo_id}"
    }
}

```

The indexed data in Elasticsearch contains all other fields except "_IsRestrictedView_".

So,  
1. How can I get this boolean field correctly into Elasticsearch?  
2. Also the first document is always missing in indexed data. Why so?

```
* Logstash Version: 6.5.4
* Windows 10
```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 9:44am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/2 "2019-01-17T09:44:33Z")

</div>

How do you know it is not a boolean already?

What is the output from `stdout { codec => { rubydebug } }`? with the mutate/convert and without?

The mutate/convert (boolean) is expecting a string representation of some boolean like value.

```auto
  def convert_boolean(value)
    return true if value =~ TRUE_REGEX
    return false if value.empty? || value =~ FALSE_REGEX
    @logger.warn("Failed to convert #{value} into boolean.")
    value
  end

```

where

```auto
  TRUE_REGEX = (/^(true|t|yes|y|1)$/i).freeze
  FALSE_REGEX = (/^(false|f|no|n|0)$/i).freeze

```

Following the code snippet, you should have that WARN line logged or a trapped error message logged if the value is not a string.

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:07am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/3 "2019-01-17T10:07:40Z")

</div>

**With mutate convert, output from stdout { codec =\> {rubydebug} }**  
Some output omitted for brevity.

```
E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T14:55:56,160][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T14:55:56,178][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T14:56:03,175][INFO][logstash.inputs.mongodb] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T14:56:05,097][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T14:56:05.934000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T14:56:05.960000 #5644] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T14:56:06.005000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T14:56:06.009000 #5644] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T14:56:06.294000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T14:56:06.302000 #5644] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T14:56:06.315000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
D, [2019-01-17T14:56:06.337000 #5644] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T14:56:06.360000 #5644] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T14:56:06.363000 #5644] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T14:56:06,397][INFO][logstash.inputs.mongodb] Registering MongoDB input
D, [2019-01-17T14:56:07.433000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:07.569000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.123s
D, [2019-01-17T14:56:08.495000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:08.508000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.010s
[2019-01-17T14:56:08,721][INFO][logstash.inputs.mongodb] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false}
[2019-01-17T14:56:08,749][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6e0e8c71 run>"}
[2019-01-17T14:56:08,804][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
D, [2019-01-17T14:56:08.816000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:08.832000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.012s
[2019-01-17T14:56:09,184][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T14:56:09.604000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0x998a250dcc944f2c...>}}
D, [2019-01-17T14:56:09.661000 #5644] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.045s
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true}",
      "@version" => "1",
      "EntityID" => "abc456",
    "@timestamp" => 2019-01-17T09:56:09.001Z,
          "host" => "faisal-LP",
       "logdate" => "2017-08-28T13:49:23+00:00",
      "mongo_id" => "59a41f63dba6488b33268388"
}
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false}",
      "@version" => "1",
      "EntityID" => "abc789",
    "@timestamp" => 2019-01-17T09:56:09.063Z,
          "host" => "faisal-LP",
       "logdate" => "2017-08-28T13:49:23+00:00",
      "mongo_id" => "59a41f63dba6488b33268399"
}
```

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:08am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/4 "2019-01-17T10:08:22Z")

</div>

**Without mutate convert output from the stdout { codec =\> {rubydebug} }**  
Some output omitted for brevity.

```
E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T15:00:09,721][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T15:00:09,741][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T15:00:15,093][INFO][logstash.inputs.mongodb] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T15:00:17,123][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T15:00:17.725000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T15:00:17.759000 #2552] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T15:00:17.806000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T15:00:17.810000 #2552] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T15:00:18.130000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T15:00:18.138000 #2552] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T15:00:18.153000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
D, [2019-01-17T15:00:18.181000 #2552] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T15:00:18.203000 #2552] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T15:00:18.206000 #2552] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T15:00:18,241][INFO][logstash.inputs.mongodb] Registering MongoDB input
D, [2019-01-17T15:00:19.155000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:19.297000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.126s
D, [2019-01-17T15:00:20.355000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:20.368000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.011s
[2019-01-17T15:00:20,643][INFO][logstash.inputs.mongodb] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false}
[2019-01-17T15:00:20,686][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xee4536b run>"}
[2019-01-17T15:00:20,736][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
D, [2019-01-17T15:00:20.750000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:20.764000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.011s
[2019-01-17T15:00:21,076][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T15:00:21.125000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xceb656241da04b83...>}}
D, [2019-01-17T15:00:21.131000 #2552] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.006s
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false}",
          "host" => "faisal-LP",
      "@version" => "1",
      "EntityID" => "abc789",
    "@timestamp" => 2019-01-17T10:00:20.985Z,
      "mongo_id" => "59a41f63dba6488b33268399",
       "logdate" => "2017-08-28T13:49:23+00:00"
}
{
     "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true}",
          "host" => "faisal-LP",
      "@version" => "1",
      "EntityID" => "abc456",
    "@timestamp" => 2019-01-17T10:00:20.925Z,
      "mongo_id" => "59a41f63dba6488b33268388",
       "logdate" => "2017-08-28T13:49:23+00:00"
}
```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 10:20am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/5 "2019-01-17T10:20:57Z")

</div>

It looks like the mongodb input is not adding the `IsRestrictedView` field to the event - I'm not sure why is adding `EntityID` only. I wonder if there is an off-by-one bug. To test this, fetch a doc that has more than two fields (excluding `_id`) and we can see if the last one, as seen in `log_entry`, is missing.

Looking at the mongodb input code I think you need to set `parse_method => "simple"`.

Try that with and without as you just did.

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:22am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/6 "2019-01-17T10:22:23Z")

</div>

I am new to Elasticsearch, ruby. Please can you explain in detail how can I accomplish it? Thanks

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 10:25am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/7 "2019-01-17T10:25:47Z")

</div>

It is not ruby, I'm saying to add another setting to the mongodb input settings.

```auto
input {
    mongodb {
        uri => 'mongodb://localhost/TestDB'
        placeholder_db_dir => 'E:/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 'test'
        batch_size => 5000
        generateId => false
        # added setting below
        parse_method => "simple"
    }
}

```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 10:28am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/8 "2019-01-17T10:28:34Z")

</div>

I edited my post - you may not have seen my additions.

> I wonder if there is an off-by-one bug. To test this, fetch a doc that has more than two fields (excluding `_id` ) and we can see if the last one, as seen in `log_entry` , is missing.

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:34am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/9 "2019-01-17T10:34:41Z")

</div>

` **With mutate convert stdoutput** `

E:\Elasticsearch\logstash-6.5.4\bin\>logstash -f impTestCollFromLocalhost.conf  
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties  
[2019-01-17T15:30:16,821][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2019-01-17T15:30:16,838][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.5.4"}  
[2019-01-17T15:30:22,180][INFO][logstash.inputs.mongodb] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.  
[2019-01-17T15:30:24,092][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
D, [2019-01-17T15:30:24.757000 #1308] DEBUG -- : MONGODB | EVENT: #\<Mongo::Monitoring::Event::TopologyOpening topology=Unknown\>  
D, [2019-01-17T15:30:24.796000 #1308] DEBUG -- : MONGODB | Topology type 'unknown' initializing.  
D, [2019-01-17T15:30:24.847000 #1308] DEBUG -- : MONGODB | EVENT: #\<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown\>  
D, [2019-01-17T15:30:24.853000 #1308] DEBUG -- : MONGODB | Server localhost:27017 initializing.  
D, [2019-01-17T15:30:25.211000 #1308] DEBUG -- : MONGODB | EVENT: #\<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single\>  
D, [2019-01-17T15:30:25.222000 #1308] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.  
D, [2019-01-17T15:30:25.239000 #1308] DEBUG -- : MONGODB | EVENT: #Mongo::Monitoring::Event::ServerDescriptionChanged  
D, [2019-01-17T15:30:25.267000 #1308] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.  
D, [2019-01-17T15:30:25.303000 #1308] DEBUG -- : MONGODB | EVENT: #\<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single\>  
D, [2019-01-17T15:30:25.307000 #1308] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.  
[2019-01-17T15:30:25,353][INFO][logstash.inputs.mongodb] Registering MongoDB input  
D, [2019-01-17T15:30:26.303000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=\>1, "cursor"=\>{}, "nameOnly"=\>true, "filter"=\>{"name"=\>{"$not"=\>/system.|$/}}, "lsid"=\>{"id"=\>\<BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...\>}}  
D, [2019-01-17T15:30:26.439000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.124s  
D, [2019-01-17T15:30:27.289000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=\>"test", "filter"=\>{}, "sort"=\>{"\_id"=\>1}, "limit"=\>1, "lsid"=\>{"id"=\>\<BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...\>}}  
D, [2019-01-17T15:30:27.304000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.013s  
[2019-01-17T15:30:27,592][INFO][logstash.inputs.mongodb] init placeholder for logstash\_since\_test: {"\_id"=\>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=\>"abc123", "IsRestrictedView"=\>false, "Description"=\>"Here is description of 377"}  
[2019-01-17T15:30:27,614][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x1411c0ef run\>"}  
[2019-01-17T15:30:27,669][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
D, [2019-01-17T15:30:27.679000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=\>"test", "filter"=\>{"\_id"=\>{"$gt"=\>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=\>5000, "lsid"=\>{"id"=\>\<BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...\>}}  
D, [2019-01-17T15:30:27.694000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.012s  
[2019-01-17T15:30:27,998][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
D, [2019-01-17T15:30:28.431000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=\>1, "cursor"=\>{}, "nameOnly"=\>true, "filter"=\>{"name"=\>{"$not"=\>/system.|$/}}, "lsid"=\>{"id"=\>\<BSON::Binary:0x2010 type=uuid data=0xa4e1e1cc0dac4365...\>}}  
D, [2019-01-17T15:30:28.482000 #1308] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.047s  
{  
"@version" =\> "1",  
"host" =\> "faisal-LP",  
"@timestamp" =\> 2019-01-17T10:30:27.897Z,  
"mongo\_id" =\> "59a41f63dba6488b33268399",  
"IsRestrictedView" =\> false,  
"Description" =\> "Here is description of 399",  
"EntityID" =\> "abc789",  
"logdate" =\> "2017-08-28T13:49:23+00:00",  
"log\_entry" =\> "{"\_id"=\>BSON::ObjectId('59a41f63dba6488b33268399'), "EntityID"=\>"abc789", "IsRestrictedView"=\>false, "Description"=\>"Here is description of 399"}"  
}  
{  
"@version" =\> "1",  
"host" =\> "faisal-LP",  
"@timestamp" =\> 2019-01-17T10:30:27.851Z,  
"mongo\_id" =\> "59a41f63dba6488b33268388",  
"IsRestrictedView" =\> true,  
"Description" =\> "Here is description of 388",  
"EntityID" =\> "abc456",  
"logdate" =\> "2017-08-28T13:49:23+00:00",  
"log\_entry" =\> "{"\_id"=\>BSON::ObjectId('59a41f63dba6488b33268388'), "EntityID"=\>"abc456", "IsRestrictedView"=\>true, "Description"=\>"Here is description of 388"}"  
}

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:36am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/10 "2019-01-17T10:36:15Z")

</div>

**Without mutate convert stdout**. Done after you parse =\> simple suggestion

```
E:\Elasticsearch\logstash-6.5.4\bin>logstash -f impTestCollFromLocalhost.conf
Sending Logstash logs to E:/Elasticsearch/logstash-6.5.4/logs which is now configured via log4j2.properties
[2019-01-17T15:33:40,851][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-17T15:33:40,872][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-01-17T15:33:46,149][INFO][logstash.inputs.mongodb] Using version 0.1.x input plugin 'mongodb'. This plugin isn't well supported by the community and likely has no maintainer.
[2019-01-17T15:33:48,262][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
D, [2019-01-17T15:33:48.922000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyOpening topology=Unknown>
D, [2019-01-17T15:33:48.947000 #17152] DEBUG -- : MONGODB | Topology type 'unknown' initializing.
D, [2019-01-17T15:33:48.989000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerOpening address=localhost:27017 topology=Unknown>
D, [2019-01-17T15:33:48.993000 #17152] DEBUG -- : MONGODB | Server localhost:27017 initializing.
D, [2019-01-17T15:33:49.331000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Unknown new=Single>
D, [2019-01-17T15:33:49.338000 #17152] DEBUG -- : MONGODB | Topology type 'unknown' changed to type 'single'.
D, [2019-01-17T15:33:49.349000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::ServerDescriptionChanged>
D, [2019-01-17T15:33:49.373000 #17152] DEBUG -- : MONGODB | Server description for localhost:27017 changed from 'unknown' to 'standalone'.
D, [2019-01-17T15:33:49.397000 #17152] DEBUG -- : MONGODB | EVENT: #<Mongo::Monitoring::Event::TopologyChanged prev=Single new=Single>
D, [2019-01-17T15:33:49.401000 #17152] DEBUG -- : MONGODB | There was a change in the members of the 'single' topology.
[2019-01-17T15:33:49,433][INFO][logstash.inputs.mongodb] Registering MongoDB input
D, [2019-01-17T15:33:50.415000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:50.558000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.130s
D, [2019-01-17T15:33:51.608000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{}, "sort"=>{"_id"=>1}, "limit"=>1, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:51.622000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.012s
[2019-01-17T15:33:51,897][INFO][logstash.inputs.mongodb] init placeholder for logstash_since_test: {"_id"=>BSON::ObjectId('59a41f63dba6488b33268377'), "EntityID"=>"abc123", "IsRestrictedView"=>false, "Description"=>"Here is description of 377"}
[2019-01-17T15:33:51,931][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7272d95b sleep>"}
[2019-01-17T15:33:51,978][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
D, [2019-01-17T15:33:51.987000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | STARTED | {"find"=>"test", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('59a41f63dba6488b33268377')}}, "limit"=>5000, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:52.001000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.find | SUCCEEDED | 0.011s
[2019-01-17T15:33:52,337][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
D, [2019-01-17T15:33:52.785000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "filter"=>{"name"=>{"$not"=>/system\.|\$/}}, "lsid"=>{"id"=><BSON::Binary:0x2010 type=uuid data=0xea123124ce1d4330...>}}
D, [2019-01-17T15:33:52.841000 #17152] DEBUG -- : MONGODB | localhost:27017 | TestDB.listCollections | SUCCEEDED | 0.045s
{
         "Description" => "Here is description of 388",
                "host" => "faisal-LP",
          "@timestamp" => 2019-01-17T10:33:52.171Z,
            "mongo_id" => "59a41f63dba6488b33268388",
            "@version" => "1",
             "logdate" => "2017-08-28T13:49:23+00:00",
           "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268388'), \"EntityID\"=>\"abc456\", \"IsRestrictedView\"=>true, \"Description\"=>\"Here is description of 388\"}",
            "EntityID" => "abc456",
    "IsRestrictedView" => "true"
}
{
         "Description" => "Here is description of 399",
                "host" => "faisal-LP",
          "@timestamp" => 2019-01-17T10:33:52.230Z,
            "mongo_id" => "59a41f63dba6488b33268399",
            "@version" => "1",
             "logdate" => "2017-08-28T13:49:23+00:00",
           "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b33268399'), \"EntityID\"=>\"abc789\", \"IsRestrictedView\"=>false, \"Description\"=>\"Here is description of 399\"}",
            "EntityID" => "abc789",
    "IsRestrictedView" => "false"
}
```

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:37am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/11 "2019-01-17T10:37:59Z")

</div>

Ah.... I guess first one of my recent replies now showing IsRectrictedView  
Great

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 10:41am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/12 "2019-01-17T10:41:09Z")

</div>

Great. So the setting `parse_method => "simple"` works and it converts the mongodb values to strings if they are scalars - meaning you need the mutate/convert.

It seems as if the `parse_method => "flatten"`, used by default has an off-by-one bug.

Can you test if hypothesis?

If so temporarily switch back to not setting `parse_method`, you can just comment it out with a leading `#` character, and then fetch a bigger document - lets see if the last field (key) is missing.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 10:49am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/13 "2019-01-17T10:49:41Z")

</div>

I don't understand, in the last run you now have a `Description` field in your mongodb doc - it was not there before.

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 10:54am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/14 "2019-01-17T10:54:57Z")

</div>

Yeah just added it before my last run to docs.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 10:57am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/15 "2019-01-17T10:57:25Z")

</div>

I see.

BTW I spotted this in the mongodb input source repo [https://github.com/phutchins/logstash-input-mongodb/issues/85](https://github.com/phutchins/logstash-input-mongodb/issues/85)  
It seems to be referring to the `flatten` method.

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 11:04am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/16 "2019-01-17T11:04:17Z")

</div>

**Here is my larger document in MongoDB:**

```
{
    "_id" : ObjectId("59a41f63dba6488b332683ee"),
    "VideoID" : "DVR_One_Backseat_1_3132017164853",
    "StartDate" : ISODate("2017-03-13T11:48:53.310Z"),
    "EndDate" : ISODate("2017-03-13T12:48:53.857Z"),
    "ClientID" : 1,
    "UnitID" : "DVR_One",
    "UserID" : "guest ",
    "UserIDList" : [ 
        "0"
    ],
    "StationID" : 1,
    "Duration" : NumberLong(3605099),
    "FileSize" : NumberLong(552412952),
    "FileType" : "mp4",
    "ClientStatus" : 3,
    "MediaStatus" : 6,
    "MediaStatusDesc" : "Available",
    "MediaState" : 4,
    "MediaStateDesc" : "Deleted",
    "IsRestrictedView" : false,
    "ClientExpiryDate" : ISODate("2017-03-23T13:50:01.547Z"),
    "MediaTag" : "",
    "Descriptions" : "",
    "VideoData" : "DVR_One_Backseat_1_3132017164853 DVR_One guest Deleted Available Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr Incident 1 | Incident 2 mp4 Video In Car PIW_Station 37,test cont videos of dvr,test cont videos of dvr ",
    "Position" : {
        "lon" : 67.0487,
        "lat" : 24.859
    },
    "IncidentTypes" : [ 
        "Incident 1", 
        "Incident 2"
    ],
    "EGDetail" : [ 
        {
            "Count" : 3,
            "EvidenceGroupId" : 53
        }
    ],
    "ACDetail" : [ 
        {
            "CaseId" : "test cont videos of dvr",
            "SysSerial" : "37",
            "title" : "test cont videos of dvr"
        }
    ],
    "SensorsData" : [ 
        {
            "sensorEventId" : 1,
            "lat" : 24.859,
            "lon" : 67.0487,
            "logTime" : ISODate("2017-03-13T11:48:55.000Z")
        }, 
        {
            "sensorEventId" : 2,
            "lat" : 24.8595,
            "lon" : 67.0482,
            "logTime" : ISODate("2017-03-13T12:48:59.000Z")
        }, 
        {
            "sensorEventId" : 1,
            "lat" : 24.859,
            "lon" : 67.0487,
            "logTime" : ISODate("2017-03-13T11:48:55.000Z")
        }, 
        {
            "sensorEventId" : 2,
            "lat" : 24.8595,
            "lon" : 67.0482,
            "logTime" : ISODate("2017-03-13T12:48:59.000Z")
        }, 
        {
            "sensorEventId" : 1,
            "lat" : 24.859,
            "lon" : 67.0487,
            "logTime" : ISODate("2017-03-13T11:48:55.000Z")
        }, 
        {
            "sensorEventId" : 2,
            "lat" : 24.8595,
            "lon" : 67.0482,
            "logTime" : ISODate("2017-03-13T12:48:59.000Z")
        }
    ],
    "Forms" : "Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr"
}

```

**Output from stdout after running logstash**

```
{
        "Position_lat" => 24.859,
          "@timestamp" => 2019-01-17T10:49:40.299Z,
            "MediaTag" => "",
              "UnitID" => "DVR_One",
    "ClientExpiryDate" => "2017-03-23T13:50:01Z",
            "@version" => "1",
         "MediaStatus" => 6,
        "Position_lon" => 67.0487,
            "FileSize" => 552412952,
        "Descriptions" => "",
                "host" => "faisal-LP",
           "StartDate" => "2017-03-13T11:48:53Z",
             "EndDate" => "2017-03-13T12:48:53Z",
            "ClientID" => 1,
            "FileType" => "mp4",
            "Duration" => 3605099,
             "logdate" => "2017-08-28T13:49:23+00:00",
           "VideoData" => "DVR_One_Backseat_1_3132017164853 DVR_One guest Deleted Available Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr Incident 1 | Incident 2 mp4 Video In Car PIW_Station 37,test cont videos of dvr,test cont videos of dvr ",
            "location" => {
        "lon" => 67.0487,
        "lat" => 24.859
    },
     "MediaStatusDesc" => "Available",
      "MediaStateDesc" => "Deleted",
        "ClientStatus" => 3,
           "log_entry" => "{\"_id\"=>BSON::ObjectId('59a41f63dba6488b332683ee'), \"VideoID\"=>\"DVR_One_Backseat_1_3132017164853\", \"StartDate\"=>2017-03-13 11:48:53 UTC, \"EndDate\"=>2017-03-13 12:48:53 UTC, \"ClientID\"=>1, \"UnitID\"=>\"DVR_One\", \"UserID\"=>\"guest \", \"UserIDList\"=>[\"0\"], \"StationID\"=>1, \"Duration\"=>3605099, \"FileSize\"=>552412952, \"FileType\"=>\"mp4\", \"ClientStatus\"=>3, \"MediaStatus\"=>6, \"MediaStatusDesc\"=>\"Available\", \"MediaState\"=>4, \"MediaStateDesc\"=>\"Deleted\", \"IsRestrictedView\"=>false, \"ClientExpiryDate\"=>2017-03-23 13:50:01 UTC, \"MediaTag\"=>\"\", \"Descriptions\"=>\"\", \"VideoData\"=>\"DVR_One_Backseat_1_3132017164853 DVR_One guest Deleted Available Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr Incident 1 | Incident 2 mp4 Video In Car PIW_Station 37,test cont videos of dvr,test cont videos of dvr \", \"Position\"=>{\"lon\"=>67.0487, \"lat\"=>24.859}, \"IncidentTypes\"=>[\"Incident 1\", \"Incident 2\"], \"EGDetail\"=>[{\"Count\"=>3, \"EvidenceGroupId\"=>53}], \"ACDetail\"=>[{\"CaseId\"=>\"test cont videos of dvr\", \"SysSerial\"=>\"37\", \"title\"=>\"test cont videos of dvr\"}], \"SensorsData\"=>[{\"sensorEventId\"=>1, \"lat\"=>24.859, \"lon\"=>67.0487, \"logTime\"=>2017-03-13 11:48:55 UTC}, {\"sensorEventId\"=>2, \"lat\"=>24.8595, \"lon\"=>67.0482, \"logTime\"=>2017-03-13 12:48:59 UTC}, {\"sensorEventId\"=>1, \"lat\"=>24.859, \"lon\"=>67.0487, \"logTime\"=>2017-03-13 11:48:55 UTC}, {\"sensorEventId\"=>2, \"lat\"=>24.8595, \"lon\"=>67.0482, \"logTime\"=>2017-03-13 12:48:59 UTC}, {\"sensorEventId\"=>1, \"lat\"=>24.859, \"lon\"=>67.0487, \"logTime\"=>2017-03-13 11:48:55 UTC}, {\"sensorEventId\"=>2, \"lat\"=>24.8595, \"lon\"=>67.0482, \"logTime\"=>2017-03-13 12:48:59 UTC}], \"Forms\"=>\"Form 1 , Form 2 : test cont videos of dvr , test cont videos of dvr , test cont videos of dvr\"}",
              "UserID" => "guest ",
               "Forms" => [
        [0] "Form 1 ",
        [1] " Form 2 : test cont videos of dvr ",
        [2] " test cont videos of dvr ",
        [3] " test cont videos of dvr"
    ],
          "MediaState" => 4,
           "StationID" => 1,
            "mongo_id" => "59a41f63dba6488b332683ee",
             "VideoID" => "DVR_One_Backseat_1_3132017164853"
}

```

You can see `IsRestrictedView` not coming in indexed data if I don't set  
`parse_method => "simple"`

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 11:05am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/17 "2019-01-17T11:05:00Z")

</div>

The issue that I linked to disproves the off-by-one bug theory - the `flatten` code path simply ignores boolean values.

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 11:05am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/18 "2019-01-17T11:05:53Z")

</div>

Here is my input file for previous output

```
input {
	mongodb {
		uri => 'mongodb://localhost/TestDB'
		placeholder_db_dir => 'E:/logstash-mongodb/'
		placeholder_db_name => 'logstash_sqlite.db'
		collection => 'MediaSummary'
		batch_size => 5000
		generateId => false
		# parse_method => "simple"
    }
}
filter {
	mutate {
        rename => { "_id" => "mongo_id" }
	}
	date {
		match => ["StartDate", "yyyy-MM-dd HH:mm:ss.SSSZ"]
		timezone => "UTC"
	}
	date {
		match => ["EndDate", "yyyy-MM-dd HH:mm:ss.SSSZ"]
		timezone => "UTC"
	}
	date {
		match => ["ClientExpiryDate", "yyyy-MM-dd HH:mm:ss.SSSZ"]
		timezone => "UTC"
	}
	mutate {
		convert => { "Position_lat" => "float" }
		convert => { "Position_lon" => "float" }
    }
	mutate {
		 add_field => ["[location][lat]", "%{Position_lat}" ]
		 add_field => ["[location][lon]", "%{Position_lon}" ]
	}
	mutate {
		convert => {"[location][lat]" => "float"}
		convert => {"[location][lon]" => "float"}
	}
	mutate {
		convert => {"IsRestrictedView" => "boolean"}
	}
	mutate {
		split => {"Forms" => ","}
	}
	mutate {
		remove_field => ["path","tags"]
	}
	
}
output {
	stdout {
		codec => rubydebug
	}
}
```

---

<div class="post-metadata">

**Author:** ![topgun743](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/topgun743/32/39867_2.png) [@topgun743](https://discuss.elastic.co/u/topgun743)\
**Post date:** [January 17, 2019, 11:08am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/19 "2019-01-17T11:08:12Z")

</div>

Can you please also guide me how location field showing in my input config file be set correctly to `geo_point` type? So far its showing it only as an object with float members.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 17, 2019, 11:18am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589/20 "2019-01-17T11:18:14Z")

</div>

We first have to fix the boolean problem because you have compound (nested) documents in MongoDB and so you need to use `flatten` or `dig` parse\_method.

[Next page](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589.md?page=2)