Restheart issue with _embedded http_poller

Elastic Stack Logstash
1

Hi,

I want to insert data from MongoDB to ElasticSearch using Logstash. Since there is no input plugin for MongoDB, I am using http_poller for input plugin which will communicate with MongoDB http interface restheart.

Now I am able to communicate with MongoDB with restheart. Here is response

http://localhost:8080/test/testing/

and response is as below

{
    "_embedded": [
        {
            "_id": {
                "$oid": "5a5db987cd0cac4b8ee1c8b7"
            },
            "name": "hello"
        }
    ],
    "_id": "testing",
    "_returned": 1
}

here is my logstash pipeline

input {
  http_poller {
    urls => {
      test2 => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => get
        user => "a"
        password => "a"
        url => "http://localhost:8080/test/testing/"
        headers => {
          Accept => "application/json"
        }
     }
    }
    request_timeout => 60
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
    schedule => { cron => "* * * * *"}
    codec => "json"
    # A hash of request metadata info (timing, response headers, etc.) will be sent here
    # metadata_target => "http_poller_metadata"
  }
}

filter {
		mutate {
				rename => { "[_id]" => "[mongo_id]" }
			   }
		}

output {
elasticsearch {
    hosts => 'http://localhost:9200'
    index => 'mongodbpoller'
    codec => json
    doc_as_upsert => true
    document_id => "%{[mongo_id][$oid]}"	

 }
  # stdout {
  #  codec => rubydebug
  # }
}

Now problem is when i run pipeline , as main data is wrapped inside _embedded object . Logstash consider _embedded as data and inserts it as whole document.

How can I only insert

{
            "_id":  "5a5db987cd0cac4b8ee1c8b7",
            "name": "hello"
        }
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.