# Merge two urls json at one doc **URL:** \ **Category:** Logstash\ **Created:** [May 31, 2019, 10:41am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707 "2019-05-31T10:41:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![igor.alvarez.lanit](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@igor.alvarez.lanit](https://discuss.elastic.co/u/igor.alvarez.lanit)\ **Post date:** [May 31, 2019, 10:41am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/1 "2019-05-31T10:41:16Z")
Hi, I use http\_poller imput logstash but i have two URLs and this create two docs.... I want to create only one doc with all fields of this urls, its is possible? input { http\_poller { urls =\> { station\_information =\> "[https://gbfs.nextbike.net/maps/gbfs/v1/nextbike\_le/de/station\_information.json](https://gbfs.nextbike.net/maps/gbfs/v1/nextbike_le/de/station_information.json)" station\_status =\> "[https://gbfs.nextbike.net/maps/gbfs/v1/nextbike\_le/de/station\_status.json](https://gbfs.nextbike.net/maps/gbfs/v1/nextbike_le/de/station_status.json)" } request\_timeout =\> 60 schedule =\> { every =\> "10s"} codec =\> "json" } } filter { { field =\> "[data][stations]" } } output { elasticsearch { hosts =\> "x.x.x.x:9200" user =\> "xxxx" password =\> "xxxx" document\_type =\> "logs" index =\> "name-%{+YYYY.MM}" } } ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [May 31, 2019, 11:07am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/2 "2019-05-31T11:07:07Z")
Yes, you can do that. You want to have something like: > ``` > { > "station_id": "11249439", > "num_bikes_available": 5, > "num_docks_available": 0, > "is_installed": 1, > "is_renting": 1, > "is_returning": 1, > "last_reported": 1559300580, > "station_id": "11249439", > "name": "Durstexpress", > "short_name": "4108", > "lat": 51.384999453022, > "lon": 12.39079819381, > "region_id": "1" > } > > ``` right? That will be one document in my understanding. Other stations will create another document? ---
**Author:** ![igor.alvarez.lanit](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@igor.alvarez.lanit](https://discuss.elastic.co/u/igor.alvarez.lanit)\ **Post date:** [May 31, 2019, 11:10am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/3 "2019-05-31T11:10:51Z")
Yes!! How to made this?? And yes, one docuemnt per station... ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [May 31, 2019, 11:12am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/4 "2019-05-31T11:12:55Z")
Hold on, writing the answer and testing. ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [May 31, 2019, 11:44am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/5 "2019-05-31T11:44:04Z")
There you go: The easiest solution is to load the document with the same ID to the ES and let it handle the merge there. ``` input { http_poller { urls => { station_information => "https://gbfs.nextbike.net/maps/gbfs/v1/nextbike_le/de/station_information.json" station_status => "https://gbfs.nextbike.net/maps/gbfs/v1/nextbike_le/de/station_status.json" } request_timeout => 60 schedule => { every => "10s"} codec => "json" } } filter { if [data][stations][1]{ split { field => "[data][stations]" } } fingerprint { method => "MD5" concatenate_sources => true source => ["[stations][station_id]"] target => ["fingerprint"] } } output { elasticsearch { hosts => ["192.168.1.1:9200", "192.168.1.2:9200"] index => "stations" action => "update" document_id => "%{fingerprint}" } } ``` ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78596d4218fb41ff40843abaec315c31d7f93da4.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/a/9a7fb22572eee8853bfbb80bde0a1ef9c982e511.png) Good luck with your endeavor! ---
**Author:** ![igor.alvarez.lanit](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@igor.alvarez.lanit](https://discuss.elastic.co/u/igor.alvarez.lanit)\ **Post date:** [June 3, 2019, 6:50am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/6 "2019-06-03T06:50:35Z")
With your .conf i have this error: ``` [WARN] 2019-06-03 08:48:46.784 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"9894f753eabb697a9578eedd6a749d29", :_index=>"stations", :_type=>"doc", :routing=>nil, :_retry_on_conflict=>1}, #], :response=>{"update"=>{"_index"=>"stations", "_type"=>"doc", "_id"=>"9894f753eabb697a9578eedd6a749d29", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[doc][9894f753eabb697a9578eedd6a749d29]: document missing", "index_uuid"=>"E4B5M576Syi8RQ2ht9_a0g", "shard"=>"0", "index"=>"stations"}}}} ``` `Preformatted text`[WARN] 2019-06-03 08:48:46.785 [[main]\>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=\>404, :action=\>["update", {:\_id=\>"9894f753eabb697a9578eedd6a749d29", :\_index=\>"stations", :\_type=\>"doc", :routing=\>nil, :\_retry\_on\_conflict=\>1}, #LogStash::Event:0x9b6f80e], :response=\>{"update"=\>{"\_index"=\>"stations", "\_type"=\>"doc", "\_id"=\>"9894f753eabb697a9578eedd6a749d29", "status"=\>404, "error"=\>{"type"=\>"document\_missing\_exception", "reason"=\>"[doc][9894f753eabb697a9578eedd6a749d29]: document missing", "index\_uuid"=\>"E4B5M576Syi8RQ2ht9\_a0g", "shard"=\>"0", "index"=\>"stations"}}}} I need create one doc with all information per schedule retry (60s for example) becouse i need to do it histogram or pie, etc... ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [June 3, 2019, 10:37am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/7 "2019-06-03T10:37:20Z")
Hey. Apologies, pasted the different version of the config. The working version is: ``` input { http_poller { urls => { station_information => "https://gbfs.nextbike.net/maps/gbfs/v1/nextbike_le/de/station_information.json" station_status => "https://gbfs.nextbike.net/maps/gbfs/v1/nextbike_le/de/station_status.json" } request_timeout => 60 schedule => { every => "10s"} codec => "json" } } filter { if [data][stations][1]{ split { field => "[data][stations]" } } fingerprint { method => "MD5" concatenate_sources => true source => ["[data][stations][station_id]"] target => ["fingerprint"] } } output { elasticsearch { hosts => ["192.168.1.1:9200", "192.168.1.2:9200"] index => "stations" document_id => "%{fingerprint}" action => "update" doc_as_upsert => "true" } } ``` With this configuration I do not get any warnings or errors as you have pasted one above. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 1, 2019, 10:37am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707/8 "2019-07-01T10:37:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.