# Pulling Nested XML Data from a http API for loading into Elasticsearch

**URL:** <https://discuss.elastic.co/t/pulling-nested-xml-data-from-a-http-api-for-loading-into-elasticsearch/52492>\
**Category:** Logstash\
**Created:** [June 11, 2016, 7:22am UTC](https://discuss.elastic.co/t/pulling-nested-xml-data-from-a-http-api-for-loading-into-elasticsearch/52492 "2016-06-11T07:22:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Johnnycc](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@Johnnycc](https://discuss.elastic.co/u/Johnnycc)\
**Post date:** [June 11, 2016, 7:22am UTC](https://discuss.elastic.co/t/pulling-nested-xml-data-from-a-http-api-for-loading-into-elasticsearch/52492/1 "2016-06-11T07:22:14Z")

</div>

Ok first time poster - so please go easy on me - two questions.

I would like to be able to pull data with logstash from an API. The API is at an address that looks like the following;  
[http://domain.com/reports/1000000](http://domain.com/reports/1000000)  
[http://domain.com/reports/1000001](http://domain.com/reports/1000001)  
[http://domain.com/reports/1000002](http://domain.com/reports/1000002)  
....  
[http://domain.com/reports/9999999](http://domain.com/reports/9999999)

Is there are way to make incremental http requests to pull this data into logstash so it can be pushed into elasticsearch? Or is there a recommended tool for this purpose?

Here is the data that will be returned by the API.

`

 123456 2015-01-01 COKE FANTA `

As I am sure you know logstash will convert this to the following JSON

For the first name.  
"name" =\> [  
[0] "COKE"  
]

For the second name  
"name" =\> [  
[0] {  
"type" =\> "M",  
"content" =\> "FANTA"  
}  
]

And when I try to push this into elastic it will fail because the mapping will expect a string for the name field or expect an object - it cant't have both. How best to deal with this problem. I want to keep the party type and name details together - so xpathing just the name contents out does not meet my requirements.

---

<div class="post-metadata">

**Author:** ![Johnnycc](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@Johnnycc](https://discuss.elastic.co/u/Johnnycc)\
**Post date:** [June 23, 2016, 11:07am UTC](https://discuss.elastic.co/t/pulling-nested-xml-data-from-a-http-api-for-loading-into-elasticsearch/52492/2 "2016-06-23T11:07:38Z")

</div>

Thanks Guys I am over the moon able the latest logstash release which included the force\_content configuration. Basically the below configuration

> filter {  
> xml {source =\> "message"  
> force\_content =\> "true"  
> target =\> "parsed"}  
> }

Gives the below output which is now consistent even when some fields have attributes while others fields have not attributes.

```
"parsed" => {
 "xmlns" => "http://domain.com/schema/reporting",
"header" => [
    [0] {
        "repNumber" => [
            [0] {
                "content" => "123456"
            }
        ],
          "repDate" => [
            [0] {
                "content" => "2015-01-01"
            }
        ]
    }
],
 "party" => [
    [0] {
        "type" => "R",
        "name" => [
            [0] {
                "content" => "COKE"
            }
        ]
    },
    [1] {
        "type" => "S",
        "name" => [
            [0] {
                   "type" => "M",
                "content" => "FANTA"
            }
        ]
    }
]

```

}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:51am UTC](https://discuss.elastic.co/t/pulling-nested-xml-data-from-a-http-api-for-loading-into-elasticsearch/52492/3 "2017-07-06T04:51:00Z")

</div>

Closed on July 6, 2017, 4:51am UTC