# Issue with modelling data while bulk import
**URL:** https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032
**Category:** Logstash
**Created:** [April 6, 2018, 6:32am UTC](https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032 "2018-04-06T06:32:31Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Amit\_Jaiswal](https://discuss.elastic.co/u/Amit_Jaiswal)
#### Post date: [April 6, 2018, 6:32am UTC](https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032/1 "2018-04-06T06:32:31Z")
Hello,
I have to bulk import data via LogStash as below:
Source: Restful Get APIs
Destination : ElasticSearch
My .conf file looks as below:
input {
http\_poller {
urls =\> {
test1 =\> {
method =\> get
url =\> "[https://someurl/movies](https://someurl/movies)"
headers =\> {
Accept =\> "application/json"
}
}
}
output {
elasticsearch {
action =\>"index"
index =\> "movies"
document\_type =\> "movie"
manage\_template =\> false
document\_id =\> "%{movieId}"
doc\_as\_upsert =\> true
hosts =\> ["[http://localhost:9200/](http://localhost:9200/)"]
}
}
This pushes all the data returned from api to Elastic Search but not in desired format.
Problem statement:
Source API returns an object with Arraylist (that has 100 movie objects).
{"movieList":[{"movieId":1,"movieName":"Bank1","uniqueName":"Bank1","showInSearch":true,"address":{"name":"Bank1"},"notes":"Preversion for webservice more data comes later"},{"movieId":2,"movieName":"Alpha Omega","uniqueName":"Alpha Omega","showInSearch":true,"address":{"name":"Alpha Omega"},"description":"Alpha Omega's offering ."}]}
This inserts data into Elastic search with only single record "movielist" which internally has multiple records.
For proper searching , i want to insert individual records instead of 1 complete arraylist .
Please guide me , how can i accomplish this .
Thanks!
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [April 6, 2018, 8:51am UTC](https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032/2 "2018-04-06T08:51:00Z")
If you get one big event with something like this ...
```
"movieList" => [
[0] {
"address" => {
"name" => "Bank1"
},
"notes" => "Preversion for webservice more data comes later",
"uniqueName" => "Bank1",
"movieId" => 1,
"showInSearch" => true,
"movieName" => "Bank1"
},
[1] {
"address" => {
"name" => "Alpha Omega"
},
"uniqueName" => "Alpha Omega",
"description" => "Alpha Omegas offering .",
"movieId" => 2,
"showInSearch" => true,
"movieName" => "Alpha Omega"
}
]
```
you can do this:
```
split {
field => "movieList"
}
ruby {
code => "
event.get('movieList').each {|k, v| event.set(k, v) }
event.remove('movieList')
"
}
```
The split filter should give you seperate events for the array entries and the ruby code moves the data from the field "movieList" to the root of those events.
---
### Author:  [@Amit\_Jaiswal](https://discuss.elastic.co/u/Amit_Jaiswal)
#### Post date: [April 9, 2018, 6:33am UTC](https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032/3 "2018-04-09T06:33:52Z")
Thanks Jenni !!
It now shows different record instead of one big event.
---
### Author:  [@Amit\_Jaiswal](https://discuss.elastic.co/u/Amit_Jaiswal)
#### Post date: [April 10, 2018, 6:55am UTC](https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032/4 "2018-04-10T06:55:02Z")
> [@Amit\_Jaiswal](#):
>
> Hello,
>
> I have to bulk import data via LogStash as below:
>
> Source: Restful Get APIs
>
> Destination : Elasticsearch
>
> My .conf file looks as below:
>
> input {
>
> http\_poller {
>
> urls =\> {
>
> test1 =\> {
>
> method =\> get
>
> url =\> "[https://someurl/movies](https://someurl/movies)"
>
> headers =\> {
>
> Accept =\> "application/json"
>
> }
>
> }
>
> }
>
> output {
>
> elasticsearch {
>
> action =\>"index"
>
> index =\> "movies"
>
> document\_type =\> "movie"
>
> manage\_template =\> false
>
> document\_id =\> "%{movieId}"
>
> doc\_as\_upsert =\> true
>
> hosts =\> ["[http://localhost:9200/](http://localhost:9200/)"]
>
> }
>
> }
How can i perform pagination while providing Input URL to HTTP Poller
url =\> "[https://someurl/movies?limit=100&offset=0](https://someurl/movies?limit=100&offset=0)"
Total records are more than 5000, how can i write pagination logic in this .conf file?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 8, 2018, 6:55am UTC](https://discuss.elastic.co/t/issue-with-modelling-data-while-bulk-import/127032/5 "2018-05-08T06:55:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.