# Bulk Data upload in logstash

**URL:** <https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804>\
**Category:** Logstash\
**Created:** [November 15, 2018, 8:11am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804 "2018-11-15T08:11:43Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 15, 2018, 8:11am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/1 "2018-11-15T08:11:43Z")

</div>

Hi All,

We have been trying to index data using logstash from servicenow through restapi's, we dont see any errors and data is getting indexed only issue we are facing is we do have around 2lakh records to be indexed but only 10k records are getting indexed.  
Is there any restriction applied in logstash to index only certain amount of data, or in other words how can i index a bulk data like 2lakh+ records.

Any advice please

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [November 20, 2018, 5:51am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/2 "2018-11-20T05:51:04Z")

</div>

hi Gautham,

yes, you can index 2lakh+ records(bulk data) and there is no restriction.  
if you could provide input code it will be easy to identify the problem and give solution.

Regards,  
Balu

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 20, 2018, 7:19am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/3 "2018-11-20T07:19:02Z")

</div>

hi @balumurari1 here is the config file

```
input {
  http_poller {
    urls => {
      url => "https://demo1.service-now.com/api/now/table/incident?sysparm_display_value=true&sysparm_exclude_reference_link=True&sysparm_fields=number%2Ccategory%2Cpriority%2Cstate%2Cassignment_group%2Cassigned_to%2Cchild_incidents%2Cclose_code%2Cclosed_at%2Cclosed_by%2Ccompany%2Ccmdb_ci%2Ccontact_type%2Csys_created_on%2Csys_created_by%2Cdescription%2Cescalation%2Cimpact%2Cknowledge%2Cproblem_id%2Creassignment_count%2Creopen_count%2Cresolved_at%2Cseverity%2Curgency%2Ccaller_id.location.latitude%2Ccaller_id.location.longitude"
    }
    request_timeout => 60
proxy => { host => "1.1.1.2" port => "9090" scheme => "http"}
    user => "G435421"
    password => " *******"
    schedule => { cron => "* * * * *"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}
filter
       {
         split
                 {
                 field => "result"
                 }
  }
}
output {
  elasticsearch {
    hosts => ["1.1.1.3:9200"]
    index => "servicenow"
  }
#stdout { codec => rubydebug }
} 

```

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 20, 2018, 2:07pm UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/4 "2018-11-20T14:07:18Z")

</div>

What happens if you increase the request timeout?

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 21, 2018, 6:50am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/5 "2018-11-21T06:50:02Z")

</div>

@Eniqmatic even after changing, only 10k documents are getting indexed out of 2lakh documents.

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [November 21, 2018, 6:55am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/6 "2018-11-21T06:55:19Z")

</div>

probably, it is taking more time to get the data from the api, which is causing to reach your timeout.  
How much timeout have you specified in your input code?

---

<div class="post-metadata">

**Author:** ![Gauti](https://avatars.discourse-cdn.com/v4/letter/g/cdc98d/32.png) [@Gauti](https://discuss.elastic.co/u/Gauti)\
**Post date:** [November 21, 2018, 7:00am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/7 "2018-11-21T07:00:58Z")

</div>

@balumurari1 Now i have give 600, but still no change, in actual it is indexing the same 10k records again and again.

Is there anything i need to do with the shards allocation or something like that?

Thanks  
Gauti

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2018, 7:01am UTC](https://discuss.elastic.co/t/bulk-data-upload-in-logstash/156804/8 "2018-12-19T07:01:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.