# Unable to load large set of records from a table in Servicenow into Elasticsearch using Logstash **URL:** \ **Category:** Logstash\ **Created:** [February 6, 2020, 5:05am UTC](https://discuss.elastic.co/t/unable-to-load-large-set-of-records-from-a-table-in-servicenow-into-elasticsearch-using-logstash/218085 "2020-02-06T05:05:27Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![ramypala](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@ramypala](https://discuss.elastic.co/u/ramypala)\ **Post date:** [February 6, 2020, 5:05am UTC](https://discuss.elastic.co/t/unable-to-load-large-set-of-records-from-a-table-in-servicenow-into-elasticsearch-using-logstash/218085/1 "2020-02-06T05:05:27Z")
Hi Team, I am trying to load Servicenow tables in Elasticsearch using Logstash HTTP Poller Plugin.I am able to load a table with 42 records into Elasticsearch index without any issue.But when I tried to load a table with 8000+ and 10000+ records I am getting below error.I am attaching the error and config file for your reference.Please help. Config File: input { http\_poller { urls =\> { url =\> "[https://dev63285.service-now.com/incident\_metric\_list.do?JSONv2&display\_value=True&sysparm\_exclude\_reference\_link=True&sysparm\_fields=number%2Cstate%2Copened\_at%2Cclosed\_at%2Cpriority%2Cassigned\_to%2Cassignment\_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm\_limit=1sysparm\_view=json\_view](https://dev63285.service-now.com/incident_metric_list.do?JSONv2&display_value=True&sysparm_exclude_reference_link=True&sysparm_fields=number%2Cstate%2Copened_at%2Cclosed_at%2Cpriority%2Cassigned_to%2Cassignment_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm_limit=1sysparm_view=json_view)" } connect\_timeout =\> 1000000 request\_timeout =\> 900000 socket\_timeout =\> 180 user =\> "admin" password =\> "Snow2020\*" schedule =\> { cron =\> "\* \* \* \* \*"} codec =\> "json" metadata\_target =\> "http\_poller\_metadata" } } filter { split { field =\> "records" } } output { elasticsearch { hosts =\> ["localhost:9200"] index =\> "servicenowinc" document\_id =\> "%{[records][inc\_number]}" } stdout { codec =\> rubydebug } } Error in Logstash: Sending Logstash logs to C:/Elastic\_Stack/logstash-7.5.1/logs which is now configured via log4j2.properties [2020-02-05T17:21:48,560][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-02-05T17:21:48,724][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.5.1"} [2020-02-05T17:21:54,137][INFO][org.reflections.Reflections] Reflections took 155 ms to scan 1 urls, producing 20 keys and 40 values [2020-02-05T17:21:58,405][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}} [2020-02-05T17:21:59,020][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} [2020-02-05T17:21:59,141][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es\_version=\>7} [2020-02-05T17:21:59,151][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7} [2020-02-05T17:21:59,252][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]} [2020-02-05T17:21:59,353][INFO][logstash.outputs.elasticsearch][main] Using default mapping template [2020-02-05T17:21:59,430][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster\_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2020-02-05T17:21:59,444][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, "pipeline.sources"=\>["C:/Elastic\_Stack/logstash-7.5.1/bin/logstash\_incident\_test2.conf"], :thread=\>"#\"} [2020-02-05T17:21:59,563][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s", "number\_of\_shards"=\>1}, "mappings"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}} [2020-02-05T17:21:59,640][INFO][logstash.inputs.http\_poller][main] Registering http\_poller Input {:type=\>nil, :schedule=\>{"cron"=\>"\* \* \* \* \*"}, :timeout=\>nil} [2020-02-05T17:21:59,686][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=\>"main"} [2020-02-05T17:22:00,075][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} C:/Elastic\_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated [2020-02-05T17:22:02,407][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2020-02-05T17:28:25,304][WARN][logstash.filters.split][main] Only String and Array types are splittable. field:records is of type = NilClass C:/Elastic\_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/awesome\_print-1.7.0/lib/awesome\_print/formatters/base\_formatter.rb:31: warning: constant ::Fixnum is deprecated { "@version" =\> "1", "@timestamp" =\> 2020-02-05T11:58:25.101Z, "http\_poller\_metadata" =\> { "runtime\_seconds" =\> nil, "request" =\> { "method" =\> "get", "url" =\> "[https://dev63285.service-now.com/incident\_metric\_list.do?JSONv2&display\_value=True&sysparm\_exclude\_reference\_link=True&sysparm\_fields=number%2Cstate%2Copened\_at%2Cclosed\_at%2Cpriority%2Cassigned\_to%2Cassignment\_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm\_limit=1sysparm\_view=json\_view](https://dev63285.service-now.com/incident_metric_list.do?JSONv2&display_value=True&sysparm_exclude_reference_link=True&sysparm_fields=number%2Cstate%2Copened_at%2Cclosed_at%2Cpriority%2Cassigned_to%2Cassignment_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm_limit=1sysparm_view=json_view)" }, "host" =\> "DESKTOP-JRAAHA2", "name" =\> "url" }, "tags" =\> [ [0] "\_http\_request\_failure", [1] "\_split\_type\_failure" ], "http\_request\_failure" =\> { "backtrace" =\> [ [0] "C:/Elastic\_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:122:in `body'", [1] "C:/Elastic_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:162:in `handle\_success'", [2] "C:/Elastic\_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-http\_poller-5.0.1/lib/logstash/inputs/http\_poller.rb:154:in `block in request_async'", [3] "C:/Elastic_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:268:in `handleResponse'", [4] "C:/Elastic\_Stack/logstash-7.5.1/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:50:in `call'" ], "runtime\_seconds" =\> 181.838, "request" =\> { "method" =\> "get", "url" =\> "[https://dev63285.service-now.com/incident\_metric\_list.do?JSONv2&display\_value=True&sysparm\_exclude\_reference\_link=True&sysparm\_fields=number%2Cstate%2Copened\_at%2Cclosed\_at%2Cpriority%2Cassigned\_to%2Cassignment\_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm\_limit=1sysparm\_view=json\_view](https://dev63285.service-now.com/incident_metric_list.do?JSONv2&display_value=True&sysparm_exclude_reference_link=True&sysparm_fields=number%2Cstate%2Copened_at%2Cclosed_at%2Cpriority%2Cassigned_to%2Cassignment_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm_limit=1sysparm_view=json_view)" }, "error" =\> "Could not read from stream: Connection reset", "name" =\> "url" } } [2020-02-05T17:32:36,209][WARN][logstash.filters.split][main] Only String and Array types are splittable. field:records is of type = NilClass Many Thanks, Ramya ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 6, 2020, 5:00pm UTC](https://discuss.elastic.co/t/unable-to-load-large-set-of-records-from-a-table-in-servicenow-into-elasticsearch-using-logstash/218085/2 "2020-02-06T17:00:31Z")
The split failure is occurring because the event does not contain a [records] field. You have an \_http\_request\_failure tag. I see that [runtime\_seconds] is just over three minutes, and that the http endpoint reset the connection. Perhaps it is a timeout. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 5, 2020, 5:00pm UTC](https://discuss.elastic.co/t/unable-to-load-large-set-of-records-from-a-table-in-servicenow-into-elasticsearch-using-logstash/218085/3 "2020-03-05T17:00:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.