# Options to index data in ES **URL:** \ **Category:** Elasticsearch\ **Created:** [November 5, 2013, 3:40pm UTC](https://discuss.elastic.co/t/options-to-index-data-in-es/14268 "2013-11-05T15:40:20Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![hari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari/32/1582_2.png) [@hari](https://discuss.elastic.co/u/hari)\ **Post date:** [November 5, 2013, 3:40pm UTC](https://discuss.elastic.co/t/options-to-index-data-in-es/14268/1 "2013-11-05T15:40:20Z")
Based on my understanding ES has 4 out of the box options to index data. although I understand in theory how they work. I'm wondering if someone has done some real world implementation and comparison on what works best considering huge volumes of data (100 K records per hour) in regular intervals that needs to be indexed and error handling is of importance in case indexing fails. curl -XPUT - This is perhaps the simplest way to index a document, you just perform a PUT on a REST endpoint, this is best seen as an option during development to index documents for testing quickly. HTTP Bulk API - Push approach to index data, if you have an external application that consolidates the data in a timely manner and then formats it to JSON to be indexed. This is much more reliable as compared to UDP bulk import as you get an acknowledgement of index operation and can take corrective steps based on the response. UDP Bulk API - Connectionless datagram protocol. This is faster but not so reliable E.g. cat bulk.txt | nc -w 0 -u localhost 9700 River plugin - Pull approach, runs within ES node and can pull data from any datasource. Can be used when we are expecting a constant change of data that needs to be indexed and we don't want to write another external application to push data into ES for indexing. River plugin also supports import using Bulk API, this is usefull in cases where the river plugin wants to accumulate the data for certain threshold before performing an import / indexing. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [November 5, 2013, 3:52pm UTC](https://discuss.elastic.co/t/options-to-index-data-in-es/14268/2 "2013-11-05T15:52:05Z")
On Tue, Nov 5, 2013 at 10:40 AM, Hariharan Vadivelu [hariinfo@gmail.com](mailto:hariinfo@gmail.com)wrote: HTTP Bulk API - Push approach to index data, if you have an external > application that consolidates the data in a timely manner > and then formats it to JSON to be indexed. This is much more reliable as > compared to UDP bulk import as you get an acknowledgement > of index operation and can take corrective steps based on the response. We have pretty good success with this. We're seeing ~600 big documents per second using multi process producers with a reasonably tiny Elasticsearch cluster. We like push over pull because it gives us a lot of control and we have good tools around other push processes. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 5, 2013, 4:09pm UTC](https://discuss.elastic.co/t/options-to-index-data-in-es/14268/3 "2013-11-05T16:09:32Z")
How big are the records? 100k records per hour are not much, if a record is just 1k average size. You have forgotten the Java TransportClient bulk indexing method, it's the method I prefer: you can connect from remote, you can connect to cluster nodes you configure, you can index with multiple threads, and you save a bit of HTTP overhead by using the native ES protocol. With the Java TransportClient, I can saturate the network interface (~10-11 MB/sec) for hours while indexing... Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:08am UTC](https://discuss.elastic.co/t/options-to-index-data-in-es/14268/4 "2017-07-06T02:08:46Z")
Closed on July 6, 2017, 2:08am UTC