# ANN - Announcing Surfiki Refine - Map-Reduce targeting Elasticsearch Indices **URL:** \ **Category:** Elasticsearch\ **Created:** [June 12, 2013, 10:50pm UTC](https://discuss.elastic.co/t/ann-announcing-surfiki-refine-map-reduce-targeting-elasticsearch-indices/12391 "2013-06-12T22:50:50Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![AnthonyNystrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anthonynystrom/32/1066_2.png) [@AnthonyNystrom](https://discuss.elastic.co/u/AnthonyNystrom)\ **Post date:** [June 12, 2013, 10:50pm UTC](https://discuss.elastic.co/t/ann-announcing-surfiki-refine-map-reduce-targeting-elasticsearch-indices/12391/1 "2013-06-12T22:50:50Z")
GITHUB PAGE LINK [http://intridea.github.io/surfiki-refine-elasticsearch/](http://intridea.github.io/surfiki-refine-elasticsearch/) What is Refine? Originally designed to work with Elasticsearch indices and API. Refine allows you to take an input stream, transform and or manipulate it for your needs, then output it in a variety of formats. With the variety of uses of Elasticsearch being seen in the its ecosystem (Search, Data Stores, API), a powerful and extensible processing tier was needed to meet Intridea’s needs. Our belief, is this can fit in to your Elasticsearch environment as well. And in turn, provide you with additional transformation and or manipulation capabilities. At its heart, Refine is a collection of scripts, written by you in Python[http://www.python.org/](http://www.python.org/) . How does it work? Technically, Refine is Map-Reduce, which for each job you define incorporates: - A stream script defines the source of your data. A query’s results (In the case of Elasticsearch) A file or files, or a Web API such as Twitter. - - A mapper script takes the input, divides it into smaller sub-problems, and distributes them to workers. A worker may do this again in turn, leading to a multi-level tree structure. The worker processes the smaller problem, and passes the answer back to its master. - - A reducer script collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve. - - You can add as many scripts that your job may require, however the core is the stream, mapper and reducer scripts. What are some examples? - Combining data from multiple indices in to a new index. With Refine you can either do this ad hoc or on a scheduled basis. - - Splitting data from a single index in to new indices based upon some criteria. Refine makes this relatively simple to perform. - - Statistical facets can be taxing on the heap. You can use Refine to process/transform/count this data and expose within a new index on a set schedule. - - Transforming data already indexed with additional data. For example, some data has location information (lat, long) however you also want to store meta data associated with those locations Such as State, City and County information. While it makes sense to do some of this inline, with Refine you can do it after the initial data is already indexed. - - Accessing third party API's to append data to existing indexed data. With Refine, accessing those API’s, injecting and or manipulating is a core feature. - - Given a body of text, you can use Refine to split your stream into its separate words, perform a word count for the given word, then reduce it to a de-duped sorted array. - - Given a collection of results, you can use Refine to further score each result against a series of validation criteria such as geographic region or category. - - Need text to appear in various ways, before it can be processed further? Use Refine to prepend suffixes or concatenate strings, and dynamically add these as new attributes. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:31am UTC](https://discuss.elastic.co/t/ann-announcing-surfiki-refine-map-reduce-targeting-elasticsearch-indices/12391/2 "2017-07-06T02:31:29Z")
Closed on July 6, 2017, 2:31am UTC