# ElasticSerach java client or ElasticSerach java plugin for enrich all document at index time? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 30, 2013, 10:58am UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170 "2013-10-30T10:58:11Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![stefano\_ruggiero](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@stefano\_ruggiero](https://discuss.elastic.co/u/stefano_ruggiero)\ **Post date:** [October 30, 2013, 10:58am UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/1 "2013-10-30T10:58:11Z")
hi all, iam trying to create a new custom node or plugin that listen to all stream of data in ES cluster and then if a particular document satisfy my condition add a specific field or something similar. How is the best approach? there are some sort of examples? i try to explain better what is my goal: 1. i would like to have a document parser job inside the ES input data stream. 2. i would like to chose if that document has a particular sintax then if yes add custom field or better invoke an other class that store some information over the time. 3. doing this in async way, i mean dont stop the indexing process but only read and parse document while indexing. have you some suggestions on where to start ? iam at very initial point on decide if it is better to create a plugin or this can be done with java Client API given from ES. ( i expect that there will be an ovveride action like onParse(parse doc) that let me read all the flow of data.. ) Thanks for all your time and attention, i will appreciate all type of help. Regards. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 30, 2013, 11:19am UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/2 "2013-10-30T11:19:53Z")
There are some aspects. First, ES is distributed. That means, it is very cumbersome to chase docs while they are on their way from the client to the Lucene index (the shard). There is a built-in chasing. ES provides a perculator mechanism: you can register queries, and given a doc to index, clients can find out what query has matched. This works similar to your condition detection scenario. Second, it would be much easy to have full control over the ingest if you are after filtering docs that are going to be indexed. If there is only one client, or only one doc source, you should be able to filter the docs for the condition with little effort, outside ES, also taking load from the cluster for this job. And third, if you prefer parsing documents for word detection, look at the basic Lucene methods. Lucene analyzes token streams for doc fields to generate the tokens that finally are getting passed to the index. So maybe you want to focus on the field level to examine words. ES uses the source JSON just as a handy mechanism to create a list of fields for a Lucene doc. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![stefano\_ruggiero](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@stefano\_ruggiero](https://discuss.elastic.co/u/stefano_ruggiero)\ **Post date:** [October 30, 2013, 11:31am UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/3 "2013-10-30T11:31:36Z")
Hi jorg, thanks for your clarification. but consider this scenario where all the features build in ES could not be applied: i would like to parse every document at index time ( yes, only document that i send to that specific node.. ), if there is a document with RU geolocation then invoke a class ( my cusotm one ) for storing this information in a hashmap, then go ahead with process... after 3 hours i get another document with RU geolocation and some others information i would like to add this to the same key of hashmap, then if the value associated with that key is \> then 3 in 3 hours drop all other RU document or index a new document with this information or tell me this behaviour... is it possible in some way by hook into ES indexing action ? have you some suggestions ? yes my goal is something like to correlate my document with same field and then give me the answer.. an extension of percolator ( that how it is implemented in ES is useless for me... ) > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 30, 2013, 2:54pm UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/4 "2013-10-30T14:54:33Z")
Not sure where your docs come from, a stream maybe. It looks like time-based data series. If you have only one geolocation (RU) you are after, it is straightforward. Index all documents on a time-based scale with timestamps and the geo location as a geo field. Then you can periodically fire a geo distance filter query. Join the hits, sort them by timestamp, throw away the docs which fall outside your range. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![stefano\_ruggiero](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@stefano\_ruggiero](https://discuss.elastic.co/u/stefano_ruggiero)\ **Post date:** [October 30, 2013, 3:05pm UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/5 "2013-10-30T15:05:25Z")
yes that is exaclty what i have to do! but i need to do it in real time ( i cant do 100 search per seconds, 100 search = 100 different geo field ) , thats way iam asking for a custom node or plugin let me hook in indexing doc procress. Stefano Il giorno mercoledì 30 ottobre 2013 15:54:33 UTC+1, Jörg Prante ha scritto: > Not sure where your docs come from, a stream maybe. It looks like > time-based data series. > > If you have only one geolocation (RU) you are after, it is > straightforward. Index all documents on a time-based scale with timestamps > and the geo location as a geo field. Then you can periodically fire a geo > distance filter query. Join the hits, sort them by timestamp, throw away > the docs which fall outside your range. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 30, 2013, 3:31pm UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/6 "2013-10-30T15:31:43Z")
If you have 100 geo locations, my suggestion is like this. Core of the solution is the translation of a doc with geo location to a geo name which is used as field content and retrieved by the percolator. - register 100 geolocation queries to the percolator. The name of the registered query is equivalent to the content of the target field (ideally a geo location name) - use a client to fire documents, the docs get auto timestamped, and have a geo location. Run them against the percolator API. - use the geo location name obtained from the percolator API to retrieve an old doc, add info (geo location name plus other info), write new doc - check for index alias API. It might be useful to set up virtual indexes. So, for example, each geo name could be exposed as an ES index to the API but you only have one concrete index for 100 geo locations. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![stefano\_ruggiero](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@stefano\_ruggiero](https://discuss.elastic.co/u/stefano_ruggiero)\ **Post date:** [October 30, 2013, 4:45pm UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/7 "2013-10-30T16:45:58Z")
exactly what i have done, but it is too much performance intensive and on 10+ gb of data per day its not scale, if i would like to check for more days my percolator it is a real pain with logstash sintax, that's why we are trying to understand how to do it in a built in way ( with Java client node or a plugin ),because we need to solve the problem of real time analysis, unfortunally percolator its just another way to search document and it isnent really integrated with logstash... Stefano Il giorno mercoledì 30 ottobre 2013 16:31:43 UTC+1, Jörg Prante ha scritto: > If you have 100 geo locations, my suggestion is like this. Core of the > solution is the translation of a doc with geo location to a geo name which > is used as field content and retrieved by the percolator. > > - register 100 geolocation queries to the percolator. The name of the > registered query is equivalent to the content of the target field (ideally > a geo location name) > - use a client to fire documents, the docs get auto timestamped, and have > a geo location. Run them against the percolator API. > - use the geo location name obtained from the percolator API to retrieve > an old doc, add info (geo location name plus other info), write new doc > - check for index alias API. It might be useful to set up virtual indexes. > So, for example, each geo name could be exposed as an ES index to the API > but you only have one concrete index for 100 geo locations. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 30, 2013, 6:43pm UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/8 "2013-10-30T18:43:47Z")
It will not help you immediately (unless you want early code testing), but there are known scaling issues with percolator in the 0.90 version which will be fixed in 1.0 > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![krushnat\_khawale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krushnat_khawale/32/6652_2.png) [@krushnat\_khawale](https://discuss.elastic.co/u/krushnat_khawale)\ **Post date:** [February 11, 2016, 6:38am UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/9 "2016-02-11T06:38:46Z")
As I am creating a java web application without maven, I need jar for creating java client. Where can I find jars? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 11, 2016, 7:03am UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/10 "2016-02-11T07:03:15Z")
Maven central is the best place IMO. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:17pm UTC](https://discuss.elastic.co/t/elasticserach-java-client-or-elasticserach-java-plugin-for-enrich-all-document-at-index-time/14170/11 "2017-07-05T23:17:19Z")