# Problems with plugins and index templates **URL:** \ **Category:** Elasticsearch\ **Created:** [January 11, 2013, 9:36am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313 "2013-01-11T09:36:21Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Klaus\_Brunner](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Klaus\_Brunner](https://discuss.elastic.co/u/Klaus_Brunner)\ **Post date:** [January 11, 2013, 9:36am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/1 "2013-01-11T09:36:21Z")
Hi, we've been trying to switch some of our index creation from programmatic to index-template-based. This seems to work fine with one exception: plugins do not seem to work (we're using phonetic in this case, ES version is 0.20.2): org.elasticsearch.index.mapper.MapperParsingException: Analyzer [name\_metaphone] not found for field [metaphone] at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:86) at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:136) at org.elasticsearch.index.mapper.multifield.MultiFieldMapper$TypeParser.parse(MultiFieldMapper.java:132) at org.elasticsearch.index.mapper.object.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:218) at org.elasticsearch.index.mapper.object.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:204) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:691) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:575) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:451) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:486) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:430) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:318) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:533) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:431) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) java.lang.NullPointerException at org.elasticsearch.action.bulk.BulkShardRequest.beforeLocalFork(BulkShardRequest.java:67) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.retry(TransportShardReplicationOperationAction.java:484) ... Adding the settings and mapping JSON explicitly on index creation (we're using the Java API for that) has been working fine for quite some time. Now when we create a unified configuration in an index template JSON, it works OK when you create the index. But ES reproducibly fails once the first indexing operation hits a field that uses the plugin-provided analyzer. The index template consists of the same settings and mappings JSONs that we used before. The phonetic plugin _is_ installed correctly and logged on ES startup. Any ideas, suggestions? Klaus -- ---
**Author:** ![Klaus\_Brunner](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Klaus\_Brunner](https://discuss.elastic.co/u/Klaus_Brunner)\ **Post date:** [January 11, 2013, 10:08am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/2 "2013-01-11T10:08:45Z")
Addendum: We just tried to switch back to 0.19.11 - same issue there. Klaus -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 13, 2013, 9:28am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/3 "2013-01-13T09:28:56Z")
Hi Klaus, your problem has nothing to do with plugins or index templates but with bulk indexing, or with your data. The cause is a NPE in traversing the bulk item requests on shard level. There are two reasons, maybe you managed to add a null item to the bulk requests (which seems very unlikely because the bulk API does not endorse it) or you are running multiple threads on bulk indexing and they interfere. Because the ActionRequest list in BulkRequest is not threadsafe, you need to encapsulate the bulk indexing in a threadsafe manner by yourself when you reuse a single client by many threads. Jörg -- ---
**Author:** ![Klaus\_Brunner](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Klaus\_Brunner](https://discuss.elastic.co/u/Klaus_Brunner)\ **Post date:** [January 13, 2013, 6:57pm UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/4 "2013-01-13T18:57:03Z")
On Sunday, 13 January 2013 10:28:56 UTC+1, Jörg Prante wrote: > The cause is a NPE in traversing the bulk item requests on shard level. > There are two reasons, maybe you managed to add a null item to the bulk > requests (which seems very unlikely because the bulk API does not endorse > it) Nope. No null items. We've been indexing the same data, using the same indexing code, for more than half a year now. Many times, many items. No issues whatsoever. The problem only occurs once we switch the way we _create_ the indexes. The code to add items to an index is unchanged. > or you are running multiple threads on bulk indexing and they interfere. > Because the ActionRequest list in BulkRequest is not threadsafe, you need > to encapsulate the bulk indexing in a threadsafe manner by yourself when > you reuse a single client by many threads. This is interesting. Yes, we're doing bulk indexing with several threads (dozens) on the same client object, using a per-thread pattern like this: void indexStuff(...) { final BulkRequestBuilder bulkRequest = client.prepareBulk(); bulkRequest.add(....); final BulkResponse bulkResponse = bulkRequest.execute().actionGet(); } Again, so far this has worked perfectly. Now the idea that this wouldn't be thread-safe is a bit unsettling, also because it's not really obvious from any documentation, and it makes me wonder what else isn't thread-safe on the Java API? "Encapsulating in a thread-safe manner" would effectively kill indexing performance for us, so our only alternative would be one Client object per thread. Is this the recommended way of doing things? Or am I misinterpreting you and your point is that a BulkRequest mustn't be shared by multiple threads? We're not doing that at all, as shown in the simplified code snippet above. The whole thing simply doesn't quite look to me like a threading related issue - it happens reproducibly and immediately when we use the templates, and never happens without them. Thanks Klaus -- ---
**Author:** ![Klaus\_Brunner](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Klaus\_Brunner](https://discuss.elastic.co/u/Klaus_Brunner)\ **Post date:** [January 16, 2013, 2:48pm UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/5 "2013-01-16T14:48:58Z")
I've gisted a complete curl recreation at [https://gist.github.com/4547457](https://gist.github.com/4547457) - any hints appreciated. Klaus -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [January 16, 2013, 7:32pm UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/6 "2013-01-16T19:32:31Z")
"index": {...} is missing between "settings" and "analysis". Your template should look like this: { "asset\_template": { "template": "\*", "settings": { - ``` "index": {* "analysis": { ``` On Wednesday, January 16, 2013 9:48:58 AM UTC-5, Klaus Brunner wrote: > I've gisted a complete curl recreation at [Elasticsearch issue with phonetic plugin and index templates (stored in config). Tested with elasticsearch-analysis-phonetic 1.1.0 and 1.2.0 on both Linux and Windows on ES 0.19.12 and 0.20.2. Summary: it works without using a template, it works with a template added through the template API, but consistently fails with a template that is stored as a file in the configuration folder. The template per se is found and used, but the analyzer/filter provided by the plugin does not work. · GitHub](https://gist.github.com/4547457) - > any hints appreciated. > > Klaus -- ---
**Author:** ![Klaus\_Brunner](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Klaus\_Brunner](https://discuss.elastic.co/u/Klaus_Brunner)\ **Post date:** [January 17, 2013, 12:38pm UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/7 "2013-01-17T12:38:50Z")
On Wednesday, 16 January 2013 20:32:31 UTC+1, Igor Motov wrote: > "index": {...} is missing between "settings" and "analysis". Your template > should look like this: Yay! Thanks Igor, that was indeed it. I was a little surprised though that this kind of problem isn't caught earlier or at least with a more helpful error message. But I understand the settings JSON is basically free-form. Klaus -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 18, 2013, 12:01am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/8 "2013-01-18T00:01:03Z")
Note, the bulk items are collected in a simple array list in the BulkRequest, this is not threadsafe. So, each thread should handle its own BulkRequest (and I understand you do so). I use a modified BulkRequestBuilder with Queues.newConcurrentLinkedQueue() from Guava instead of an array list which can be shared by many threads (more than 3000 docs per second throughput with 12 threads on a TransportClient singleton, each doc ~1-10k). The difference is, only a single flush() call is needed to clear the queue which makes parallel bulk indexing more manageable in case of abort or client wait and shutdown. Jörg -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 18, 2013, 7:05am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/9 "2013-01-18T07:05:35Z")
BulkRequest shared by many threads so you can fill it even faster? Nice! Jörg? Did you send a pull request about it? Regards -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 18 janv. 2013 à 01:01, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) a écrit : > Note, the bulk items are collected in a simple array list in the BulkRequest, this is not threadsafe. So, each thread should handle its own BulkRequest (and I understand you do so). > > I use a modified BulkRequestBuilder with Queues.newConcurrentLinkedQueue() from Guava instead of an array list which can be shared by many threads (more than 3000 docs per second throughput with 12 threads on a TransportClient singleton, each doc ~1-10k). The difference is, only a single flush() call is needed to clear the queue which makes parallel bulk indexing more manageable in case of abort or client wait and shutdown. > > Jörg > > -- -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 18, 2013, 1:30pm UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/10 "2013-01-18T13:30:29Z")
I doubt it's faster - ConcurrentLinkedQueue is a beast. I also use size() for catching the bulk size limit which is a big no-no. In result it comes with a penalty. I presume that parallel document construction with XContentBuilder is overweighing the extra concurrent bulk list management. But I feel Shay won't prefer bulk indexing with ConcurrentLinkedQueue. Anyway, I found my code does not work with 0.20 any longer so I need to update. The famous LMAX disruptor looks promising to experiment with, but is difficult to code and a memory hog. Maybe a simple blocking fifo batch queue will do, with reentrant locking instead of synchronization. Cheers, Jörg -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:55am UTC](https://discuss.elastic.co/t/problems-with-plugins-and-index-templates/10313/11 "2017-07-06T02:55:47Z")