# Corrupted search using 0.11 Smile format **URL:** \ **Category:** Elasticsearch\ **Created:** [September 27, 2010, 4:08pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373 "2010-09-27T16:08:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [September 27, 2010, 4:08pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/1 "2010-09-27T16:08:39Z")
Submitting the following JSON via the Java jsonBuilder works correctly. Submitting the same content through the smileBuilder causes an exception, but the content is still indexed, in a manner than is not searchable. java.lang.ArrayIndexOutOfBoundsException: 1097 at org.elasticsearch.common.Unicode.UTF8toUTF16(Unicode.java: 188) at org.elasticsearch.common.Unicode.unsafeFromBytesAsUtf16(Unicode.java: 104) at org.elasticsearch.common.Unicode.fromBytes(Unicode.java:78) at org.elasticsearch.common.xcontent.XContentBuilder.string(XContentBuilder.java: 535) Search error: error: "JsonParseException[Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: org.elasticsearch.common.io.stream.LZFStreamInput@41141325; line: 1, column: 2]]" } I uploaded the binary json file that causes the error: [http://dl.dropbox.com/u/12095883/532-WSMarketCommentary\_15884843-20100927111516.json](http://dl.dropbox.com/u/12095883/532-WSMarketCommentary_15884843-20100927111516.json) Please let me know if there is any other items we can provide. Most documents go through without issue. I can't see anything in this document which looks suspect. David {"indexid":"index21","documentkey":"532- UMSACT0520100927113431BIDU-20100927113431","subtransid":"3072748101","dmsourceid":"STDPOOR","documentdate":"2010-09-27 1 5:34:31.000","\_\_documentdate":"2010-09-27T15:34:31.000Z","documenttag":"UMSACT0520100927113431BIDU","documenttype":"ViewsNews","feedid":"532","versiontag":"2010 0927113431","author":"Matthew Morrow","bridgesymbols":"| US;BIDU","headline":"PACIFIC CREST RAISES BAIDU (BIDU) ESTIMATES, TARGET, KEEPS OUTPERFORM","productcate gory":"ViewsNews","publicationdatetime":"2010-09-27 11:34:31.000","source":"SNPMarketScope","teaser":"Analyst Steve Weinstein tells salesforce believes current nbusiness momentum at BIDU reflects the benefit of Phoenix Nest \nadvertising system and renewed sales effort. Says benefits from \nthese drivers are still at an early stage and should continue\nwell through 2011. Also believes that 2011 and2012 will be the\nyear e-commerce reaches critical mass in China and merchants\n begin to embrace search marketing, providing another leg of\ngrowth. Raises $1.35 '10 EPS estimate to $1.37, $2.02 '11 to\n$2.29. Also raises $80 price target t o $140./Morrow","wsodcompany":"|89117","wsodissue":"|2837269" ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2010, 5:14pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/2 "2010-09-27T17:14:40Z")
Do you have more stack trace other than the one that ends with string(...)? I wonder who calls that string(..) as it does not really apply to the smile format... On Mon, Sep 27, 2010 at 6:08 PM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > Submitting the following JSON via the Java jsonBuilder works > correctly. Submitting the same content through the smileBuilder causes > an exception, but the content is still indexed, in a manner than is > not searchable. > > java.lang.ArrayIndexOutOfBoundsException: 1097 > at org.elasticsearch.common.Unicode.UTF8toUTF16(Unicode.java: > 188) > at > org.elasticsearch.common.Unicode.unsafeFromBytesAsUtf16(Unicode.java: > 104) > at org.elasticsearch.common.Unicode.fromBytes(Unicode.java:78) > at > > org.elasticsearch.common.xcontent.XContentBuilder.string(XContentBuilder.java: > 535) > > Search error: > error: "JsonParseException[Unexpected character (':' (code 58)): > expected a valid value (number, String, array, object, 'true', 'false' > or 'null') at [Source: > org.elasticsearch.common.io.stream.LZFStreamInput@41141325; line: 1, > column: 2]]" > } > > I uploaded the binary json file that causes the error: > > [http://dl.dropbox.com/u/12095883/532-WSMarketCommentary\_15884843-20100927111516.json](http://dl.dropbox.com/u/12095883/532-WSMarketCommentary_15884843-20100927111516.json) > > Please let me know if there is any other items we can provide. Most > documents go through without issue. I can't see anything in this > document which looks suspect. > > David > > {"indexid":"index21","documentkey":"532- > > UMSACT0520100927113431BIDU-20100927113431","subtransid":"3072748101","dmsourceid":"STDPOOR","documentdate":"2010-09-27 > 1 > > 5:34:31.000","\_\_documentdate":"2010-09-27T15:34:31.000Z","documenttag":"UMSACT0520100927113431BIDU","documenttype":"ViewsNews","feedid":"532","versiontag":"2010 > 0927113431","author":"Matthew Morrow","bridgesymbols":"| > US;BIDU","headline":"PACIFIC CREST RAISES BAIDU (BIDU) ESTIMATES, > TARGET, KEEPS OUTPERFORM","productcate > gory":"ViewsNews","publicationdatetime":"2010-09-27 > 11:34:31.000","source":"SNPMarketScope","teaser":"Analyst Steve > Weinstein tells salesforce believes current > nbusiness momentum at BIDU reflects the benefit of Phoenix Nest > \nadvertising system and renewed sales effort. Says benefits from > \nthese drivers are still at an > early stage and should continue\nwell through 2011. Also believes that > 2011 and2012 will be the\nyear e-commerce reaches critical mass in > China and merchants\n > begin to embrace search marketing, providing another leg of\ngrowth. > Raises $1.35 '10 EPS estimate to $1.37, $2.02 '11 to\n$2.29. Also > raises $80 price target t > o $140./Morrow","wsodcompany":"|89117","wsodissue":"|2837269" ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [September 27, 2010, 5:42pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/3 "2010-09-27T17:42:28Z")
I cut the stack trace where it was called from my code XContentBuilder content = XContentFactory.jsonBuilder().startObject(); series of: content.field(field, value); ... client.prepareIndex(index, type, getDocID()) .setSource(content).setOperationThreaded(false).execute() .actionGet(); java.lang.ArrayIndexOutOfBoundsException: 1097 at org.elasticsearch.common.Unicode.UTF8toUTF16(Unicode.java: 188) at org.elasticsearch.common.Unicode.unsafeFromBytesAsUtf16(Unicode.java: 104) at org.elasticsearch.common.Unicode.fromBytes(Unicode.java:78) at org.elasticsearch.common.xcontent.XContentBuilder.string(XContentBuil der.java:535) at com.wsod.dm.SubTran.submitDocument(SubTran.java:606) at com.wsod.dm.SubTran.processDoc(SubTran.java:399) at com.wsod.dm.SubTran.run(SubTran.java:418) at com.wsod.dm.Internal.BoundedExecutor $1.run(BoundedExecutor.java:28) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(Unknown Source ) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) On Sep 27, 11:14 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Do you have more stack trace other than the one that ends with string(...)? > I wonder who calls that string(..) as it does not really apply to the smile > format... > > On Mon, Sep 27, 2010 at 6:08 PM, dbenson [dben...@dbenson.net](mailto:dben...@dbenson.net) wrote: > > > Submitting the following JSON via the Java jsonBuilder works > > correctly. Submitting the same content through the smileBuilder causes > > an exception, but the content is still indexed, in a manner than is > > not searchable. > > > java.lang.ArrayIndexOutOfBoundsException: 1097 > > at org.elasticsearch.common.Unicode.UTF8toUTF16(Unicode.java: > > 188) > > at > > org.elasticsearch.common.Unicode.unsafeFromBytesAsUtf16(Unicode.java: > > 104) > > at org.elasticsearch.common.Unicode.fromBytes(Unicode.java:78) > > at > > > org.elasticsearch.common.xcontent.XContentBuilder.string(XContentBuilder.java: > > 535) > > > Search error: > > error: "JsonParseException[Unexpected character (':' (code 58)): > > expected a valid value (number, String, array, object, 'true', 'false' > > or 'null') at [Source: > > org.elasticsearch.common.io.stream.LZFStreamInput@41141325; line: 1, > > column: 2]]" > > } > > > I uploaded the binary json file that causes the error: > > > [http://dl.dropbox.com/u/12095883/532-WSMarketCommentary\_15884843-2010](http://dl.dropbox.com/u/12095883/532-WSMarketCommentary_15884843-2010)... > > > Please let me know if there is any other items we can provide. Most > > documents go through without issue. I can't see anything in this > > document which looks suspect. > > > David > > > {"indexid":"index21","documentkey":"532- > > > UMSACT0520100927113431BIDU-20100927113431","subtransid":"3072748101","dmsourceid":"STDPOOR","documentdate":"2010-09-27 > > 1 > > > 5:34:31.000","\_\_documentdate":"2010-09-27T15:34:31.000Z","documenttag":"UMSACT0520100927113431BIDU","documenttype":"ViewsNews","feedid":"532","versiontag":"2010 > > 0927113431","author":"Matthew Morrow","bridgesymbols":"| > > US;BIDU","headline":"PACIFIC CREST RAISES BAIDU (BIDU) ESTIMATES, > > TARGET, KEEPS OUTPERFORM","productcate > > gory":"ViewsNews","publicationdatetime":"2010-09-27 > > 11:34:31.000","source":"SNPMarketScope","teaser":"Analyst Steve > > Weinstein tells salesforce believes current > > nbusiness momentum at BIDU reflects the benefit of Phoenix Nest > > \nadvertising system and renewed sales effort. Says benefits from > > \nthese drivers are still at an > > early stage and should continue\nwell through 2011. Also believes that > > 2011 and2012 will be the\nyear e-commerce reaches critical mass in > > China and merchants\n > > begin to embrace search marketing, providing another leg of\ngrowth. > > Raises $1.35 '10 EPS estimate to $1.37, $2.02 '11 to\n$2.29. Also > > raises $80 price target t > > o $140./Morrow","wsodcompany":"|89117","wsodissue":"|2837269" ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2010, 5:47pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/4 "2010-09-27T17:47:11Z")
Then it seems like your code calls string(): SubTran.run(SubTran.java:418) on the smile builder, which is a binary format..., so it does not make sense to call string on it. I am a bit confused, you get the exception when you index, but then there is also the exception when searching. Do they correlate? On Mon, Sep 27, 2010 at 7:42 PM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > I cut the stack trace where it was called from my code > > XContentBuilder content = XContentFactory.jsonBuilder().startObject(); > > series of: > content.field(field, value); > ... > > client.prepareIndex(index, type, getDocID()) > > .setSource(content).setOperationThreaded(false).execute() > .actionGet(); > > java.lang.ArrayIndexOutOfBoundsException: 1097 > at org.elasticsearch.common.Unicode.UTF8toUTF16(Unicode.java: > 188) > at > org.elasticsearch.common.Unicode.unsafeFromBytesAsUtf16(Unicode.java: > 104) > at org.elasticsearch.common.Unicode.fromBytes(Unicode.java:78) > at > org.elasticsearch.common.xcontent.XContentBuilder.string(XContentBuil > der.java:535) > at com.wsod.dm.SubTran.submitDocument(SubTran.java:606) > at com.wsod.dm.SubTran.processDoc(SubTran.java:399) > at com.wsod.dm.SubTran.run(SubTran.java:418) > at com.wsod.dm.Internal.BoundedExecutor > $1.run(BoundedExecutor.java:28) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(Unknown Source > ) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > On Sep 27, 11:14 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Do you have more stack trace other than the one that ends with > > string(...)? > > I wonder who calls that string(..) as it does not really apply to the > > smile > > format... > > > > On Mon, Sep 27, 2010 at 6:08 PM, dbenson [dben...@dbenson.net](mailto:dben...@dbenson.net) wrote: > > > > > Submitting the following JSON via the Java jsonBuilder works > > > correctly. Submitting the same content through the smileBuilder causes > > > an exception, but the content is still indexed, in a manner than is > > > not searchable. > > > > > java.lang.ArrayIndexOutOfBoundsException: 1097 > > > at org.elasticsearch.common.Unicode.UTF8toUTF16(Unicode.java: > > > 188) > > > at > > > org.elasticsearch.common.Unicode.unsafeFromBytesAsUtf16(Unicode.java: > > > 104) > > > at org.elasticsearch.common.Unicode.fromBytes(Unicode.java:78) > > > at > > > > > > > org.elasticsearch.common.xcontent.XContentBuilder.string(XContentBuilder.java: > > > > 1. > > > > > Search error: > > > error: "JsonParseException[Unexpected character (':' (code 58)): > > > expected a valid value (number, String, array, object, 'true', 'false' > > > or 'null') at [Source: > > > org.elasticsearch.common.io.stream.LZFStreamInput@41141325; line: 1, > > > column: 2]]" > > > } > > > > > I uploaded the binary json file that causes the error: > > > > > [http://dl.dropbox.com/u/12095883/532-WSMarketCommentary\_15884843-2010](http://dl.dropbox.com/u/12095883/532-WSMarketCommentary_15884843-2010). > > > .. > > > > > Please let me know if there is any other items we can provide. Most > > > documents go through without issue. I can't see anything in this > > > document which looks suspect. > > > > > David > > > > > {"indexid":"index21","documentkey":"532- > > > > > > > UMSACT0520100927113431BIDU-20100927113431","subtransid":"3072748101","dmsourceid":"STDPOOR","documentdate":"2010-09-27 > > > > 1 > > > > > > > 5:34:31.000","\_\_documentdate":"2010-09-27T15:34:31.000Z","documenttag":"UMSACT0520100927113431BIDU","documenttype":"ViewsNews","feedid":"532","versiontag":"2010 > > > > 0927113431","author":"Matthew Morrow","bridgesymbols":"| > > > US;BIDU","headline":"PACIFIC CREST RAISES BAIDU (BIDU) ESTIMATES, > > > TARGET, KEEPS OUTPERFORM","productcate > > > gory":"ViewsNews","publicationdatetime":"2010-09-27 > > > 11:34:31.000","source":"SNPMarketScope","teaser":"Analyst Steve > > > Weinstein tells salesforce believes current > > > nbusiness momentum at BIDU reflects the benefit of Phoenix Nest > > > \nadvertising system and renewed sales effort. Says benefits from > > > \nthese drivers are still at an > > > early stage and should continue\nwell through 2011. Also believes that > > > 2011 and2012 will be the\nyear e-commerce reaches critical mass in > > > China and merchants\n > > > begin to embrace search marketing, providing another leg of\ngrowth. > > > Raises $1.35 '10 EPS estimate to $1.37, $2.02 '11 to\n$2.29. Also > > > raises $80 price target t > > > o $140./Morrow","wsodcompany":"|89117","wsodissue":"|2837269" ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [September 27, 2010, 6:12pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/5 "2010-09-27T18:12:38Z")
I'm not sure where the original exception was coming from, but I was causing a secondary exception when I attempted to log the binary json content. I apologize for the extra confusion. I'm no longer getting the exception when submitting the document, but it is causing the search to fail. I can reproduce it by deleting the index, recreating and resubmitting the same document. David ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2010, 6:16pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/6 "2010-09-27T18:16:31Z")
is there a chance that you can create a simple "main" program that recreates it? It will help resolving this faster. -shay.banon On Mon, Sep 27, 2010 at 8:12 PM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > I'm not sure where the original exception was coming from, but I was > causing a secondary exception when I attempted to log the binary json > content. I apologize for the extra confusion. > > I'm no longer getting the exception when submitting the document, but > it is causing the search to fail. I can reproduce it by deleting the > index, recreating and resubmitting the same document. > > David ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [September 27, 2010, 8:16pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/7 "2010-09-27T20:16:09Z")
I've been able to make a program to isolate this. I haven't as yet reduced out may be the extra fields... David yml: index: analysis : analyzer : verity\_tokenizer : type: pattern lowercase: true pattern: '(?:(?!\w).)+' stopwords: _none_ sortable\_tokenizer : type : keyword lowercase\_keyword : type : custom filter : [lowercase] tokenizer : keyword index creation: curl -XPUT '[http://localhost:9200/rebuild\_index21/](http://localhost:9200/rebuild_index21/)' -d ' index : number\_of\_shards : 1 number\_of\_replicas : 1 ' curl -XPUT '[http://localhost:9200/rebuild\_index21/index21/\_mapping](http://localhost:9200/rebuild_index21/index21/_mapping)' -d ' {"index21":{"dynamic":false,"date\_formats": ["date\_optional\_time"],"\_source":{"compress":true},"properties": {"author": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"bridgesymbols": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"fidelitycategory": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"fidelitymarketcategory": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"headline": {"type":"multi\_field","fields":{"headline": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sortable": {"type":"string","analyzer":"lowercase\_keyword","omit\_norms":true}}},"industries": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"industrygicscodes": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"priceclosing": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"pricecurrent": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"productcategory": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"productcode": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"publicationdatetime": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectorgicscodes": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectorindustries": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectorindustrycodes": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectors": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"source": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"storytext": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"teaser": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"dmsourceid": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"\_\_documentdate": {"type":"date","omit\_norms":true},"documentdate": {"type":"string","index":"no","omit\_norms":true},"documenttag": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"documenttype": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"feedid": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"versiontag": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"wsodcompany": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"wsodissue": {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"indexid": {"type":"string","analyzer":"lowercase\_keyword","omit\_norms":true,"omit\_term\_freq\_and\_positions":true},"documentkey": {"type":"string","analyzer":"lowercase\_keyword","omit\_norms":true,"omit\_term\_freq\_and\_positions":true},"subtransid": {"type":"long","include\_in\_all":false},"\_\_content": {"type":"string","index":"no","omit\_norms":true}}}} ' ## Code import static org.elasticsearch.node.NodeBuilder.nodeBuilder; import org.elasticsearch.client.Client; import org.elasticsearch.common.xcontent.XContentBuilder; import org.elasticsearch.common.xcontent.XContentFactory; import org.elasticsearch.node.Node; import org.elasticsearch.node.NodeBuilder; class Hack { ``` public static void main(String args[]) throws Exception { NodeBuilder nb = ``` nodeBuilder().client(true).clusterName("devcluster"); Node node = nb.node(); Client client = node.client(); ``` XContentBuilder content = ``` XContentFactory.smileBuilder().startObject(); ``` content.field("indexid", "index21"); content.field("documentkey", "532- ``` UMSACT0420100927111149CMCSA-20100927111149"); content.field("subtransid", "3072773936"); content.field("dmsourceid", "index21"); content.field("documentdate", "index21"); content.field("documenttag", "WSMarketCommentary\_15884843"); content.field("documenttype", "MarketCommentary"); content.field("feedid", "532"); content.field("versiontag", "20100927111516"); content.field("bridgesymbols", ""); content.field("headline", "S&P Market Commentary"); content.field("productcategory", "MarketCommentary"); content.field("publicationdatetime", "2010-09-27 11:15:16.000"); content.field("source", "SNPMarketScope"); content.field("teaser", ""); content.field("wsodcompany", ""); content.field("wsodissue", ""); ``` client.prepareIndex("rebuild_index21", "index21", "532- ``` UMSACT0420100927111149CMCSA-20100927111149") .setSource(content).setOperationThreaded(false).execute() .actionGet(); } } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2010, 8:33pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/8 "2010-09-27T20:33:40Z")
Ran this on my end, seems to work. Here is the updated Java client (I added an endObject call, though it seems to work without out, but just to be safe, and a search request to list all docs): [gist:599771 ยท GitHub](http://gist.github.com/599771). I basically started a single node, ran the curls against it, and then ran the client code. I am using latest master. -shay.banon On Mon, Sep 27, 2010 at 10:16 PM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > I've been able to make a program to isolate this. I haven't as yet > reduced out may be the extra fields... > > David > > yml: > index: > analysis : > analyzer : > verity\_tokenizer : > type: pattern > lowercase: true > pattern: '(?:(?!\w).)+' > stopwords: _none_ > sortable\_tokenizer : > type : keyword > lowercase\_keyword : > type : custom > filter : [lowercase] > tokenizer : keyword > > index creation: > curl -XPUT '[http://localhost:9200/rebuild\_index21/](http://localhost:9200/rebuild_index21/)' -d ' > index : > number\_of\_shards : 1 > number\_of\_replicas : 1 > ' > > curl -XPUT '[http://localhost:9200/rebuild\_index21/index21/\_mapping](http://localhost:9200/rebuild_index21/index21/_mapping)' -d > ' > {"index21":{"dynamic":false,"date\_formats": > ["date\_optional\_time"],"\_source":{"compress":true},"properties": > {"author": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"bridgesymbols": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"fidelitycategory": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"fidelitymarketcategory": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"headline": > {"type":"multi\_field","fields":{"headline": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sortable": > > {"type":"string","analyzer":"lowercase\_keyword","omit\_norms":true}}},"industries": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"industrygicscodes": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"priceclosing": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"pricecurrent": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"productcategory": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"productcode": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"publicationdatetime": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectorgicscodes": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectorindustries": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectorindustrycodes": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"sectors": > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"source": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"storytext": > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"teaser": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"dmsourceid": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"\_\_documentdate": > {"type":"date","omit\_norms":true},"documentdate": > {"type":"string","index":"no","omit\_norms":true},"documenttag": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"documenttype": > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"feedid": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"versiontag": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"wsodcompany": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"wsodissue": > > {"type":"string","analyzer":"verity\_tokenizer","omit\_norms":true},"indexid": > > {"type":"string","analyzer":"lowercase\_keyword","omit\_norms":true,"omit\_term\_freq\_and\_positions":true},"documentkey": > > {"type":"string","analyzer":"lowercase\_keyword","omit\_norms":true,"omit\_term\_freq\_and\_positions":true},"subtransid": > {"type":"long","include\_in\_all":false},"\_\_content": > {"type":"string","index":"no","omit\_norms":true}}}} > ' > > ## Code > > import static org.elasticsearch.node.NodeBuilder.nodeBuilder; > > import org.elasticsearch.client.Client; > import org.elasticsearch.common.xcontent.XContentBuilder; > import org.elasticsearch.common.xcontent.XContentFactory; > import org.elasticsearch.node.Node; > import org.elasticsearch.node.NodeBuilder; > > class Hack { > > ``` > public static void main(String args[]) throws Exception { > NodeBuilder nb = > > ``` > > nodeBuilder().client(true).clusterName("devcluster"); > Node node = nb.node(); > Client client = node.client(); > > ``` > XContentBuilder content = > > ``` > > XContentFactory.smileBuilder().startObject(); > > ``` > content.field("indexid", "index21"); > content.field("documentkey", "532- > > ``` > > UMSACT0420100927111149CMCSA-20100927111149"); > content.field("subtransid", "3072773936"); > content.field("dmsourceid", "index21"); > content.field("documentdate", "index21"); > content.field("documenttag", "WSMarketCommentary\_15884843"); > content.field("documenttype", "MarketCommentary"); > content.field("feedid", "532"); > content.field("versiontag", "20100927111516"); > content.field("bridgesymbols", ""); > content.field("headline", "S&P Market Commentary"); > content.field("productcategory", "MarketCommentary"); > content.field("publicationdatetime", "2010-09-27 > 11:15:16.000"); > content.field("source", "SNPMarketScope"); > content.field("teaser", ""); > content.field("wsodcompany", ""); > content.field("wsodissue", ""); > > ``` > client.prepareIndex("rebuild_index21", "index21", "532- > > ``` > > UMSACT0420100927111149CMCSA-20100927111149") > > .setSource(content).setOperationThreaded(false).execute() > .actionGet(); > } > > } ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [September 27, 2010, 9:45pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/9 "2010-09-27T21:45:54Z")
We upgraded to latest 0.11 master on the server and client. Running your version with a search via the java api works. But running the same query via the URL interface still produces the JsonParseException [http://localhost:9200/rebuild\_index21/\_search?q=\*](http://localhost:9200/rebuild_index21/_search?q=*) error: "JsonParseException[Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: org.elasticsearch.common.io.stream.LZFStreamInput@2ed3196e; line: 1, column: 2]]" David ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2010, 10:02pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/10 "2010-09-27T22:02:28Z")
Ahh, I missed the part where you do the search using the REST API. Yea, there is a bug in the auto conversion from the SMILE \_source content\_type to the REST JSON content\_type (with LZF compression on top of that ๐Ÿ˜‰ ). Pushed and fixed. On Mon, Sep 27, 2010 at 11:45 PM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > We upgraded to latest 0.11 master on the server and client. > > Running your version with a search via the java api works. But running > the same query via the URL interface still produces the > JsonParseException > > [http://localhost:9200/rebuild\_index21/\_search?q=](http://localhost:9200/rebuild_index21/_search?q=)\* > > error: "JsonParseException[Unexpected character (':' (code 58)): > expected a valid value (number, String, array, object, 'true', 'false' > or 'null') at [Source: > org.elasticsearch.common.io.stream.LZFStreamInput@2ed3196e; line: 1, > column: 2]]" > > David ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [September 27, 2010, 10:49pm UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/11 "2010-09-27T22:49:10Z")
Updated to master, confirmed this has been fixed. Thanks for the prompt service! David ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/corrupted-search-using-0-11-smile-format/3373/12 "2017-07-06T04:18:51Z")