# Problem indexing with my analyzer **URL:** \ **Category:** Elasticsearch\ **Created:** [June 19, 2014, 3:57pm UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214 "2014-06-19T15:57:46Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 19, 2014, 3:57pm UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/1 "2014-06-19T15:57:46Z")
Hello I have some issue, when I index a particular data "note\_source" (sql longtext). I use the same analyzer for each fields (except date\_source and id\_source) but for "note\_source", I have a "warn monitor.jvm". When I remove "note\_source", everything fine. If I don't use analyzer on "note\_source", everything fine, but if I use my analyzer on "note\_source" I have some crash. I think I have enough memory, I have used ES\_HEAP\_SIZE. Maybe my problem it's with accent (ascii, utf-8) Can you help me with this ? _My Setting_ public function createSetting($pf){ $params = array('index' =\> $pf, 'body' =\> array( 'settings' =\> array( 'number\_of\_shards' =\> 5, 'number\_of\_replicas' =\> 0, 'analysis' =\> array( 'filter' =\> array( 'nGram' =\> array( "token\_chars" =\>array(), "type" =\> "nGram", "min\_gram" =\> 3, "max\_gram" =\> 250 ) ), 'analyzer' =\> array( 'reuters' =\> array( 'type' =\> 'custom', 'tokenizer' =\> 'standard', 'filter' =\> array('lowercase', 'asciifolding', 'nGram') ) ) ) ) )); $this-\>elasticsearchClient-\>indices()-\>create($params); return; } _My Indexing_ public function indexTable($pf,$typeElement){ ``` $params =array( "index" =>'_river', "type" => $typeElement, "id" => "_meta", "body" =>array( "type" => "jdbc", "jdbc" => array( "url" => "jdbc:mysql://ip/name", "user" => 'root', "password" => 'mdp', "index" => $pf, "type" => $typeElement, "sql" => select id_source as _id, id_sous_theme, ``` titre\_source, desc\_source, note\_source, adresse\_source, type\_source, date\_source from source, "max\_bulk\_requests" =\> 5, ) ) ``` ); $this->elasticsearchClient->index($params); ``` } Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd6e60dc-d394-4d7d-b994-2105002d7bd7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd6e60dc-d394-4d7d-b994-2105002d7bd7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 20, 2014, 7:09am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/2 "2014-06-20T07:09:36Z")
Information My "note\_source" contain picture (.jpg, .png ...) in base64 and text. For my mapping I have used : "type" =\> "string" "analyzer" =\> "reuteurs" (the name of my analyzer) Any idea ? Le jeudi 19 juin 2014 17:57:46 UTC+2, Tanguy Bernard a écrit : > Hello > I have some issue, when I index a particular data "note\_source" (sql > longtext). > I use the same analyzer for each fields (except date\_source and id\_source) > but for "note\_source", I have a "warn monitor.jvm". > When I remove "note\_source", everything fine. If I don't use analyzer on > "note\_source", everything fine, but if I use my analyzer on "note\_source" I > have some crash. > > I think I have enough memory, I have used ES\_HEAP\_SIZE. > Maybe my problem it's with accent (ascii, utf-8) > > Can you help me with this ? > > _My Setting_ > > public function createSetting($pf){ > $params = array('index' =\> $pf, 'body' =\> array( > 'settings' =\> array( > 'number\_of\_shards' =\> 5, > 'number\_of\_replicas' =\> 0, > 'analysis' =\> array( > 'filter' =\> array( > 'nGram' =\> array( > "token\_chars" =\>array(), > "type" =\> "nGram", > "min\_gram" =\> 3, > "max\_gram" =\> 250 > ) > ), > 'analyzer' =\> array( > 'reuters' =\> array( > 'type' =\> 'custom', > 'tokenizer' =\> 'standard', > 'filter' =\> array('lowercase', 'asciifolding', > 'nGram') > ) > ) > ) > ) > )); > $this-\>elasticsearchClient-\>indices()-\>create($params); > return; > } > > _My Indexing_ > > public function indexTable($pf,$typeElement){ > > ``` > $params =array( > "index" =>'_river', > "type" => $typeElement, > "id" => "_meta", > "body" =>array( > > "type" => "jdbc", > "jdbc" => array( > "url" => "jdbc:mysql://ip/name", > "user" => 'root', > "password" => 'mdp', > "index" => $pf, > "type" => $typeElement, > "sql" => select id_source as _id, id_sous_theme, > > ``` > > titre\_source, desc\_source, note\_source, adresse\_source, type\_source, > date\_source from source, > "max\_bulk\_requests" =\> 5, > ) > ) > > ``` > ); > > > $this->elasticsearchClient->index($params); > > ``` > > } > > Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Cedric\_Hourcade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cedric_hourcade/32/1451_2.png) [@Cedric\_Hourcade](https://discuss.elastic.co/u/Cedric_Hourcade)\ **Post date:** [June 20, 2014, 8:58am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/3 "2014-06-20T08:58:26Z")
Does it mean your applying the "reuters" analyzer on your base64 encoded pictures? I guess it generates a really huge number of tokens for each entry because of your nGram filter (with a max at 250). Cédric Hourcade [ced@wal.fr](mailto:ced@wal.fr) On Fri, Jun 20, 2014 at 9:09 AM, Tanguy Bernard [bernardtanguy1pro@gmail.com](mailto:bernardtanguy1pro@gmail.com) wrote: > Information > My "note\_source" contain picture (.jpg, .png ...) in base64 and text. > > For my mapping I have used : > "type" =\> "string" > "analyzer" =\> "reuteurs" (the name of my analyzer) > > Any idea ? > > Le jeudi 19 juin 2014 17:57:46 UTC+2, Tanguy Bernard a écrit : > > > Hello > > I have some issue, when I index a particular data "note\_source" (sql > > longtext). > > I use the same analyzer for each fields (except date\_source and id\_source) > > but for "note\_source", I have a "warn monitor.jvm". > > When I remove "note\_source", everything fine. If I don't use analyzer on > > "note\_source", everything fine, but if I use my analyzer on "note\_source" I > > have some crash. > > > > I think I have enough memory, I have used ES\_HEAP\_SIZE. > > Maybe my problem it's with accent (ascii, utf-8) > > > > Can you help me with this ? > > > > My Setting > > > > public function createSetting($pf){ > > $params = array('index' =\> $pf, 'body' =\> array( > > 'settings' =\> array( > > 'number\_of\_shards' =\> 5, > > 'number\_of\_replicas' =\> 0, > > 'analysis' =\> array( > > 'filter' =\> array( > > 'nGram' =\> array( > > "token\_chars" =\>array(), > > "type" =\> "nGram", > > "min\_gram" =\> 3, > > "max\_gram" =\> 250 > > ) > > ), > > 'analyzer' =\> array( > > 'reuters' =\> array( > > 'type' =\> 'custom', > > 'tokenizer' =\> 'standard', > > 'filter' =\> array('lowercase', 'asciifolding', > > 'nGram') > > ) > > ) > > ) > > ) > > )); > > $this-\>elasticsearchClient-\>indices()-\>create($params); > > return; > > } > > > > My Indexing > > > > public function indexTable($pf,$typeElement){ > > > > ``` > > $params =array( > > "index" =>'_river', > > "type" => $typeElement, > > "id" => "_meta", > > "body" =>array( > > > > "type" => "jdbc", > > "jdbc" => array( > > "url" => "jdbc:mysql://ip/name", > > "user" => 'root', > > "password" => 'mdp', > > "index" => $pf, > > "type" => $typeElement, > > "sql" => select id_source as _id, id_sous_theme, > > > > ``` > > > > titre\_source, desc\_source, note\_source, adresse\_source, type\_source, > > date\_source from source, > > "max\_bulk\_requests" =\> 5, > > ) > > ) > > > > ``` > > ); > > > > $this->elasticsearchClient->index($params); > > > > ``` > > > > } > > > > Thanks in advance. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJQxjPM8qvsmcxB7Xu4KqN28pfvk%2BcBn5bpV2Emw42M5HzAAUA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJQxjPM8qvsmcxB7Xu4KqN28pfvk%2BcBn5bpV2Emw42M5HzAAUA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 20, 2014, 9:25am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/4 "2014-06-20T09:25:39Z")
Yes, I am applying "reuters" on my document (compose by text and picture). My goal is to do my research on the text of the document with any word or part of a word. Yes the problem it's my nGram filter. How do I solve this problem ? Deacrease nGram max ? Change Analyzer by an other but who satisfy my goal ? Le vendredi 20 juin 2014 10:58:49 UTC+2, Cédric Hourcade a écrit : > Does it mean your applying the "reuters" analyzer on your base64 > encoded pictures? > > I guess it generates a really huge number of tokens for each entry > because of your nGram filter (with a max at 250). > > Cédric Hourcade > [c...@wal.fr](mailto:c...@wal.fr) \ > > On Fri, Jun 20, 2014 at 9:09 AM, Tanguy Bernard > \<[bernardt...@gmail.com](mailto:bernardt...@gmail.com) \\> wrote: > > > Information > > My "note\_source" contain picture (.jpg, .png ...) in base64 and text. > > > > For my mapping I have used : > > "type" =\> "string" > > "analyzer" =\> "reuteurs" (the name of my analyzer) > > > > Any idea ? > > > > Le jeudi 19 juin 2014 17:57:46 UTC+2, Tanguy Bernard a écrit : > > > > > Hello > > > I have some issue, when I index a particular data "note\_source" (sql > > > longtext). > > > I use the same analyzer for each fields (except date\_source and > > > id\_source) > > > but for "note\_source", I have a "warn monitor.jvm". > > > When I remove "note\_source", everything fine. If I don't use analyzer > > > on > > > "note\_source", everything fine, but if I use my analyzer on > > > "note\_source" I > > > have some crash. > > > > > > I think I have enough memory, I have used ES\_HEAP\_SIZE. > > > Maybe my problem it's with accent (ascii, utf-8) > > > > > > Can you help me with this ? > > > > > > My Setting > > > > > > public function createSetting($pf){ > > > $params = array('index' =\> $pf, 'body' =\> array( > > > 'settings' =\> array( > > > 'number\_of\_shards' =\> 5, > > > 'number\_of\_replicas' =\> 0, > > > 'analysis' =\> array( > > > 'filter' =\> array( > > > 'nGram' =\> array( > > > "token\_chars" =\>array(), > > > "type" =\> "nGram", > > > "min\_gram" =\> 3, > > > "max\_gram" =\> 250 > > > ) > > > ), > > > 'analyzer' =\> array( > > > 'reuters' =\> array( > > > 'type' =\> 'custom', > > > 'tokenizer' =\> 'standard', > > > 'filter' =\> array('lowercase', 'asciifolding', > > > 'nGram') > > > ) > > > ) > > > ) > > > ) > > > )); > > > $this-\>elasticsearchClient-\>indices()-\>create($params); > > > return; > > > } > > > > > > My Indexing > > > > > > public function indexTable($pf,$typeElement){ > > > > > > ``` > > > $params =array( > > > "index" =>'_river', > > > "type" => $typeElement, > > > "id" => "_meta", > > > "body" =>array( > > > > > > "type" => "jdbc", > > > "jdbc" => array( > > > "url" => "jdbc:mysql://ip/name", > > > "user" => 'root', > > > "password" => 'mdp', > > > "index" => $pf, > > > "type" => $typeElement, > > > "sql" => select id_source as _id, id_sous_theme, > > > > > > ``` > > > > > > titre\_source, desc\_source, note\_source, adresse\_source, type\_source, > > > date\_source from source, > > > "max\_bulk\_requests" =\> 5, > > > ) > > > ) > > > > > > ``` > > > ); > > > > > > $this->elasticsearchClient->index($params); > > > > > > ``` > > > > > > } > > > > > > Thanks in advance. > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b7daa716-cb5f-45cc-916b-43c7c0aea6b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7daa716-cb5f-45cc-916b-43c7c0aea6b9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Cedric\_Hourcade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cedric_hourcade/32/1451_2.png) [@Cedric\_Hourcade](https://discuss.elastic.co/u/Cedric_Hourcade)\ **Post date:** [June 20, 2014, 9:44am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/5 "2014-06-20T09:44:59Z")
If you are only searching in the text you should index the images in an other field field. With no analyzer ("index: not\_analyzed"), or even better "index: no" (not indexed). If you need to retrieve the image data it's still in the \_source. But to be honest I wouldn't even store this kind of information in ES, your index is going to be bigger, merges are going to be slower... I'd keep the binary files stored elsewhere. Cédric Hourcade [ced@wal.fr](mailto:ced@wal.fr) On Fri, Jun 20, 2014 at 11:25 AM, Tanguy Bernard [bernardtanguy1pro@gmail.com](mailto:bernardtanguy1pro@gmail.com) wrote: > Yes, I am applying "reuters" on my document (compose by text and picture). > My goal is to do my research on the text of the document with any word or > part of a word. > > Yes the problem it's my nGram filter. > How do I solve this problem ? Deacrease nGram max ? Change Analyzer by an > other but who satisfy my goal ? > > Le vendredi 20 juin 2014 10:58:49 UTC+2, Cédric Hourcade a écrit : > > > Does it mean your applying the "reuters" analyzer on your base64 > > encoded pictures? > > > > I guess it generates a really huge number of tokens for each entry > > because of your nGram filter (with a max at 250). > > > > Cédric Hourcade > > [c...@wal.fr](mailto:c...@wal.fr) > > > > On Fri, Jun 20, 2014 at 9:09 AM, Tanguy Bernard > > [bernardt...@gmail.com](mailto:bernardt...@gmail.com) wrote: > > > > > Information > > > My "note\_source" contain picture (.jpg, .png ...) in base64 and text. > > > > > > For my mapping I have used : > > > "type" =\> "string" > > > "analyzer" =\> "reuteurs" (the name of my analyzer) > > > > > > Any idea ? > > > > > > Le jeudi 19 juin 2014 17:57:46 UTC+2, Tanguy Bernard a écrit : > > > > > > > Hello > > > > I have some issue, when I index a particular data "note\_source" (sql > > > > longtext). > > > > I use the same analyzer for each fields (except date\_source and > > > > id\_source) > > > > but for "note\_source", I have a "warn monitor.jvm". > > > > When I remove "note\_source", everything fine. If I don't use analyzer > > > > on > > > > "note\_source", everything fine, but if I use my analyzer on > > > > "note\_source" I > > > > have some crash. > > > > > > > > I think I have enough memory, I have used ES\_HEAP\_SIZE. > > > > Maybe my problem it's with accent (ascii, utf-8) > > > > > > > > Can you help me with this ? > > > > > > > > My Setting > > > > > > > > public function createSetting($pf){ > > > > $params = array('index' =\> $pf, 'body' =\> array( > > > > 'settings' =\> array( > > > > 'number\_of\_shards' =\> 5, > > > > 'number\_of\_replicas' =\> 0, > > > > 'analysis' =\> array( > > > > 'filter' =\> array( > > > > 'nGram' =\> array( > > > > "token\_chars" =\>array(), > > > > "type" =\> "nGram", > > > > "min\_gram" =\> 3, > > > > "max\_gram" =\> 250 > > > > ) > > > > ), > > > > 'analyzer' =\> array( > > > > 'reuters' =\> array( > > > > 'type' =\> 'custom', > > > > 'tokenizer' =\> 'standard', > > > > 'filter' =\> array('lowercase', 'asciifolding', > > > > 'nGram') > > > > ) > > > > ) > > > > ) > > > > ) > > > > )); > > > > $this-\>elasticsearchClient-\>indices()-\>create($params); > > > > return; > > > > } > > > > > > > > My Indexing > > > > > > > > public function indexTable($pf,$typeElement){ > > > > > > > > ``` > > > > $params =array( > > > > "index" =>'_river', > > > > "type" => $typeElement, > > > > "id" => "_meta", > > > > "body" =>array( > > > > > > > > "type" => "jdbc", > > > > "jdbc" => array( > > > > "url" => "jdbc:mysql://ip/name", > > > > "user" => 'root', > > > > "password" => 'mdp', > > > > "index" => $pf, > > > > "type" => $typeElement, > > > > "sql" => select id_source as _id, id_sous_theme, > > > > > > > > ``` > > > > > > > > titre\_source, desc\_source, note\_source, adresse\_source, type\_source, > > > > date\_source from source, > > > > "max\_bulk\_requests" =\> 5, > > > > ) > > > > ) > > > > > > > > ``` > > > > ); > > > > > > > > $this->elasticsearchClient->index($params); > > > > > > > > ``` > > > > > > > > } > > > > > > > > Thanks in advance. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b7daa716-cb5f-45cc-916b-43c7c0aea6b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7daa716-cb5f-45cc-916b-43c7c0aea6b9%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJQxjPOf8kbDpr-EuDfskLj4UjQs4FAq04GrWH87fFy0df8EPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJQxjPOf8kbDpr-EuDfskLj4UjQs4FAq04GrWH87fFy0df8EPQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 20, 2014, 9:48am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/6 "2014-06-20T09:48:55Z")
I set max\_gram=20. It's better but at the end I have this many times : [2014-06-20 11:42:14,201][WARN][monitor.jvm] [ik-test2] [gc][young][528][263] duration [2s], collections [1]/[2.1s], total [2s]/[43.9s], memory [536mb]-\>[580.2mb]/[1015.6mb], all\_pools {[young] [22.5mb]-\>[22.3mb]/[66.5mb]}{[survivor] [14.9kb]-\>[49.3kb]/[8.3mb]}{[old] [513.4mb]-\>[557.8mb]/[940.8mb]} I put ES\_HEAP\_SIZE : 2G. I think it's enough. Something wrong ? Le jeudi 19 juin 2014 17:57:46 UTC+2, Tanguy Bernard a écrit : > Hello > I have some issue, when I index a particular data "note\_source" (sql > longtext). > I use the same analyzer for each fields (except date\_source and id\_source) > but for "note\_source", I have a "warn monitor.jvm". > When I remove "note\_source", everything fine. If I don't use analyzer on > "note\_source", everything fine, but if I use my analyzer on "note\_source" I > have some crash. > > I think I have enough memory, I have used ES\_HEAP\_SIZE. > Maybe my problem it's with accent (ascii, utf-8) > > Can you help me with this ? > > _My Setting_ > > public function createSetting($pf){ > $params = array('index' =\> $pf, 'body' =\> array( > 'settings' =\> array( > 'number\_of\_shards' =\> 5, > 'number\_of\_replicas' =\> 0, > 'analysis' =\> array( > 'filter' =\> array( > 'nGram' =\> array( > "token\_chars" =\>array(), > "type" =\> "nGram", > "min\_gram" =\> 3, > "max\_gram" =\> 250 > ) > ), > 'analyzer' =\> array( > 'reuters' =\> array( > 'type' =\> 'custom', > 'tokenizer' =\> 'standard', > 'filter' =\> array('lowercase', 'asciifolding', > 'nGram') > ) > ) > ) > ) > )); > $this-\>elasticsearchClient-\>indices()-\>create($params); > return; > } > > _My Indexing_ > > public function indexTable($pf,$typeElement){ > > ``` > $params =array( > "index" =>'_river', > "type" => $typeElement, > "id" => "_meta", > "body" =>array( > > "type" => "jdbc", > "jdbc" => array( > "url" => "jdbc:mysql://ip/name", > "user" => 'root', > "password" => 'mdp', > "index" => $pf, > "type" => $typeElement, > "sql" => select id_source as _id, id_sous_theme, > > ``` > > titre\_source, desc\_source, note\_source, adresse\_source, type\_source, > date\_source from source, > "max\_bulk\_requests" =\> 5, > ) > ) > > ``` > ); > > > $this->elasticsearchClient->index($params); > > ``` > > } > > Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/154b8ca2-a130-4062-b5ce-0e0fa63d98fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/154b8ca2-a130-4062-b5ce-0e0fa63d98fe%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 20, 2014, 9:55am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/7 "2014-06-20T09:55:39Z")
The user copy/paste the content of an html page and me, I index this information. I take the entire document with image. I can't change this behavior. I set max\_gram=20. It's better but at the end I have this many times : [2014-06-20 11:42:14,201][WARN][monitor.jvm] [ik-test2] [gc][young][528][263] duration [2s], collections [1]/[2.1s], total [2s]/[43.9s], memory [536mb]-\>[580.2mb]/[1015.6mb], all\_pools {[young] [22.5mb]-\>[22.3mb]/[66.5mb]}{[survivor] [14.9kb]-\>[49.3kb]/[8.3mb]}{[old] [513.4mb]-\>[557.8mb]/[940.8mb]} I put ES\_HEAP\_SIZE : 2G. I think it's enough. Something wrong ? Le vendredi 20 juin 2014 11:45:22 UTC+2, Cédric Hourcade a écrit : > If you are only searching in the text you should index the images in > an other field field. With no analyzer ("index: not\_analyzed"), or > even better "index: no" (not indexed). If you need to retrieve the > image data it's still in the \_source. > > But to be honest I wouldn't even store this kind of information in ES, > your index is going to be bigger, merges are going to be slower... I'd > keep the binary files stored elsewhere. > > Cédric Hourcade > [c...@wal.fr](mailto:c...@wal.fr) \ > > On Fri, Jun 20, 2014 at 11:25 AM, Tanguy Bernard > \<[bernardt...@gmail.com](mailto:bernardt...@gmail.com) \\> wrote: > > > Yes, I am applying "reuters" on my document (compose by text and > > picture). > > My goal is to do my research on the text of the document with any word > > or > > part of a word. > > > > Yes the problem it's my nGram filter. > > How do I solve this problem ? Deacrease nGram max ? Change Analyzer by > > an > > other but who satisfy my goal ? > > > > Le vendredi 20 juin 2014 10:58:49 UTC+2, Cédric Hourcade a écrit : > > > > > Does it mean your applying the "reuters" analyzer on your base64 > > > encoded pictures? > > > > > > I guess it generates a really huge number of tokens for each entry > > > because of your nGram filter (with a max at 250). > > > > > > Cédric Hourcade > > > [c...@wal.fr](mailto:c...@wal.fr) > > > > > > On Fri, Jun 20, 2014 at 9:09 AM, Tanguy Bernard > > > [bernardt...@gmail.com](mailto:bernardt...@gmail.com) wrote: > > > > > > > Information > > > > My "note\_source" contain picture (.jpg, .png ...) in base64 and text. > > > > > > > > For my mapping I have used : > > > > "type" =\> "string" > > > > "analyzer" =\> "reuteurs" (the name of my analyzer) > > > > > > > > Any idea ? > > > > > > > > Le jeudi 19 juin 2014 17:57:46 UTC+2, Tanguy Bernard a écrit : > > > > > > > > > Hello > > > > > I have some issue, when I index a particular data "note\_source" (sql > > > > > longtext). > > > > > I use the same analyzer for each fields (except date\_source and > > > > > id\_source) > > > > > but for "note\_source", I have a "warn monitor.jvm". > > > > > When I remove "note\_source", everything fine. If I don't use > > > > > analyzer > > > > > on > > > > > "note\_source", everything fine, but if I use my analyzer on > > > > > "note\_source" I > > > > > have some crash. > > > > > > > > > > I think I have enough memory, I have used ES\_HEAP\_SIZE. > > > > > Maybe my problem it's with accent (ascii, utf-8) > > > > > > > > > > Can you help me with this ? > > > > > > > > > > My Setting > > > > > > > > > > public function createSetting($pf){ > > > > > $params = array('index' =\> $pf, 'body' =\> array( > > > > > 'settings' =\> array( > > > > > 'number\_of\_shards' =\> 5, > > > > > 'number\_of\_replicas' =\> 0, > > > > > 'analysis' =\> array( > > > > > 'filter' =\> array( > > > > > 'nGram' =\> array( > > > > > "token\_chars" =\>array(), > > > > > "type" =\> "nGram", > > > > > "min\_gram" =\> 3, > > > > > "max\_gram" =\> 250 > > > > > ) > > > > > ), > > > > > 'analyzer' =\> array( > > > > > 'reuters' =\> array( > > > > > 'type' =\> 'custom', > > > > > 'tokenizer' =\> 'standard', > > > > > 'filter' =\> array('lowercase', > > > > > 'asciifolding', > > > > > 'nGram') > > > > > ) > > > > > ) > > > > > ) > > > > > ) > > > > > )); > > > > > $this-\>elasticsearchClient-\>indices()-\>create($params); > > > > > return; > > > > > } > > > > > > > > > > My Indexing > > > > > > > > > > public function indexTable($pf,$typeElement){ > > > > > > > > > > ``` > > > > > $params =array( > > > > > "index" =>'_river', > > > > > "type" => $typeElement, > > > > > "id" => "_meta", > > > > > "body" =>array( > > > > > > > > > > "type" => "jdbc", > > > > > "jdbc" => array( > > > > > "url" => "jdbc:mysql://ip/name", > > > > > "user" => 'root', > > > > > "password" => 'mdp', > > > > > "index" => $pf, > > > > > "type" => $typeElement, > > > > > "sql" => select id_source as _id, id_sous_theme, > > > > > > > > > > ``` > > > > > > > > > > titre\_source, desc\_source, note\_source, adresse\_source, type\_source, > > > > > date\_source from source, > > > > > "max\_bulk\_requests" =\> 5, > > > > > ) > > > > > ) > > > > > > > > > > ``` > > > > > ); > > > > > > > > > > $this->elasticsearchClient->index($params); > > > > > > > > > > ``` > > > > > > > > > > } > > > > > > > > > > Thanks in advance. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, > > > > send > > > > an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d93217c-bded-40fa-8fd2-fdac576c57ee%40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b7daa716-cb5f-45cc-916b-43c7c0aea6b9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7daa716-cb5f-45cc-916b-43c7c0aea6b9%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7e086bdb-6eac-4d92-a9b1-c60262576588%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7e086bdb-6eac-4d92-a9b1-c60262576588%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Cedric\_Hourcade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cedric_hourcade/32/1451_2.png) [@Cedric\_Hourcade](https://discuss.elastic.co/u/Cedric_Hourcade)\ **Post date:** [June 20, 2014, 1:32pm UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/8 "2014-06-20T13:32:04Z")
If your base64 encodes are long, they are going to be splited in a lot of tokens by the standard tokenizer. Theses tokens are often going to be a lot longer than standard words, so your nGram filter will generate even more tokens, a lot more than with standard text. That may be your problem there. You should really try to strip the encoded images with a simple regex from your documents before indexing them. If you need to keep the source, put the raw text in an unindexed field, and the cleaned one in another. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJQxjPPD4UXAjX%2Buwi84LSsPeiy0C80uzcb4C1QFxwLzfyjQGA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJQxjPPD4UXAjX%2Buwi84LSsPeiy0C80uzcb4C1QFxwLzfyjQGA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 20, 2014, 2:05pm UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/9 "2014-06-20T14:05:01Z")
Thank you Cédric Hourcade ! Le vendredi 20 juin 2014 15:32:29 UTC+2, Cédric Hourcade a écrit : > If your base64 encodes are long, they are going to be splited in a lot > of tokens by the standard tokenizer. > > Theses tokens are often going to be a lot longer than standard words, > so your nGram filter will generate even more tokens, a lot more than > with standard text. That may be your problem there. > > You should really try to strip the encoded images with a simple regex > from your documents before indexing them. If you need to keep the > source, put the raw text in an unindexed field, and the cleaned one in > another. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 20, 2014, 10:35pm UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/10 "2014-06-20T22:35:33Z")
You seriously don't want 3..250 length ngrams!!!! That's ENORMOUS Typically set min/max to 3 or 4, and that's it > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On 20 June 2014 16:05, Tanguy Bernard [bernardtanguy1pro@gmail.com](mailto:bernardtanguy1pro@gmail.com) wrote: > Thank you Cédric Hourcade ! > > Le vendredi 20 juin 2014 15:32:29 UTC+2, Cédric Hourcade a écrit : > > > If your base64 encodes are long, they are going to be splited in a lot > > of tokens by the standard tokenizer. > > > > Theses tokens are often going to be a lot longer than standard words, > > so your nGram filter will generate even more tokens, a lot more than > > with standard text. That may be your problem there. > > > > You should really try to strip the encoded images with a simple regex > > from your documents before indexing them. If you need to keep the > > source, put the raw text in an unindexed field, and the cleaned one in > > another. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPt3XKRS\_zD%3DkVpKBpqp3hkcgJacAWsETGgJwMQJM%2BqJMuvscw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPt3XKRS_zD%3DkVpKBpqp3hkcgJacAWsETGgJwMQJM%2BqJMuvscw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tanguy\_Bernard](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@Tanguy\_Bernard](https://discuss.elastic.co/u/Tanguy_Bernard)\ **Post date:** [June 25, 2014, 2:31pm UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/11 "2014-06-25T14:31:27Z")
Yes I did not know how nGram works ! I find a perfect solution for my picture (base64) problem : use _'char\_filter' =\>array('html\_strip'),_ public function createSetting($pf){ $params = array('index' =\> $pf, 'body' =\> array( 'settings' =\> array( 'number\_of\_shards' =\> 5, 'number\_of\_replicas' =\> 0, 'analysis' =\> array( 'filter' =\> array( 'MYnGram' =\> array( "token\_chars" =\>array(), "type" =\> "nGram", "min\_gram" =\> 3, "max\_gram" =\> 20 ) ), 'analyzer' =\> array( 'reuters' =\> array( 'type' =\> 'custom', 'tokenizer' =\> 'standard', 'filter' =\> array('lowercase', 'asciifolding', 'MYnGram'), 'char\_filter' =\>array('html\_strip'), ), ``` ) ) ) )); $this->elasticsearchClient->indices()->create($params); ``` } Thanks to all of you ! Le samedi 21 juin 2014 00:35:39 UTC+2, Clinton Gormley a écrit : > You seriously don't want 3..250 length ngrams!!!! That's ENORMOUS > > Typically set min/max to 3 or 4, and that's it > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_ngrams_for_partial_matching.html#_ngrams_for_partial_matching) > > On 20 June 2014 16:05, Tanguy Bernard \<[bernardt...@gmail.com](mailto:bernardt...@gmail.com) \ > > > wrote: > > > Thank you Cédric Hourcade ! > > > > Le vendredi 20 juin 2014 15:32:29 UTC+2, Cédric Hourcade a écrit : > > > > > If your base64 encodes are long, they are going to be splited in a lot > > > of tokens by the standard tokenizer. > > > > > > Theses tokens are often going to be a lot longer than standard words, > > > so your nGram filter will generate even more tokens, a lot more than > > > with standard text. That may be your problem there. > > > > > > You should really try to strip the encoded images with a simple regex > > > from your documents before indexing them. If you need to keep the > > > source, put the raw text in an unindexed field, and the cleaned one in > > > another. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b62f4e12-1b54-4621-986a-93411404f7af%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2bdd5f30-8e97-43e0-8478-08cc26a03ed9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2bdd5f30-8e97-43e0-8478-08cc26a03ed9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:19am UTC](https://discuss.elastic.co/t/problem-indexing-with-my-analyzer/18214/12 "2017-07-06T01:19:51Z")