# Trying to remove stopwords from ES-index JAVA API **URL:** \ **Category:** Elasticsearch\ **Created:** [July 22, 2015, 3:13pm UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093 "2015-07-22T15:13:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fluehmann/32/3813_2.png) [@fluehmann](https://discuss.elastic.co/u/fluehmann)\ **Post date:** [July 22, 2015, 3:13pm UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/1 "2015-07-22T15:13:43Z")
Hey guys I'm all new to ES. I'm indexing website content with a crawler into ES. So now I'm trying to apply an analyzer to my index (website content) to remove all stopwords from the website content throught the Java API. Therefore I load the settings from a .json settings file when indexing with a FileInputStream. When looking into the metadata in the head plugin the analyzer is shown properly. I can even access the analyzer via the sense plugin. But what I actually wanted to achieve was to remove stopwords from the index data that I'm indexing. I expected the content to be cleaned from stopwords. But all stopwords remain there. What am I doing wrong? Can anyone help me? My indexing code: ``` public boolean index(String index, String type, String id, HashMap fields) throws ElasticsearchException, IOException { if (!checkIfIndexExists(index)){ Settings indexSettings = ImmutableSettings.settingsBuilder() .put("number_of_shards", 5) .put("number_of_replicas", 1) .build(); CreateIndexRequest indexRequest = new CreateIndexRequest(index, indexSettings); indexRequest.settings(getSettingsJsonString()); client.admin().indices().create(indexRequest).actionGet(); } ``` my settings file: ``` { "analysis": { "analyzer": { "de_std": { "type": "standard", "stopwords": "_german_" } } } } ``` Cheers Simon ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [July 22, 2015, 3:18pm UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/2 "2015-07-22T15:18:41Z")
You will need to set the `analyzer` field in your mappings for the fields you want to use this analyzer. See here for more information on the settings for String fields including the `analyzer` parameter: [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#string](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#string) If you want to set all String fields to use that analyzer you should look at dynamic templates: [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#\_dynamic\_templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#_dynamic_templates) Hope that helps ---
**Author:** ![fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fluehmann/32/3813_2.png) [@fluehmann](https://discuss.elastic.co/u/fluehmann)\ **Post date:** [July 23, 2015, 7:28am UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/3 "2015-07-23T07:28:39Z")
Hey colings86 Thank you very much for your reply - I appreciate it very much! The use of mapping makes a lot of sense to me. My mapping file: ``` { website": { "properties": { "content": { "type": "string" "analyzer": "de_std" } "timestamp": { "type": "string" } "url": { "type": "string" } } } } ``` Now I see the following in the metadata - and still no effect. ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0967edb0f8f6f27e2b38aca242121b92cb7f6edd.png) Do I have to overwrite the default Mapping somehow? I'm sorry for my noob questions... Cheers Simon ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [July 23, 2015, 10:28am UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/4 "2015-07-23T10:28:20Z")
When you applied your new mappings did you create a new index with the new mappings and then re-index your data into the new index? You cannot update mappings on an existing index so if you didn't re-index you will need to for your analyzer to be used in your mappings > [@fluehmann](#): > > I'm sorry for my noob questions... No worries these are good questions 🙂 ---
**Author:** ![fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fluehmann/32/3813_2.png) [@fluehmann](https://discuss.elastic.co/u/fluehmann)\ **Post date:** [July 23, 2015, 11:47am UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/5 "2015-07-23T11:47:51Z")
Hey Thanks for your reply. Yes, I always delete the index and reindex it if I want to apply the Settings/Mapping: ``` [2015-07-23 13:40:53,661][INFO][cluster.metadata] [Chemistro] [emmental.ch] deleting index [2015-07-23 13:41:09,559][INFO][cluster.metadata] [Chemistro] [emmental.ch] creating index, cause [api], templates [], shards [5]/[1], mappings [{ website": { "properties": { "content": { "type": "string" "analyzer": "de_std" } "timestamp": { "type": "string" } "url": { "type": "string" } } } } ] [2015-07-23 13:41:09,631][INFO][cluster.metadata] [Chemistro] [emmental.ch] update_mapping [website] (dynamic) ``` This is how I load the mapping: ``` CreateIndexRequest indexRequest = new CreateIndexRequest(index, indexSettings); indexRequest.settings(getSettingsJsonString()); indexRequest.mapping(getMappingsJsonString()); client.admin().indices().create(indexRequest).actionGet(); ``` In the metadata the Settings are displayed correctly - but the mapping isn't (screenshot post before). Do you see any mistakes? ---
**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\ **Post date:** [July 23, 2015, 12:03pm UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/6 "2015-07-23T12:03:59Z")
> [@fluehmann](#): > > Yes, I always delete the index and reindex it if I want to apply the Settings/Mapping Good, you should always do this 😄 I don't see any mistakes in what you've put here. Maybe @Clinton_Gormley has some ideas on what could be going wrong? ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [July 23, 2015, 1:05pm UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/7 "2015-07-23T13:05:04Z")
@fluehmann It's not very clear what you mean by: > I expected the content to be cleaned from stopwords. But all stopwords remain there. If you're expecting the stopwords to be removed from the `_source` field, that doesn't happen. The `_source` field is never changed. It just removes stopwords from the field before indexing the terms for search. ---
**Author:** ![fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fluehmann/32/3813_2.png) [@fluehmann](https://discuss.elastic.co/u/fluehmann)\ **Post date:** [July 24, 2015, 10:12am UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/8 "2015-07-24T10:12:21Z")
Hey guys Thank you for your help. I had to do some thinking... 😄 Now it's clear to me that source field will never be changed and only when retrieving the index the stopwords will be ommited. I think it works now. I had to change the mapping to fit the constructor: ``` indexRequest.mapping("website", getMappingsJsonString()); ``` That's why the mapping was first interpreted as a single string instead of a JSON-file. Thank you guys very much for your help and patience! I guess I'll be back soon with some more questions... 😉 Cheers Simon ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:59pm UTC](https://discuss.elastic.co/t/trying-to-remove-stopwords-from-es-index-java-api/26093/9 "2017-07-05T23:59:18Z")