# Stopwords(Elasticsearch7.14+ fscrawler2.7)

**URL:** <https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053>\
**Category:** Elasticsearch\
**Created:** [September 13, 2021, 10:37am UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053 "2021-09-13T10:37:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tarik\_Bouhasse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tarik_bouhasse/32/79281_2.png) [@Tarik\_Bouhasse](https://discuss.elastic.co/u/Tarik_Bouhasse)\
**Post date:** [September 13, 2021, 10:37am UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/1 "2021-09-13T10:37:14Z")

</div>

how to configure stopwords. I am in fscrawler2.7 and Elasticsearch7.14

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 13, 2021, 11:07am UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/2 "2021-09-13T11:07:47Z")

</div>

Welcome!

You need to define your own analyzer as explained in [https://fscrawler.readthedocs.io/en/fscrawler-2.7/admin/fs/elasticsearch.html#creating-your-own-mapping-analyzers](https://fscrawler.readthedocs.io/en/fscrawler-2.7/admin/fs/elasticsearch.html#creating-your-own-mapping-analyzers)

I'd recommend doing it manually with (quote from docs):

> Note that if you want to push manually the mapping to elasticsearch you can use the classic REST calls:
> 
> ```auto
> # Create index (don't forget to add the fscrawler_path analyzer)
> PUT docs
> {
> // Same index settings as previously seen
> }
> 
> ```

You can configure the stopword of the language analyzer you want to use. See [Language analyzers | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html#_stopwords)

Or for example:

```auto
PUT /docs
{
  "settings": {
    "number_of_shards": 1,
    "index.mapping.total_fields.limit": 2000,
    "analysis": {
      "tokenizer": {
        "fscrawler_path": {
          "type": "path_hierarchy"
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": ["_english_", "fscrawler"]
        },
        "english_keywords": {
          "type": "keyword_marker",
          "keywords": ["example"] 
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "english_possessive_stemmer": {
          "type": "stemmer",
          "language": "possessive_english"
        }
      },
      "analyzer": {
        "fscrawler_path": {
          "tokenizer": "fscrawler_path"
        },
        "rebuilt_english": {
          "tokenizer": "standard",
          "filter": [
            "english_possessive_stemmer",
            "lowercase",
            "english_stop",
            "english_keywords",
            "english_stemmer"
          ]
        }
      }
    }
  },
  "mappings": {
    "_doc": {
      "dynamic_templates": [
        {
          "raw_as_text": {
            "path_match": "meta.raw.*",
            "mapping": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ],
      "properties": {
        "attachment": {
          "type": "binary",
          "doc_values": false
        },
        "attributes": {
          "properties": {
            "group": {
              "type": "keyword"
            },
            "owner": {
              "type": "keyword"
            }
          }
        },
        "content": {
          "type": "text",
          "analyzer": "rebuilt_english"
        },
        "file": {
          "properties": {
            "content_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword",
              "store": true
            },
            "extension": {
              "type": "keyword"
            },
            "filesize": {
              "type": "long"
            },
            "indexed_chars": {
              "type": "long"
            },
            "indexing_date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "created": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "last_modified": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "last_accessed": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "checksum": {
              "type": "keyword"
            },
            "url": {
              "type": "keyword",
              "index": false
            }
          }
        },
        "meta": {
          "properties": {
            "author": {
              "type": "text"
            },
            "date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "keywords": {
              "type": "text"
            },
            "title": {
              "type": "text"
            },
            "language": {
              "type": "keyword"
            },
            "format": {
              "type": "text"
            },
            "identifier": {
              "type": "text"
            },
            "contributor": {
              "type": "text"
            },
            "coverage": {
              "type": "text"
            },
            "modifier": {
              "type": "text"
            },
            "creator_tool": {
              "type": "keyword"
            },
            "publisher": {
              "type": "text"
            },
            "relation": {
              "type": "text"
            },
            "rights": {
              "type": "text"
            },
            "source": {
              "type": "text"
            },
            "type": {
              "type": "text"
            },
            "description": {
              "type": "text"
            },
            "created": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "print_date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "metadata_date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "latitude": {
              "type": "text"
            },
            "longitude": {
              "type": "text"
            },
            "altitude": {
              "type": "text"
            },
            "rating": {
              "type": "byte"
            },
            "comments": {
              "type": "text"
            }
          }
        },
        "path": {
          "properties": {
            "real": {
              "type": "keyword",
              "fields": {
                "tree": {
                  "type": "text",
                  "analyzer": "fscrawler_path",
                  "fielddata": true
                },
                "fulltext": {
                  "type": "text"
                }
              }
            },
            "root": {
              "type": "keyword"
            },
            "virtual": {
              "type": "keyword",
              "fields": {
                "tree": {
                  "type": "text",
                  "analyzer": "fscrawler_path",
                  "fielddata": true
                },
                "fulltext": {
                  "type": "text"
                }
              }
            }
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Tarik\_Bouhasse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tarik_bouhasse/32/79281_2.png) [@Tarik\_Bouhasse](https://discuss.elastic.co/u/Tarik_Bouhasse)\
**Post date:** [September 13, 2021, 11:26am UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/3 "2021-09-13T11:26:32Z")

</div>

Thank you

---

<div class="post-metadata">

**Author:** ![Tarik\_Bouhasse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tarik_bouhasse/32/79281_2.png) [@Tarik\_Bouhasse](https://discuss.elastic.co/u/Tarik_Bouhasse)\
**Post date:** [September 17, 2021, 12:23pm UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/4 "2021-09-17T12:23:39Z")

</div>

Hello,  
i'm beginner in elasticsearch and fscrawler, i can't configure stopwords i use elasticsearch 7.14 and fscrawler2.7, can you help me please?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 17, 2021, 7:51pm UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/5 "2021-09-17T19:51:49Z")

</div>

I think that's what I did already.

What is the problem?

---

<div class="post-metadata">

**Author:** ![Tarik\_Bouhasse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tarik_bouhasse/32/79281_2.png) [@Tarik\_Bouhasse](https://discuss.elastic.co/u/Tarik_Bouhasse)\
**Post date:** [September 17, 2021, 8:36pm UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/6 "2021-09-17T20:36:35Z")

</div>

the problem is  
{  
"error" : {  
"root\_cause" : [  
{  
"type" : "illegal\_argument\_exception",  
"reason" : "The mapping definition cannot be nested under a type [\_doc] unless include\_type\_name is set to true."  
}  
],  
"type" : "illegal\_argument\_exception",  
"reason" : "The mapping definition cannot be nested under a type [\_doc] unless include\_type\_name is set to true."  
},  
"status" : 400  
}

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 18, 2021, 1:29am UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/7 "2021-09-18T01:29:49Z")

</div>

Could you try:

```auto
PUT /docs
{
  "settings": {
    "number_of_shards": 1,
    "index.mapping.total_fields.limit": 2000,
    "analysis": {
      "tokenizer": {
        "fscrawler_path": {
          "type": "path_hierarchy"
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": ["_english_", "fscrawler"]
        },
        "english_keywords": {
          "type": "keyword_marker",
          "keywords": ["example"] 
        },
        "english_stemmer": {
          "type": "stemmer",
          "language": "english"
        },
        "english_possessive_stemmer": {
          "type": "stemmer",
          "language": "possessive_english"
        }
      },
      "analyzer": {
        "fscrawler_path": {
          "tokenizer": "fscrawler_path"
        },
        "rebuilt_english": {
          "tokenizer": "standard",
          "filter": [
            "english_possessive_stemmer",
            "lowercase",
            "english_stop",
            "english_keywords",
            "english_stemmer"
          ]
        }
      }
    }
  },
  "mappings": {
      "dynamic_templates": [
        {
          "raw_as_text": {
            "path_match": "meta.raw.*",
            "mapping": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ],
      "properties": {
        "attachment": {
          "type": "binary",
          "doc_values": false
        },
        "attributes": {
          "properties": {
            "group": {
              "type": "keyword"
            },
            "owner": {
              "type": "keyword"
            }
          }
        },
        "content": {
          "type": "text",
          "analyzer": "rebuilt_english"
        },
        "file": {
          "properties": {
            "content_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword",
              "store": true
            },
            "extension": {
              "type": "keyword"
            },
            "filesize": {
              "type": "long"
            },
            "indexed_chars": {
              "type": "long"
            },
            "indexing_date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "created": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "last_modified": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "last_accessed": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "checksum": {
              "type": "keyword"
            },
            "url": {
              "type": "keyword",
              "index": false
            }
          }
        },
        "meta": {
          "properties": {
            "author": {
              "type": "text"
            },
            "date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "keywords": {
              "type": "text"
            },
            "title": {
              "type": "text"
            },
            "language": {
              "type": "keyword"
            },
            "format": {
              "type": "text"
            },
            "identifier": {
              "type": "text"
            },
            "contributor": {
              "type": "text"
            },
            "coverage": {
              "type": "text"
            },
            "modifier": {
              "type": "text"
            },
            "creator_tool": {
              "type": "keyword"
            },
            "publisher": {
              "type": "text"
            },
            "relation": {
              "type": "text"
            },
            "rights": {
              "type": "text"
            },
            "source": {
              "type": "text"
            },
            "type": {
              "type": "text"
            },
            "description": {
              "type": "text"
            },
            "created": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "print_date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "metadata_date": {
              "type": "date",
              "format": "dateOptionalTime"
            },
            "latitude": {
              "type": "text"
            },
            "longitude": {
              "type": "text"
            },
            "altitude": {
              "type": "text"
            },
            "rating": {
              "type": "byte"
            },
            "comments": {
              "type": "text"
            }
          }
        },
        "path": {
          "properties": {
            "real": {
              "type": "keyword",
              "fields": {
                "tree": {
                  "type": "text",
                  "analyzer": "fscrawler_path",
                  "fielddata": true
                },
                "fulltext": {
                  "type": "text"
                }
              }
            },
            "root": {
              "type": "keyword"
            },
            "virtual": {
              "type": "keyword",
              "fields": {
                "tree": {
                  "type": "text",
                  "analyzer": "fscrawler_path",
                  "fielddata": true
                },
                "fulltext": {
                  "type": "text"
                }
              }
            }
          }
        }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 16, 2021, 1:30am UTC](https://discuss.elastic.co/t/stopwords-elasticsearch7-14-fscrawler2-7/284053/8 "2021-10-16T01:30:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.