# Ignore Hate temrs

**URL:** <https://discuss.elastic.co/t/ignore-hate-temrs/11148>\
**Category:** Elasticsearch\
**Created:** [March 14, 2013, 7:22am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148 "2013-03-14T07:22:58Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![hemantsingal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hemantsingal/32/2442_2.png) [@hemantsingal](https://discuss.elastic.co/u/hemantsingal)\
**Post date:** [March 14, 2013, 7:22am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/1 "2013-03-14T07:22:58Z")

</div>

Is there a way to Ignore documents containing hate terms like fag**t,  
Ni**er etc from output of my search without having to specify them in each  
and every query.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 14, 2013, 7:41am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/2 "2013-03-14T07:41:47Z")

</div>

Hey,

check

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

You can ignore any words at index time (note: they will still be in the  
source of your document, if a search hit is returned).

On Thu, Mar 14, 2013 at 8:22 AM, cavebird [hemants.iitk@gmail.com](mailto:hemants.iitk@gmail.com) wrote:

> Is there a way to Ignore documents containing hate terms like fag**t,  
> Ni**er etc from output of my search without having to specify them in each  
> and every query.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![hemantsingal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hemantsingal/32/2442_2.png) [@hemantsingal](https://discuss.elastic.co/u/hemantsingal)\
**Post date:** [March 14, 2013, 7:51am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/3 "2013-03-14T07:51:13Z")

</div>

Not indexing them is good but I really can't show these documents so I  
still have to put these terms in every query.

On Thursday, March 14, 2013 1:11:47 PM UTC+5:30, Alexander Reelsen wrote:

> Hey,
> 
> check  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/stop-tokenfilter.html)  
> You can ignore any words at index time (note: they will still be in the  
> source of your document, if a search hit is returned).
> 
> On Thu, Mar 14, 2013 at 8:22 AM, cavebird \<[hemant...@gmail.com](mailto:hemant...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > Is there a way to Ignore documents containing hate terms like fag**t,  
> > Ni**er etc from output of my search without having to specify them in each  
> > and every query.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_G\_Ortega](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_g_ortega/32/46359_2.png) [@David\_G\_Ortega](https://discuss.elastic.co/u/David_G_Ortega)\
**Post date:** [March 14, 2013, 10:05am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/4 "2013-03-14T10:05:53Z")

</div>

Hi, would you like to post it in StackOverflow? Personally I would prefer  
to answer there

On Thursday, March 14, 2013 8:51:13 AM UTC+1, cavebird wrote:

> Not indexing them is good but I really can't show these documents so I  
> still have to put these terms in every query.
> 
> On Thursday, March 14, 2013 1:11:47 PM UTC+5:30, Alexander Reelsen wrote:
> 
> > Hey,
> > 
> > check  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/stop-tokenfilter.html)  
> > You can ignore any words at index time (note: they will still be in the  
> > source of your document, if a search hit is returned).
> > 
> > On Thu, Mar 14, 2013 at 8:22 AM, cavebird [hemant...@gmail.com](mailto:hemant...@gmail.com) wrote:
> > 
> > > Is there a way to Ignore documents containing hate terms like fag**t,  
> > > Ni**er etc from output of my search without having to specify them in each  
> > > and every query.
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\
**Post date:** [March 14, 2013, 10:22am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/5 "2013-03-14T10:22:43Z")

</div>

Why dont you make sure that such documents are not indexed. Or at-least  
periodically run a delete by query on all the documents which needs to be  
black listed.

Thanks  
Vineeth

On Thu, Mar 14, 2013 at 3:35 PM, David G Ortega [g.ortega.david@gmail.com](mailto:g.ortega.david@gmail.com)wrote:

> Hi, would you like to post it in StackOverflow? Personally I would prefer  
> to answer there
> 
> On Thursday, March 14, 2013 8:51:13 AM UTC+1, cavebird wrote:
> 
> > Not indexing them is good but I really can't show these documents so I  
> > still have to put these terms in every query.
> > 
> > On Thursday, March 14, 2013 1:11:47 PM UTC+5:30, Alexander Reelsen wrote:
> > 
> > > Hey,
> > > 
> > > check [http://www](http://www).\*\*[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://elasticsearch.org/guide/) **reference/index-modules/**  
> > > analysis/stop-tokenfilter.html[http://www.elasticsearch.org/guide/reference/index-modules/analysis/stop-tokenfilter.html](http://www.elasticsearch.org/guide/reference/index-modules/analysis/stop-tokenfilter.html)  
> > > You can ignore any words at index time (note: they will still be in the  
> > > source of your document, if a search hit is returned).
> > > 
> > > On Thu, Mar 14, 2013 at 8:22 AM, cavebird [hemant...@gmail.com](mailto:hemant...@gmail.com) wrote:
> > > 
> > > > Is there a way to Ignore documents containing hate terms like fag**t,  
> > > > Ni**er etc from output of my search without having to specify them in each  
> > > > and every query.
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to elasticsearc...@googlegroups.\*\*com.  
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > > .
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![hemantsingal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hemantsingal/32/2442_2.png) [@hemantsingal](https://discuss.elastic.co/u/hemantsingal)\
**Post date:** [March 14, 2013, 10:26am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/6 "2013-03-14T10:26:42Z")

</div>

Why dont you make sure that such documents are not indexed.

- How do I do that?

Or at-least periodically run a delete by query on all the documents which  
needs to be black listed.

- Deletes are expensive and not real time.

On Thursday, March 14, 2013 3:52:43 PM UTC+5:30, Vineeth Mohan wrote:

> Why dont you make sure that such documents are not indexed. Or at-least  
> periodically run a delete by query on all the documents which needs to be  
> black listed.
> 
> Thanks  
> Vineeth
> 
> On Thu, Mar 14, 2013 at 3:35 PM, David G Ortega \<[g.orteg...@gmail.com](mailto:g.orteg...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > Hi, would you like to post it in StackOverflow? Personally I would prefer  
> > to answer there
> > 
> > On Thursday, March 14, 2013 8:51:13 AM UTC+1, cavebird wrote:
> > 
> > > Not indexing them is good but I really can't show these documents so I  
> > > still have to put these terms in every query.
> > > 
> > > On Thursday, March 14, 2013 1:11:47 PM UTC+5:30, Alexander Reelsen wrote:
> > > 
> > > > Hey,
> > > > 
> > > > check [http://www](http://www).\*\*[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://elasticsearch.org/guide/) **reference/index-modules/**  
> > > > analysis/stop-tokenfilter.html[http://www.elasticsearch.org/guide/reference/index-modules/analysis/stop-tokenfilter.html](http://www.elasticsearch.org/guide/reference/index-modules/analysis/stop-tokenfilter.html)  
> > > > You can ignore any words at index time (note: they will still be in the  
> > > > source of your document, if a search hit is returned).
> > > > 
> > > > On Thu, Mar 14, 2013 at 8:22 AM, cavebird [hemant...@gmail.com](mailto:hemant...@gmail.com) wrote:
> > > > 
> > > > > Is there a way to Ignore documents containing hate terms like fag**t,  
> > > > > Ni**er etc from output of my search without having to specify them in each  
> > > > > and every query.
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to elasticsearc...@googlegroups.\*\*com.  
> > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > > > .
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google Groups  
> > > > "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [March 14, 2013, 11:29am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/7 "2013-03-14T11:29:22Z")

</div>

On Thu, 2013-03-14 at 03:26 -0700, cavebird wrote:

> Why dont you make sure that such documents are not indexed.
> 
> - How do I do that?

Check them in your application before you index them

> 

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![hemantsingal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hemantsingal/32/2442_2.png) [@hemantsingal](https://discuss.elastic.co/u/hemantsingal)\
**Post date:** [March 14, 2013, 11:39am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/8 "2013-03-14T11:39:20Z")

</div>

Well the data comes into system from multiple entry points and through  
different stacks as well (RoR, Java).  
Also, I would like the ability to modify terms list in the future.

On Thursday, March 14, 2013 4:59:22 PM UTC+5:30, Clinton Gormley wrote:

> On Thu, 2013-03-14 at 03:26 -0700, cavebird wrote:
> 
> > Why dont you make sure that such documents are not indexed.
> > 
> > - How do I do that?
> 
> Check them in your application before you index them
> 
> >

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_G\_Ortega](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_g_ortega/32/46359_2.png) [@David\_G\_Ortega](https://discuss.elastic.co/u/David_G_Ortega)\
**Post date:** [March 14, 2013, 11:47am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/9 "2013-03-14T11:47:31Z")

</div>

Just in case you wanna have a 100% ES solution or just in case you want to  
have all your data available  
here you have a possible solution:

1. Create an analyzer that transforms all your words into the same token

{  
"index" : {  
"analysis" : {  
"char\_filter" : {  
"my\_mapping" : {  
"type" : "mapping",  
"mappings" : ["badword1=\>bad", "badword2=\>bad"]  
}  
},  
"analyzer" : {  
"isBad" : {  
"tokenizer" : "standard",  
"filter" : ["lowercase", "asciifolding", "unique"]  
"char\_filter" : ["my\_mapping"]  
},  
}  
}  
}  
}

1. Set your field or fields as multi\_field setting isBad with your isBad  
analyzer

"myTextField" :  
{  
"type" : "multi\_field",  
"fields" :  
{  
"myTextField" : { "type" : "string"},  
"isBad" : { "type" : "string", "index\_analyzer" : "isBad"},  
}  
}

1. Search filtering

{  
"from" : 0,  
"size" : 10,  
"query" :  
{  
"filtered" :  
{  
"query" :  
{  
"match\_all" : { }  
},  
"filter" :  
{  
"not" : { {"term" : { "myTextField.isBad" : "bad" } } }  
}  
}  
}  
}

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![hemantsingal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hemantsingal/32/2442_2.png) [@hemantsingal](https://discuss.elastic.co/u/hemantsingal)\
**Post date:** [March 14, 2013, 11:52am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/10 "2013-03-14T11:52:11Z")

</div>

I will definitely try this and get back to you. 🙂

On Thursday, March 14, 2013 5:17:31 PM UTC+5:30, David G Ortega wrote:

> Just in case you wanna have a 100% ES solution or just in case you want to  
> have all your data available  
> here you have a possible solution:
> 
> 1. Create an analyzer that transforms all your words into the same token
> 
> {  
> "index" : {  
> "analysis" : {  
> "char\_filter" : {  
> "my\_mapping" : {  
> "type" : "mapping",  
> "mappings" : ["badword1=\>bad", "badword2=\>bad"]  
> }  
> },  
> "analyzer" : {  
> "isBad" : {  
> "tokenizer" : "standard",  
> "filter" : ["lowercase", "asciifolding", "unique"]  
> "char\_filter" : ["my\_mapping"]  
> },  
> }  
> }  
> }  
> }
> 
> 1. Set your field or fields as multi\_field setting isBad with your isBad  
> analyzer
> 
> "myTextField" :  
> {  
> "type" : "multi\_field",  
> "fields" :  
> {  
> "myTextField" : { "type" : "string"},  
> "isBad" : { "type" : "string", "index\_analyzer" : "isBad"},  
> }  
> }
> 
> 1. Search filtering
> 
> {  
> "from" : 0,  
> "size" : 10,  
> "query" :  
> {  
> "filtered" :  
> {  
> "query" :  
> {  
> "match\_all" : { }  
> },  
> "filter" :  
> {  
> "not" : { {"term" : { "myTextField.isBad" : "bad" } } }  
> }  
> }  
> }  
> }

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\
**Post date:** [March 14, 2013, 12:36pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/11 "2013-03-14T12:36:42Z")

</div>

I am not getting the whole concept here.  
For this to work , shouldnt i write the text both to myTextField and isBad  
field ?

Thanks  
Vineeth

On Thu, Mar 14, 2013 at 5:22 PM, cavebird [hemants.iitk@gmail.com](mailto:hemants.iitk@gmail.com) wrote:

> I will definitely try this and get back to you. 🙂
> 
> On Thursday, March 14, 2013 5:17:31 PM UTC+5:30, David G Ortega wrote:
> 
> > Just in case you wanna have a 100% ES solution or just in case you want  
> > to have all your data available  
> > here you have a possible solution:
> > 
> > 1. Create an analyzer that transforms all your words into the same token
> > 
> > {  
> > "index" : {  
> > "analysis" : {  
> > "char\_filter" : {  
> > "my\_mapping" : {  
> > "type" : "mapping",  
> > "mappings" : ["badword1=\>bad", "badword2=\>bad"]  
> > }  
> > },  
> > "analyzer" : {  
> > "isBad" : {  
> > "tokenizer" : "standard",  
> > "filter" : ["lowercase", "asciifolding", "unique"]  
> > "char\_filter" : ["my\_mapping"]  
> > },  
> > }  
> > }  
> > }  
> > }
> > 
> > 1. Set your field or fields as multi\_field setting isBad with your isBad  
> > analyzer
> > 
> > "myTextField" :  
> > {  
> > "type" : "multi\_field",  
> > "fields" :  
> > {  
> > "myTextField" : { "type" : "string"},  
> > "isBad" : { "type" : "string", "index\_analyzer" : "isBad"},  
> > }  
> > }
> > 
> > 1. Search filtering
> > 
> > {  
> > "from" : 0,  
> > "size" : 10,  
> > "query" :  
> > {  
> > "filtered" :  
> > {  
> > "query" :  
> > {  
> > "match\_all" : { }  
> > },  
> > "filter" :  
> > {  
> > "not" : { {"term" : { "myTextField.isBad" : "bad" } } }  
> > }  
> > }  
> > }  
> > }
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_G\_Ortega](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_g_ortega/32/46359_2.png) [@David\_G\_Ortega](https://discuss.elastic.co/u/David_G_Ortega)\
**Post date:** [March 14, 2013, 1:21pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/12 "2013-03-14T13:21:17Z")

</div>

When you set a multi\_field and you send a document with that field name ES  
internally creates the multi\_field using the mapping deffinition. What is  
going to happen is this:

You send:  
{text: "this is a badword1 text"}

In ES:  
{text.text: [this, is, a, badword1, text]}  
{text.isBad: [this, is, a, bad, text]}

Oviously "bad" is so much generic word, is better to have something like  
tagFlagged instead of "bad" like in the mapping so in another example with  
this tagFlagged this is going to happen

You send:  
{text: "this is a badword1, badword2, badword3 text"}

In ES:  
{text.text: [this, is, a, badword1, badword2, badword3, text]}  
{text.isBad: [this, is, a, tagFlagged, text]} (lowercase, unique)

since you are filtering in the search to not have the term tagFlagged in  
text.isBad, no posts flagged are going to appear

> 

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\
**Post date:** [March 14, 2013, 1:32pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/13 "2013-03-14T13:32:36Z")

</div>

Also you should consider the impacts of false positives on your system.  
Take the following phrase from The Hobbit - "The faggots are reeking".  
Perhaps the elves are homophobic but research shows that they are just  
admiring burning wood.

> **[The Hobbit](https://books.google.com/books?id=hFfhrCWiLSMC&pg=PA48&lpg=PA48#v=onepage&q&f=false)**
>
> The stirring adventure that begins The Lord of the Rings, the greatest fantasy epic of all timeWhen Thorin Oakenshield and his band of dwarves embark upon a dangerous quest to reclaim the hoard of gold stolen from them by the evil dragon Smaug,...

Since analysis for context and sentiment is difficult, you might setup a  
system for review where the words that you are trying to exclude change a  
state filed, something like: censorStatus=ok,review,notOk so that on most  
reads you only retrieve the "ok" value and some stewards review the posts  
that require it and either allow or disallow. Without knowing the context  
of your system, not sure how likely it is that you need to care but if you  
do you'll find that being "smart" about the exclusions can be a pain.

On Thu, Mar 14, 2013 at 9:21 AM, David G Ortega [g.ortega.david@gmail.com](mailto:g.ortega.david@gmail.com)wrote:

> When you set a multi\_field and you send a document with that field name ES  
> internally creates the multi\_field using the mapping deffinition. What is  
> going to happen is this:
> 
> You send:  
> {text: "this is a badword1 text"}
> 
> In ES:  
> {text.text: [this, is, a, badword1, text]}  
> {text.isBad: [this, is, a, bad, text]}
> 
> Oviously "bad" is so much generic word, is better to have something like  
> tagFlagged instead of "bad" like in the mapping so in another example with  
> this tagFlagged this is going to happen
> 
> You send:  
> {text: "this is a badword1, badword2, badword3 text"}
> 
> In ES:  
> {text.text: [this, is, a, badword1, badword2, badword3, text]}  
> {text.isBad: [this, is, a, tagFlagged, text]} (lowercase, unique)
> 
> since you are filtering in the search to not have the term tagFlagged in  
> text.isBad, no posts flagged are going to appear
> 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![mrflip](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@mrflip](https://discuss.elastic.co/u/mrflip)\
**Post date:** [March 14, 2013, 2:13pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/14 "2013-03-14T14:13:23Z")

</div>

You may enjoy(?) the lists of obscene words I've gathered here: [Our team - Website.Design](http://www.infochimps.com/datasets/list-of-dirty-obscene-banned-and-otherwise-unacceptable-words)

I believe this kind of thing -- for which regexps and lookup tables fall short, yet for which proper NLP is too much work -- is perfect for the percolate feature.

As you index each document, percolate against rule sets as complex or simple-term-matchy as you like, and tag documents with a "probably offensive" flag. Now exclude such altogether, or let visitors opt in/out to flagged documents.

Flip

Sent from my iPad

On Mar 14, 2013, at 9:32 AM, Michael Sick [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com) wrote:

> Also you should consider the impacts of false positives on your system. Take the following phrase from The Hobbit - "The faggots are reeking". Perhaps the elves are homophobic but research shows that they are just admiring burning wood.
> 
> [The Hobbit: The Enchanting Prelude to The Lord of the Rings - J.R.R. Tolkien - Google Books](http://books.google.com/books?id=hFfhrCWiLSMC&pg=PA48&lpg=PA48#v=onepage&q&f=false)
> 
> Since analysis for context and sentiment is difficult, you might setup a system for review where the words that you are trying to exclude change a state filed, something like: censorStatus=ok,review,notOk so that on most reads you only retrieve the "ok" value and some stewards review the posts that require it and either allow or disallow. Without knowing the context of your system, not sure how likely it is that you need to care but if you do you'll find that being "smart" about the exclusions can be a pain.
> 
> On Thu, Mar 14, 2013 at 9:21 AM, David G Ortega [g.ortega.david@gmail.com](mailto:g.ortega.david@gmail.com) wrote:
> 
> > When you set a multi\_field and you send a document with that field name ES internally creates the multi\_field using the mapping deffinition. What is going to happen is this:
> > 
> > You send:  
> > {text: "this is a badword1 text"}
> > 
> > In ES:  
> > {text.text: [this, is, a, badword1, text]}  
> > {text.isBad: [this, is, a, bad, text]}
> > 
> > Oviously "bad" is so much generic word, is better to have something like tagFlagged instead of "bad" like in the mapping so in another example with this tagFlagged this is going to happen
> > 
> > You send:  
> > {text: "this is a badword1, badword2, badword3 text"}
> > 
> > In ES:  
> > {text.text: [this, is, a, badword1, badword2, badword3, text]}  
> > {text.isBad: [this, is, a, tagFlagged, text]} (lowercase, unique)
> > 
> > since you are filtering in the search to not have the term tagFlagged in text.isBad, no posts flagged are going to appear
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_G\_Ortega](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_g_ortega/32/46359_2.png) [@David\_G\_Ortega](https://discuss.elastic.co/u/David_G_Ortega)\
**Post date:** [March 14, 2013, 2:23pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/15 "2013-03-14T14:23:43Z")

</div>

I think that we both do the same since we both flag the doc but I love your  
solution Flip. Brillaint.

On Thursday, March 14, 2013 3:13:23 PM UTC+1, Philip (Flip) Kromer wrote:

> You may enjoy(?) the lists of obscene words I've gathered here:  
> [Our team - Website.Design](http://www.infochimps.com/datasets/list-of-dirty-obscene-banned-and-otherwise-unacceptable-words)
> 
> I believe this kind of thing -- for which regexps and lookup tables fall  
> short, yet for which proper NLP is too much work -- is perfect for the  
> percolate feature.
> 
> As you index each document, percolate against rule sets as complex or  
> simple-term-matchy as you like, and tag documents with a "probably  
> offensive" flag. Now exclude such altogether, or let visitors opt in/out to  
> flagged documents.
> 
> Flip
> 
> Sent from my iPad
> 
> On Mar 14, 2013, at 9:32 AM, Michael Sick \<[michae...@serenesoftware.com](mailto:michae...@serenesoftware.com)\<javascript:\>\>  
> wrote:
> 
> Also you should consider the impacts of false positives on your system.  
> Take the following phrase from The Hobbit - "The faggots are reeking".  
> Perhaps the elves are homophobic but research shows that they are just  
> admiring burning wood.
> 
> [The Hobbit: The Enchanting Prelude to The Lord of the Rings - J.R.R. Tolkien - Google Books](http://books.google.com/books?id=hFfhrCWiLSMC&pg=PA48&lpg=PA48#v=onepage&q&f=false)
> 
> Since analysis for context and sentiment is difficult, you might setup a  
> system for review where the words that you are trying to exclude change a  
> state filed, something like: censorStatus=ok,review,notOk so that on most  
> reads you only retrieve the "ok" value and some stewards review the posts  
> that require it and either allow or disallow. Without knowing the context  
> of your system, not sure how likely it is that you need to care but if you  
> do you'll find that being "smart" about the exclusions can be a pain.
> 
> On Thu, Mar 14, 2013 at 9:21 AM, David G Ortega \<[g.orteg...@gmail.com](mailto:g.orteg...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > When you set a multi\_field and you send a document with that field name  
> > ES internally creates the multi\_field using the mapping deffinition. What  
> > is going to happen is this:
> > 
> > You send:  
> > {text: "this is a badword1 text"}
> > 
> > In ES:  
> > {text.text: [this, is, a, badword1, text]}  
> > {text.isBad: [this, is, a, bad, text]}
> > 
> > Oviously "bad" is so much generic word, is better to have something like  
> > tagFlagged instead of "bad" like in the mapping so in another example with  
> > this tagFlagged this is going to happen
> > 
> > You send:  
> > {text: "this is a badword1, badword2, badword3 text"}
> > 
> > In ES:  
> > {text.text: [this, is, a, badword1, badword2, badword3, text]}  
> > {text.isBad: [this, is, a, tagFlagged, text]} (lowercase, unique)
> > 
> > since you are filtering in the search to not have the term tagFlagged in  
> > text.isBad, no posts flagged are going to appear
> > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\
**Post date:** [March 14, 2013, 3:38pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/16 "2013-03-14T15:38:13Z")

</div>

Thanks David ,  
This is a handy piece of knowledge.

Thanks  
Vineeth

On Thu, Mar 14, 2013 at 7:53 PM, David G Ortega [g.ortega.david@gmail.com](mailto:g.ortega.david@gmail.com)wrote:

> I think that we both do the same since we both flag the doc but I love  
> your solution Flip. Brillaint.
> 
> On Thursday, March 14, 2013 3:13:23 PM UTC+1, Philip (Flip) Kromer wrote:
> 
> > You may enjoy(?) the lists of obscene words I've gathered here:  
> > [http://www.infochimps](http://www.infochimps). **com/datasets/list-of-dirty-**  
> > obscene-banned-and-otherwise-\*\*unacceptable-words[http://www.infochimps.com/datasets/list-of-dirty-obscene-banned-and-otherwise-unacceptable-words](http://www.infochimps.com/datasets/list-of-dirty-obscene-banned-and-otherwise-unacceptable-words)
> > 
> > I believe this kind of thing -- for which regexps and lookup tables fall  
> > short, yet for which proper NLP is too much work -- is perfect for the  
> > percolate feature.
> > 
> > As you index each document, percolate against rule sets as complex or  
> > simple-term-matchy as you like, and tag documents with a "probably  
> > offensive" flag. Now exclude such altogether, or let visitors opt in/out to  
> > flagged documents.
> > 
> > Flip
> > 
> > Sent from my iPad
> > 
> > On Mar 14, 2013, at 9:32 AM, Michael Sick \<michae...@serenesoftware.\*\*com\>  
> > wrote:
> > 
> > Also you should consider the impacts of false positives on your system.  
> > Take the following phrase from The Hobbit - "The faggots are reeking".  
> > Perhaps the elves are homophobic but research shows that they are just  
> > admiring burning wood.
> > 
> > [Google Books](http://books.google.com/books?**id=hFfhrCWiLSMC&pg=PA48&lpg=)\*\*  
> > PA48#v=onepage&q&f=false[http://books.google.com/books?id=hFfhrCWiLSMC&pg=PA48&lpg=PA48#v=onepage&q&f=false](http://books.google.com/books?id=hFfhrCWiLSMC&pg=PA48&lpg=PA48#v=onepage&q&f=false)
> > 
> > Since analysis for context and sentiment is difficult, you might setup a  
> > system for review where the words that you are trying to exclude change a  
> > state filed, something like: censorStatus=ok,review,notOk so that on most  
> > reads you only retrieve the "ok" value and some stewards review the posts  
> > that require it and either allow or disallow. Without knowing the context  
> > of your system, not sure how likely it is that you need to care but if you  
> > do you'll find that being "smart" about the exclusions can be a pain.
> > 
> > On Thu, Mar 14, 2013 at 9:21 AM, David G Ortega [g.orteg...@gmail.com](mailto:g.orteg...@gmail.com)wrote:
> > 
> > > When you set a multi\_field and you send a document with that field name  
> > > ES internally creates the multi\_field using the mapping deffinition. What  
> > > is going to happen is this:
> > > 
> > > You send:  
> > > {text: "this is a badword1 text"}
> > > 
> > > In ES:  
> > > {text.text: [this, is, a, badword1, text]}  
> > > {text.isBad: [this, is, a, bad, text]}
> > > 
> > > Oviously "bad" is so much generic word, is better to have something like  
> > > tagFlagged instead of "bad" like in the mapping so in another example with  
> > > this tagFlagged this is going to happen
> > > 
> > > You send:  
> > > {text: "this is a badword1, badword2, badword3 text"}
> > > 
> > > In ES:  
> > > {text.text: [this, is, a, badword1, badword2, badword3, text]}  
> > > {text.isBad: [this, is, a, tagFlagged, text]} (lowercase, unique)
> > > 
> > > since you are filtering in the search to not have the term tagFlagged in  
> > > text.isBad, no posts flagged are going to appear
> > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > 
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > .
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > 
> > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > .
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_G\_Ortega](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_g_ortega/32/46359_2.png) [@David\_G\_Ortega](https://discuss.elastic.co/u/David_G_Ortega)\
**Post date:** [March 14, 2013, 11:59pm UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/17 "2013-03-14T23:59:36Z")

</div>

You are welcome Vineeth 🙂

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![David\_Zachariah](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@David\_Zachariah](https://discuss.elastic.co/u/David_Zachariah)\
**Post date:** [March 20, 2013, 12:36am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/18 "2013-03-20T00:36:45Z")

</div>

Hi David,

Thanks for the useful information. For the censored words, if I have a huge lists of offensive words, do I need to list all of them in the "term" :{"badw1"=\>"bad", "badw2"=\>"bad", ..., "badw9999"=\>"bad"}, or is there another way doing this tedious task - as a file?

Second what is the performance of percolate? Is it acceptable to use it as sentiment analysis?

Thanks,

David

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:45am UTC](https://discuss.elastic.co/t/ignore-hate-temrs/11148/19 "2017-07-06T02:45:41Z")

</div>