# Junk content detection **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2014, 2:58pm UTC](https://discuss.elastic.co/t/junk-content-detection/20400 "2014-10-23T14:58:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Igor\_Kupczynski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_kupczynski/32/1157_2.png) [@Igor\_Kupczynski](https://discuss.elastic.co/u/Igor_Kupczynski)\ **Post date:** [October 23, 2014, 2:58pm UTC](https://discuss.elastic.co/t/junk-content-detection/20400/1 "2014-10-23T14:58:03Z")
Hello Elasticsearch Community, We index content of some files. We use apache tika to extract the content. What I'm worried about is that some of the documents contain "junk" content, like a lot of numbers in excel. In such a case we'll pollute the indexing with many tokens, but they'll no useful at all as nobody will search for them. Similar thing if someone pastes binary data into a text file. Is there a good way (in es or external) to detect if a content may be "junk"? Thanks, Igor -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/24e84e90-0569-45d9-ba6f-1974970bc0da%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/24e84e90-0569-45d9-ba6f-1974970bc0da%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mark\_Harwood\_2](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Mark\_Harwood\_2](https://discuss.elastic.co/u/Mark_Harwood_2)\ **Post date:** [October 23, 2014, 3:17pm UTC](https://discuss.elastic.co/t/junk-content-detection/20400/2 "2014-10-23T15:17:23Z")
Possibly useful: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Thursday, October 23, 2014 3:58:03 PM UTC+1, Igor Kupczyński wrote: > Hello Elasticsearch Community, > > We index content of some files. We use apache tika to extract the content. > What I'm worried about is that some of the documents contain "junk" > content, like a lot of numbers in excel. In such a case we'll pollute the > indexing with many tokens, but they'll no useful at all as nobody will > search for them. Similar thing if someone pastes binary data into a text > file. > > Is there a good way (in es or external) to detect if a content may be > "junk"? > > Thanks, > Igor -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8a0690d2-dbe8-424e-8c31-28b340035c5b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8a0690d2-dbe8-424e-8c31-28b340035c5b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![tallison](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@tallison](https://discuss.elastic.co/u/tallison)\ **Post date:** [March 28, 2017, 11:22am UTC](https://discuss.elastic.co/t/junk-content-detection/20400/3 "2017-03-28T11:22:38Z")
We'd love to offer a junk detector w/in Tika see [TIKA-1443](https://issues.apache.org/jira/browse/TIKA-1443). However, the challenge of making something work across all languages, file formats and genres is daunting. In our soon to be released tika-eval module, we've added "common words" by taking the top 20k most frequent words per language in wikipedia dumps. If you divide that by the number of words with alphabetic/ideographic characters, that offers some insight into the junk-i-ness of the text. So, junk is hard. Numbers should be easy. PatternReplaceFilter, perhaps? ---
**Author:** ![igor\_k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_k/32/1157_2.png) [@igor\_k](https://discuss.elastic.co/u/igor_k)\ **Post date:** [March 30, 2017, 8:04am UTC](https://discuss.elastic.co/t/junk-content-detection/20400/4 "2017-03-30T08:04:22Z")
Hi Tim, Thanks for the answer. It is 2 years after the original question and I've moved to a different opportunity, But the project I had in mind at that time is up and running so I'll forward TIKA-1443 and your message to them. Thanks, Igor ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:01pm UTC](https://discuss.elastic.co/t/junk-content-detection/20400/5 "2017-07-05T22:01:42Z")