# Skip indexing based on field value **URL:** \ **Category:** Elasticsearch\ **Created:** [November 11, 2017, 5:49pm UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280 "2017-11-11T17:49:42Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![beng](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@beng](https://discuss.elastic.co/u/beng)\ **Post date:** [November 11, 2017, 5:49pm UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/1 "2017-11-11T17:49:42Z")
We're doing some nested indexing on an events property. Some of the events we don't need in elasticsearch and are causing for a large amount of unneeded internal documents. Is there a way to skip indexing a field based on it's value? ie: if events.type = foo, skip indexing this I've been trying to find a solution all morning but no luck. Thanks in advance for the help 🙂 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 11, 2017, 6:09pm UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/2 "2017-11-11T18:09:48Z")
That's something you need to fix in the ingest layer. If you are using logstash, it's easy to call the drop filter. @magnusbaeck wrote a nice example here: > [@Creating drop filters based on a string search in a message field](https://discuss.elastic.co/t/creating-drop-filters-based-on-a-string-search-in-a-message-field/64050/7): > > There's nothing wrong with the filter. Perhaps you're not running with the configuration you expect? $ cat test.config input { stdin { } } output { stdout { codec =\> rubydebug } } filter { if "ASA-6-302013" in [message] { drop{ } } } $ ( echo 'first message' ; echo 'Oct 27 12:43:28 asa10 : %ASA-6-302013: Built inbound TCP connection 1033541997 for outside:10.150.0.0./46742 (10.150.0.0/46742) to inside.customer:172.26.0.0/8080 (161.215.0.0/8080)' ; echo 'third message' ) | /opt/logstash/bin/l… ---
**Author:** ![beng](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@beng](https://discuss.elastic.co/u/beng)\ **Post date:** [November 11, 2017, 6:33pm UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/3 "2017-11-11T18:33:53Z")
@dadoonet Thank you for the quick reply. As for the ingest layer, I do see the ability to drop a field at [https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html), is there a way to drop based on the field value? We're currently using the couchbase Elasticsearch connector found at [https://github.com/couchbaselabs/elasticsearch-transport-couchbase](https://github.com/couchbaselabs/elasticsearch-transport-couchbase) to transport data from CB into ES from our application so we can have better search/reporting capabilities for our customers. We've already requested they add support for pipelines so we can route documents to an index based off the created\_at field, and hope to have it within the next 1-2 weeks. From what I understand, we're pretty blocked until we can specify a pipeline to use the ingest layer? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 11, 2017, 6:53pm UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/4 "2017-11-11T18:53:29Z")
remove processor removes a field. It does not drop the document by itself. If I understood you want the later. So you need to do that in another way. I don't think there is something doable like this. May be an ingest script though which test the field value and generates then an exception which will "fail" the document. @talevy has may be more ideas? Or read from LS if possible the content of CB... Unsure if it's feasible easily. ---
**Author:** ![beng](https://avatars.discourse-cdn.com/v4/letter/b/b3f665/32.png) [@beng](https://discuss.elastic.co/u/beng)\ **Post date:** [November 11, 2017, 7:17pm UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/5 "2017-11-11T19:17:46Z")
Correct. We’re basically looking to drop that specific nested doc.event where doc.event.type = X from indexing, but keep the entire doc and the rest of the events. For more insight, we have 200k individual docs, but due to the nested events on doc.events it translates to 9m docs in the index. If we could ignore a specific event.type it would shed millions off. It seems Ultimately we’ll likely just have to change our data model. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 12, 2017, 12:34am UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/6 "2017-11-12T00:34:24Z")
Can’t you do that as a view in couchbase and replicate that view? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 12, 2017, 3:19am UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/7 "2017-11-12T03:19:34Z")
> [@beng](#): > > is there a way to drop based on the field value? Using Logstash you can with a conditional; ```auto if [fieldname] == "value" { drop{ } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 10, 2017, 3:19am UTC](https://discuss.elastic.co/t/skip-indexing-based-on-field-value/107280/8 "2017-12-10T03:19:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.