# Validate data before indexing them **URL:** \ **Category:** Elasticsearch\ **Created:** [February 15, 2014, 9:40pm UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825 "2014-02-15T21:40:25Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Thierry\_Templier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thierry_templier/32/1745_2.png) [@Thierry\_Templier](https://discuss.elastic.co/u/Thierry_Templier)\ **Post date:** [February 15, 2014, 9:40pm UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825/1 "2014-02-15T21:40:25Z")
Hello, I wonder if there is a built-in way to validate data before indexing them. I see two kinds of validation: - Structural validation of fields based on a regular expression for example. Perhaps something can be configured in the mapping... - Integrity validation of document. For example preventing from indexing a document with a field value that already exists. In the case where there is no built-in support at the moment, is there a way to extend ElasticSearch to add such processing before indexing using the standard REST calls? Thanks very much for your help! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/52FFDEC9.7020007%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/52FFDEC9.7020007%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 15, 2014, 10:00pm UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825/2 "2014-02-15T22:00:37Z")
You can validate the data at client side in your model before serializing it to JSON, or after a complete bulk index run. There are reasons why Elasticsarch is schema-less. It is equivalent to allow any number of different fields (keys) and any content in fields (values) without any logical constraints. In a distributed system, commits per field, or transactions per field, or integrity checking can get very expensive. Because the index is inverted, and nodes can come and go, there is a significant penalty if you want document transaction safety and document integrity checks. I validate data in ES with the help o a large scan/scroll over the docs after bulk indexing, by searching for IDs if they exist or not. This is different from integrity constraint checking techniques like rule based methos known from RDBMs. Jörg On Sat, Feb 15, 2014 at 10:40 PM, Thierry Templier [templth@gmail.com](mailto:templth@gmail.com)wrote: > Hello, > > I wonder if there is a built-in way to validate data before indexing them. > I see two kinds of validation: > > - Structural validation of fields based on a regular expression for > example. Perhaps something can be configured in the mapping... > - Integrity validation of document. For example preventing from indexing a > document with a field value that already exists. > > In the case where there is no built-in support at the moment, is there a > way to extend Elasticsearch to add such processing before indexing using > the standard REST calls? > > Thanks very much for your help! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > msgid/elasticsearch/52FFDEC9.7020007%[40gmail.com](http://40gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGXXQxQ%2B5PRwrHw33uj3%2B8WwqLKiAZvnQrZ8bYUMfKYSw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGXXQxQ%2B5PRwrHw33uj3%2B8WwqLKiAZvnQrZ8bYUMfKYSw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Thierry\_Templier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thierry_templier/32/1745_2.png) [@Thierry\_Templier](https://discuss.elastic.co/u/Thierry_Templier)\ **Post date:** [February 17, 2014, 6:57am UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825/3 "2014-02-17T06:57:12Z")
Thanks very much, Jörg, for your answer! I see the approach... I understand well that having integrity checks in a schema-less engine like Elasticsearch isn't possible. However, would it be possible to have checks at field structure level before triggering indexing in the ES engine. Perhaps with specifying something like that in the mapping: { "mappings": { "mydoc": { "properties": { (...) "name": { "type":"string", "store":"yes", "index":"analyzed", "checks": "not\_null,not\_empty,regexp=\[1\]$" }, (...) } } } } Thanks very much for your help! Thierry > You can validate the data at client side in your model before > serializing it to JSON, or after a complete bulk index run. > > There are reasons why Elasticsarch is schema-less. It is equivalent to > allow any number of different fields (keys) and any content in fields > (values) without any logical constraints. > > In a distributed system, commits per field, or transactions per field, > or integrity checking can get very expensive. Because the index is > inverted, and nodes can come and go, there is a significant penalty if > you want document transaction safety and document integrity checks. > > I validate data in ES with the help o a large scan/scroll over the > docs after bulk indexing, by searching for IDs if they exist or not. > This is different from integrity constraint checking techniques like > rule based methos known from RDBMs. > > Jörg > > On Sat, Feb 15, 2014 at 10:40 PM, Thierry Templier \<[templth@gmail.com](mailto:templth@gmail.com) > [mailto:templth@gmail.com](mailto:templth@gmail.com)\> wrote: > > ``` > Hello, > > I wonder if there is a built-in way to validate data before > indexing them. I see two kinds of validation: > > * Structural validation of fields based on a regular expression > for example. Perhaps something can be configured in the mapping... > * Integrity validation of document. For example preventing from > indexing a document with a field value that already exists. > > In the case where there is no built-in support at the moment, is > there a way to extend ElasticSearch to add such processing before > indexing using the standard REST calls? > > Thanks very much for your help! > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to elasticsearch+unsubscribe@googlegroups.com > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/52FFDEC9.7020007%40gmail.com. > For more options, visit https://groups.google.com/groups/opt_out. > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGXXQxQ%2B5PRwrHw33uj3%2B8WwqLKiAZvnQrZ8bYUMfKYSw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGXXQxQ%2B5PRwrHw33uj3%2B8WwqLKiAZvnQrZ8bYUMfKYSw%40mail.gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5301B2C8.1060305%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5301B2C8.1060305%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). * * * 1. a-zA-Z ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 17, 2014, 9:08am UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825/4 "2014-02-17T09:08:04Z")
I think, if this must be server side, this could be done in a REST filter plugin, not in the mappings. Matching values against patterns do not relate to field mappings. Still, the best method to filter out unwanted values is at client side, either at JSON construction time in the official clients, or in the XContentBuilder calls in Java. Reason is to avoid extra load on the data nodes that is not related to index or search. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF6RcbsLsMewFiFgFSdHT%3Dq%2BP%2BteaHBjRSXDk8Or-JXrQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF6RcbsLsMewFiFgFSdHT%3Dq%2BP%2BteaHBjRSXDk8Or-JXrQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Thierry\_Templier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thierry_templier/32/1745_2.png) [@Thierry\_Templier](https://discuss.elastic.co/u/Thierry_Templier)\ **Post date:** [March 7, 2014, 4:19pm UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825/5 "2014-03-07T16:19:41Z")
Hi Jörg, Following our conversation, I implemented a data validation filter. I describe my approach at this address: [Implementing data validation in ElasticSearch | Sandbox for the Web stack](http://templth.wordpress.com/2014/03/07/implementing-data-validation-in-elasticsearch/). I'd be pleased to have your feedback and comments on this! Thanks very much! Thierry > I think, if this must be server side, this could be done in a REST > filter plugin, not in the mappings. Matching values against patterns > do not relate to field mappings. > > Still, the best method to filter out unwanted values is at client > side, either at JSON construction time in the official clients, or in > the XContentBuilder calls in Java. Reason is to avoid extra load on > the data nodes that is not related to index or search. > > Jörg > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF6RcbsLsMewFiFgFSdHT%3Dq%2BP%2BteaHBjRSXDk8Or-JXrQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF6RcbsLsMewFiFgFSdHT%3Dq%2BP%2BteaHBjRSXDk8Or-JXrQ%40mail.gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5319F19D.8070107%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5319F19D.8070107%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:44am UTC](https://discuss.elastic.co/t/validate-data-before-indexing-them/15825/6 "2017-07-06T01:44:51Z")