# How to extract the fields using regex? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 9, 2012, 2:26am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688 "2012-08-09T02:26:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\ **Post date:** [August 9, 2012, 2:26am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/1 "2012-08-09T02:26:27Z")
Hi, I want to extract fields using regex. for example, I have a doc {body: jackie's email is [xxx@gmail.com](mailto:xxx@gmail.com)}. I want to extract the email address from the body using a regex. I know that script fields seem for this. But I don't find any exact example in the API doc. Does anyone know how to use regex in script field? Thanks a lot. ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [August 9, 2012, 2:41am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/2 "2012-08-09T02:41:27Z")
So if I understand correctly, you have indexed documents with a single field say 'body' and you want to extract some content from that? Is this just for display purposes or this is for a search? Could you do it before the content was indexed or on the client side after the search? On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > Hi, > I want to extract fields using regex. for example, I have a doc {body: > jackie's email is [x...@gmail.com](mailto:x...@gmail.com) \}. I want to extract the > email address from the body using a regex. > I know that script fields seem for this. But I don't find any exact > example in the API doc. > Does anyone know how to use regex in script field? > Thanks a lot. ---
**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\ **Post date:** [August 9, 2012, 5:50am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/3 "2012-08-09T05:50:26Z")
Yes, I indexed the documents with a field "body", and then I want to extract some content from that. Not only for display, but also for search and statistics. 在 2012年8月9日星期四UTC+8上午10时41分27秒，Chris Male写道： > So if I understand correctly, you have indexed documents with a single > field say 'body' and you want to extract some content from that? Is this > just for display purposes or this is for a search? Could you do it before > the content was indexed or on the client side after the search? > > On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > > > Hi, > > I want to extract fields using regex. for example, I have a doc > > {body: jackie's email is [x...@gmail.com](mailto:x...@gmail.com)}. I want to extract the email > > address from the body using a regex. > > I know that script fields seem for this. But I don't find any exact > > example in the API doc. > > Does anyone know how to use regex in script field? > > Thanks a lot. ---
**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\ **Post date:** [August 9, 2012, 6:06am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/4 "2012-08-09T06:06:12Z")
In fact, the real purpose is extract the new fields from the "body" content, and add these extracted fields back into the original doc. 在 2012年8月9日星期四UTC+8下午1时50分26秒，jackiedong写道： > Yes, I indexed the documents with a field "body", and then I want to > extract some content from that. > Not only for display, but also for search and statistics. > > 在 2012年8月9日星期四UTC+8上午10时41分27秒，Chris Male写道： > > > So if I understand correctly, you have indexed documents with a single > > field say 'body' and you want to extract some content from that? Is this > > just for display purposes or this is for a search? Could you do it before > > the content was indexed or on the client side after the search? > > > > On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > > > > > Hi, > > > I want to extract fields using regex. for example, I have a doc > > > {body: jackie's email is [x...@gmail.com](mailto:x...@gmail.com)}. I want to extract the email > > > address from the body using a regex. > > > I know that script fields seem for this. But I don't find any exact > > > example in the API doc. > > > Does anyone know how to use regex in script field? > > > Thanks a lot. ---
**Author:** ![olof](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olof/32/2721_2.png) [@olof](https://discuss.elastic.co/u/olof)\ **Post date:** [August 9, 2012, 8:52am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/5 "2012-08-09T08:52:32Z")
I would say it is better to extract fields before indexing, for example in a document processing framework. Or simply do the content extraction in your indexing client. Do you want to do this for all documents or just the ones that a certain query finds? I suppose you could do the extraction in the search client and then do an \_update of the document with the new fields. This will be expensive in the long run though, since it will cause the document to be reindexed. You'll end up with lots of deletes in your index or, if you use versioning, double the index size. Den torsdagen den 9:e augusti 2012 kl. 08:06:12 UTC+2 skrev jackiedong: > In fact, the real purpose is extract the new fields from the "body" > content, and add these extracted fields back into the original doc. > > 在 2012年8月9日星期四UTC+8下午1时50分26秒，jackiedong写道： > > > Yes, I indexed the documents with a field "body", and then I want to > > extract some content from that. > > Not only for display, but also for search and statistics. > > > > 在 2012年8月9日星期四UTC+8上午10时41分27秒，Chris Male写道： > > > > > So if I understand correctly, you have indexed documents with a single > > > field say 'body' and you want to extract some content from that? Is this > > > just for display purposes or this is for a search? Could you do it before > > > the content was indexed or on the client side after the search? > > > > > > On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > > > > > > > Hi, > > > > I want to extract fields using regex. for example, I have a doc > > > > {body: jackie's email is [x...@gmail.com](mailto:x...@gmail.com)}. I want to extract the email > > > > address from the body using a regex. > > > > I know that script fields seem for this. But I don't find any exact > > > > example in the API doc. > > > > Does anyone know how to use regex in script field? > > > > Thanks a lot. ---
**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\ **Post date:** [August 9, 2012, 9:16am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/6 "2012-08-09T09:16:26Z")
I see this feature in the Splunk. Splunk allows the user defines his own regex to extract the fields from the existing docs. Then the user can use these new fields to do the search. I want to have the same feature like this. But from the above response, it seems it is expensive to do such thing in Elasticsearch. 在 2012年8月9日星期四UTC+8下午4时52分32秒，olof写道： > I would say it is better to extract fields before indexing, for example in > a document processing framework. Or simply do the content extraction in > your indexing client. > > Do you want to do this for all documents or just the ones that a certain > query finds? I suppose you could do the extraction in the search client and > then do an \_update of the document with the new fields. This will be > expensive in the long run though, since it will cause the document to be > reindexed. You'll end up with lots of deletes in your index or, if you use > versioning, double the index size. > > Den torsdagen den 9:e augusti 2012 kl. 08:06:12 UTC+2 skrev jackiedong: > > > In fact, the real purpose is extract the new fields from the "body" > > content, and add these extracted fields back into the original doc. > > > > 在 2012年8月9日星期四UTC+8下午1时50分26秒，jackiedong写道： > > > > > Yes, I indexed the documents with a field "body", and then I want to > > > extract some content from that. > > > Not only for display, but also for search and statistics. > > > > > > 在 2012年8月9日星期四UTC+8上午10时41分27秒，Chris Male写道： > > > > > > > So if I understand correctly, you have indexed documents with a single > > > > field say 'body' and you want to extract some content from that? Is this > > > > just for display purposes or this is for a search? Could you do it before > > > > the content was indexed or on the client side after the search? > > > > > > > > On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > > > > > > > > > Hi, > > > > > I want to extract fields using regex. for example, I have a doc > > > > > {body: jackie's email is [x...@gmail.com](mailto:x...@gmail.com)}. I want to extract the email > > > > > address from the body using a regex. > > > > > I know that script fields seem for this. But I don't find any > > > > > exact example in the API doc. > > > > > Does anyone know how to use regex in script field? > > > > > Thanks a lot. ---
**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\ **Post date:** [August 9, 2012, 9:16am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/7 "2012-08-09T09:16:51Z")
在 2012年8月9日星期四UTC+8下午5时16分26秒，jackiedong写道： > I see this feature in the Splunk. Splunk allows the user defines his own > regex to extract the fields from the existing docs. Then the user can use > these new fields to do the search. > I want to have the same feature like this. > But from the above response, it seems it is expensive to do such thing in > Elasticsearch. > > 在 2012年8月9日星期四UTC+8下午4时52分32秒，olof写道： > > > I would say it is better to extract fields before indexing, for example > > in a document processing framework. Or simply do the content extraction in > > your indexing client. > > > > Do you want to do this for all documents or just the ones that a certain > > query finds? I suppose you could do the extraction in the search client and > > then do an \_update of the document with the new fields. This will be > > expensive in the long run though, since it will cause the document to be > > reindexed. You'll end up with lots of deletes in your index or, if you use > > versioning, double the index size. > > > > Den torsdagen den 9:e augusti 2012 kl. 08:06:12 UTC+2 skrev jackiedong: > > > > > In fact, the real purpose is extract the new fields from the "body" > > > content, and add these extracted fields back into the original doc. > > > > > > 在 2012年8月9日星期四UTC+8下午1时50分26秒，jackiedong写道： > > > > > > > Yes, I indexed the documents with a field "body", and then I want to > > > > extract some content from that. > > > > Not only for display, but also for search and statistics. > > > > > > > > 在 2012年8月9日星期四UTC+8上午10时41分27秒，Chris Male写道： > > > > > > > > > So if I understand correctly, you have indexed documents with a single > > > > > field say 'body' and you want to extract some content from that? Is this > > > > > just for display purposes or this is for a search? Could you do it before > > > > > the content was indexed or on the client side after the search? > > > > > > > > > > On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > > > > > > > > > > > Hi, > > > > > > I want to extract fields using regex. for example, I have a doc > > > > > > {body: jackie's email is [x...@gmail.com](mailto:x...@gmail.com)}. I want to extract the > > > > > > email address from the body using a regex. > > > > > > I know that script fields seem for this. But I don't find any > > > > > > exact example in the API doc. > > > > > > Does anyone know how to use regex in script field? > > > > > > Thanks a lot. ---
**Author:** ![olof](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olof/32/2721_2.png) [@olof](https://discuss.elastic.co/u/olof)\ **Post date:** [August 9, 2012, 10:04am UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/8 "2012-08-09T10:04:37Z")
Well, it probably depends on the size of your index and the number of updates you'll be doing, and how often. If you're only modifying some documents, it might not be a problem. Also, it sounds like you want to modify just some docs that the user gets from a query, which I suppose won't work with preprocessing the data. If you have the time, do some tests. Change your client so to run a query, extract data with a regex and then update the document in the index with that. Or look into alternative scripting languages for script fields; the documentation mentions several ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/scripting.html)) plugins you an install to get javascript, python and groovy for example. That should give for regex capabilities. I'm not sure how you'd go about storing the results from those scripts in the document for later though... Den torsdagen den 9:e augusti 2012 kl. 11:16:26 UTC+2 skrev jackiedong: > I see this feature in the Splunk. Splunk allows the user defines his own > regex to extract the fields from the existing docs. Then the user can use > these new fields to do the search. > I want to have the same feature like this. > But from the above response, it seems it is expensive to do such thing in > Elasticsearch. > > 在 2012年8月9日星期四UTC+8下午4时52分32秒，olof写道： > > > I would say it is better to extract fields before indexing, for example > > in a document processing framework. Or simply do the content extraction in > > your indexing client. > > > > Do you want to do this for all documents or just the ones that a certain > > query finds? I suppose you could do the extraction in the search client and > > then do an \_update of the document with the new fields. This will be > > expensive in the long run though, since it will cause the document to be > > reindexed. You'll end up with lots of deletes in your index or, if you use > > versioning, double the index size. > > > > Den torsdagen den 9:e augusti 2012 kl. 08:06:12 UTC+2 skrev jackiedong: > > > > > In fact, the real purpose is extract the new fields from the "body" > > > content, and add these extracted fields back into the original doc. > > > > > > 在 2012年8月9日星期四UTC+8下午1时50分26秒，jackiedong写道： > > > > > > > Yes, I indexed the documents with a field "body", and then I want to > > > > extract some content from that. > > > > Not only for display, but also for search and statistics. > > > > > > > > 在 2012年8月9日星期四UTC+8上午10时41分27秒，Chris Male写道： > > > > > > > > > So if I understand correctly, you have indexed documents with a single > > > > > field say 'body' and you want to extract some content from that? Is this > > > > > just for display purposes or this is for a search? Could you do it before > > > > > the content was indexed or on the client side after the search? > > > > > > > > > > On Thursday, August 9, 2012 2:26:27 PM UTC+12, jackiedong wrote: > > > > > > > > > > > Hi, > > > > > > I want to extract fields using regex. for example, I have a doc > > > > > > {body: jackie's email is [x...@gmail.com](mailto:x...@gmail.com)}. I want to extract the > > > > > > email address from the body using a regex. > > > > > > I know that script fields seem for this. But I don't find any > > > > > > exact example in the API doc. > > > > > > Does anyone know how to use regex in script field? > > > > > > Thanks a lot. ---
**Author:** ![cwurm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cwurm/32/34882_2.png) [@cwurm](https://discuss.elastic.co/u/cwurm)\ **Post date:** [October 23, 2015, 3:17pm UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/9 "2015-10-23T15:17:48Z")
Fyi: Since I just tested what @olof suggested here's how you would use regular expressions to extract stuff from existing fields. Given a document in Elasticsearch that looks like this: ``` { "_index": "test", "_type": "tweet", "_id": "4", "_score": 1, "_source": { "date": "2014-09-16", "name": "John Smith", "tweet": "The Elasticsearch API is really easy to use", "user_id": 1 } } ``` You could extract the first name from the `name` field like this: ``` GET test/_search { "script_fields": { "first_name": { "script": "/\\w+/.exec(_source['name'])[0]", "lang": "javascript" } }, "_source": "*" } ``` Install the [Javascript plugin](https://github.com/elastic/elasticsearch-lang-javascript) before. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:42pm UTC](https://discuss.elastic.co/t/how-to-extract-the-fields-using-regex/8688/10 "2017-07-05T23:42:59Z")