# Help to isolate email adresses in a single field

**URL:** <https://discuss.elastic.co/t/help-to-isolate-email-adresses-in-a-single-field/160049>\
**Category:** Logstash\
**Created:** [December 9, 2018, 2:25pm UTC](https://discuss.elastic.co/t/help-to-isolate-email-adresses-in-a-single-field/160049 "2018-12-09T14:25:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gabriel\_Neumann](https://avatars.discourse-cdn.com/v4/letter/g/8e8cbc/32.png) [@Gabriel\_Neumann](https://discuss.elastic.co/u/Gabriel_Neumann)\
**Post date:** [December 9, 2018, 2:25pm UTC](https://discuss.elastic.co/t/help-to-isolate-email-adresses-in-a-single-field/160049/1 "2018-12-09T14:25:08Z")

</div>

**Hello, I'm new in Logstash and I need some help.**  
**I'm processing a big index with like 600 million records, that has a field called "message".**  
**This field data structure is variant, because it was created from different sources.**

**I need to re-process the whole index capturing email addresses and filtering out the rest.**  
**Also I would need to discard the duplicate email addresses.**

**Following are some samples of variant "message" data in the source index (just ignore the external quotation marks):**

"ok\_for\_all;824174284;Hanley;Maureen;Hanover;03755-1321;18 Woodmore [Dr;;NH;maureenmh@valley.net](mailto:Dr;;NH;maureenmh@valley.net);maureenmh;[valley.net](http://valley.net);;;;;;;;;;;279007"

"ok;824174799;Corbe;Herve;Youngstown;44504-1406;560 Tod [Ln;;OH;hcorbe@neo.rr.com](mailto:Ln;;OH;hcorbe@neo.rr.com);hcorbe;[neo.rr.com](http://neo.rr.com);;;;;;;;;;;279090"

""jip.geer@wxs.nl","p\_unknown\_email""

""tinsie@hetnet.nl","ok""

"ok\_for\_all;"6903420";"Joseph";"Hermo";"5 Regent St";"Ste [513N";"Livingston";"NJ";"7039";"973-535-5000";"jhermo@gmsgroup.com](mailto:513N%22;%22Livingston%22;%22NJ%22;%227039%22;%22973-535-5000%22;%22jhermo@gmsgroup.com)";"unknown";"jhermo";"[gmsgroup.com](http://gmsgroup.com)";"";"";"";"";"";"";"";"";"";"";"";"91255""

"ok;harriet;wallach;142 monterey pointe dr;;west palm [beach;fl;33418;bhavey2001@yahoo.com](mailto:beach;fl;33418;bhavey2001@yahoo.com);bhavey2001;[yahoo.com](http://yahoo.com);;"

"email\_disabled;"2759190";"Jack";"Malarik";"713 Creekview [Dr";"";"Eastlake";"OH";"44095";"";"jackm@c-p-a.com](mailto:Dr%22;%22%22;%22Eastlake%22;%22OH%22;%2244095%22;%22%22;%22jackm@c-p-a.com)";"unknown";"jackm";"[c-p-a.com](http://c-p-a.com)";"";"";"";"";"";"";"";"";"";"";"";"84973""

**The pipeline config is:**

input {  
elasticsearch {  
hosts =\> "localhost"  
index =\> "filteredemails"  
query =\> '{ "query": { "query\_string": { "query": "\*" } } }'  
size =\> 500  
scroll =\> "5m"  
docinfo =\> true  
}  
}  
filter {  
grok {  
patterns\_dir =\> ["/usr/share/logstash/patterns"]  
keep\_empty\_captures =\> true  
match =\> { "message" =\> "%{EMAILADDRESS:clean-email}"}  
}  
grok {  
match =\> {  
"clean-email" =\> ";%{EMAILADDRESS:[email\_ok]};"}  
}  
}  
output {  
elasticsearch {  
index =\> "isolated.%{[@metadata][\_index]}"  
document\_type =\> "uax\_url\_email"  
document\_id =\> "%{[@metadata][\_id]}"  
}  
}

**But the resulting "clean-email" field fails in some cases.**

**I found a way to generate an Analyzer and tokenizer to separate email addresses, but I cant figure out the way to use it for the whole dataset. It's:**

PUT isolated.filteredemails  
{  
"settings": {  
"analysis": {  
"analyzer": {  
"my\_analyzer": {  
"tokenizer": "my\_tokenizer"  
}  
},  
"tokenizer": {  
"my\_tokenizer": {  
"type": "uax\_url\_email"  
}  
},  
"filter": [  
"email",  
"lowercase",  
"unique"  
]  
}  
}  
}

**this analyzer works individually with the message data and tokenizes the email address well, :**

POST isolated.filteredemails/\_analyze  
{  
"analyzer": "my\_analyzer",  
"text": "babylon;ny;11704;bcook211@yahoo.com"  
}

**I don't know how to use this tokenizer to just get the part of message and grab it into a new field called "clean-email".**

**Thank you in advance to anyone who could give me a hand on this.**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2019, 2:25pm UTC](https://discuss.elastic.co/t/help-to-isolate-email-adresses-in-a-single-field/160049/2 "2019-01-06T14:25:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.