# Logstash filter to extract email and credit card **URL:** \ **Category:** Logstash\ **Created:** [April 6, 2020, 2:45pm UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736 "2020-04-06T14:45:31Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![AnanthMahadevan](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@AnanthMahadevan](https://discuss.elastic.co/u/AnanthMahadevan)\ **Post date:** [April 6, 2020, 2:45pm UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/1 "2020-04-06T14:45:31Z")
I am new to logstash and want to extract email IDs and credit card details from files. I am using filebeat to pass log files to logstash and also have setup a config file with the grok patterns. However, I am not able to extract either emailID or credit card. I get this error: ```auto "tags" => [ [0] "beats_input_codec_plain_applied", [1] "_grokparsefailure" ``` Given below is my config file (the pattern works in the [grokdebug app](https://grokdebug.herokuapp.com/)). ```auto input { beats { port => 5044 host => "0.0.0.0" } } filter { if [message] =~ "(?\d{4}-\d{4}-\d{4})" { mutate { add_tag => ["ccNumber"] } } else if [message] =~ "(?[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.(?:[0-9A-Za-z][0-‌​9A-Za-z-]{0,62}))*)" { mutate { add_tag => ["emailID"] } } else { grok { match => { "message" => '%{HTTPD_COMMONLOG} "%{GREEDYDATA:referrer} %{GREEDYDATA:agent}"' } } } mutate { convert => { "response" => "integer" "bytes" => "integer" } } } output { stdout { codec => rubydebug } file { path => "logs\output.txt" } } ``` ---
**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\ **Post date:** [April 6, 2020, 4:00pm UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/2 "2020-04-06T16:00:13Z")
Can you provide log message sample ? You are trying to apply HTTPD\_COMMONLOG filter on the message and this pattern is probably causing the error as it is not matching. ---
**Author:** ![AnanthMahadevan](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@AnanthMahadevan](https://discuss.elastic.co/u/AnanthMahadevan)\ **Post date:** [April 7, 2020, 6:07am UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/3 "2020-04-07T06:07:47Z")
Here is a sample log message: 109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-" 177.201.52.125 - - [13/Dec/2015:14:45:49 +0100] "[test@test.com](mailto:test@test.com)" 177.201.52.125 - - [13/Dec/2015:14:45:49 +0100] "1234-5678-9876" ---
**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\ **Post date:** [April 7, 2020, 8:16am UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/4 "2020-04-07T08:16:03Z")
Hi, Your grok pattern is not the right one for the request, have a look at [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) ---
**Author:** ![AnanthMahadevan](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@AnanthMahadevan](https://discuss.elastic.co/u/AnanthMahadevan)\ **Post date:** [April 21, 2020, 2:26pm UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/5 "2020-04-21T14:26:25Z")
Hi, Thanks for your response. I went through the documentation and tried several options. But none of them worked. What I want is to extract email address, credit card along with IP address and other details from the Apache log. Can you please help? My filter is as follows (to extract all details if they exist in one line): ```auto filter { grok { match => { "message" => [ "APACHE LOG: '%{IP:ip_address} %{USER:identity1} %{USER:identity2} \[%{HTTPDATE:req_ts}\] \"%{WORD:http_verb} %{URIPATHPARAM:req_path} HTTP/%{NUMBER:http_version}\" %{INT:http_status:int} %{INT:num_bytes:int} (?\d{4}-\d{4}-\d{4})'", "ccNumber: (?\d{4}-\d{4}-\d{4})", "emailID: (?[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.(?:[0-9A-Za-z][0-‌​9A-Za-z-]{0,62}))*)" ] } } } ``` How do I get all details from a single line and if all details are not present, then how do I get just the credit card or email address or the IP address? Below filter also did not work (to extract one detail at a time if they exist): ```auto filter { if [message] =~ "(?\d{4}-\d{4}-\d{4})" { grok { match => { "message" => '%{IP:ip_address} %{USER:identity1} %{USER:identity2} \[%{HTTPDATE:req_ts}\] \"%{WORD:http_verb} %{URIPATHPARAM:req_path} HTTP/%{NUMBER:http_version}\" %{INT:http_status:int} %{INT:num_bytes:int}' } add_tag => ["ccNumber"] } } else if [message] =~ "(?[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.(?:[0-9A-Za-z][0-‌​9A-Za-z-]{0,62}))*)" { grok { match => { "message" => '%{IP:ip_address} %{USER:identity1} %{USER:identity2} \[%{HTTPDATE:req_ts}\] \"%{WORD:http_verb} %{URIPATHPARAM:req_path} HTTP/%{NUMBER:http_version}\" %{INT:http_status:int} %{INT:num_bytes:int}' } add_tag => ["emailID"] } } else { mutate { add_tag => ["TEST123"] } } } ``` ---
**Author:** ![AnanthMahadevan](https://avatars.discourse-cdn.com/v4/letter/a/ccd318/32.png) [@AnanthMahadevan](https://discuss.elastic.co/u/AnanthMahadevan)\ **Post date:** [May 6, 2020, 11:10am UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/6 "2020-05-06T11:10:09Z")
Hi, I compared my grok patterns with that in the documentation several times and nothing seemed out of place. But what troubled me was the output from the log files (they were encoded in a different format). I had been editing the log files in notepad++. So, I decided to process an unedited log file which worked. Opening the log file in notepad, making minor changes to them and running them through grok now worked. Also, adding a new line should be done via a command line (i.e. outside an editor). I hope this helps someone with the same issue. Thanks, Ananth. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 3, 2020, 11:10am UTC](https://discuss.elastic.co/t/logstash-filter-to-extract-email-and-credit-card/226736/7 "2020-06-03T11:10:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.