# Grok after CSV filter **URL:** \ **Category:** Logstash\ **Created:** [August 8, 2017, 3:59pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303 "2017-08-08T15:59:07Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 8, 2017, 3:59pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/1 "2017-08-08T15:59:07Z")
Hi I would be thankful who helps me in extracting substring between two forward slashes using grok after csv filter My CSV file is GIS,/Dianon/CancerRegPHNMS,22876365,96589706,ID\_Dianon\_170708235043,2905,2017-07-09,01:05:10,Received, here is my logstash conf file, all I need is to create new field out of column MailboxName input { file { path =\> "/apps/mft/ELK/MBXLogFiles/20170710.log" start\_position =\> "beginning" } } filter { csv { separator =\> "," columns =\> ["Destination","MailboxName","BPId","MessageId","FileName","Bytes","Date","Time","Action"] } } output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> ["192.168.56.105:9200"] index =\> "mbxlog" document\_type =\> "mbxlogdoctype" } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 9, 2017, 6:16am UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/2 "2017-08-09T06:16:52Z")
```nohighlight grok { match => ["MailboxName", "^/(?[^/]+)/"] } ``` ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 10:49am UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/3 "2017-08-09T10:49:16Z")
Thank you Magnus What I didnt understand in your solution is "whatever", is this a new filed name to give? grok { match =\> ["MailboxName", "^/(?[^/]+)/"] } I have two csv lines like below, I need to create a new field with values ADP & Dianon, how I can achieve with grok? Internal,/ADP\_WorkForce\_IVR,22907388,96732247 Internal,/Dianon/LCLS/Orders,22907225,9673181, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 9, 2017, 11:09am UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/4 "2017-08-09T11:09:55Z")
> What I didnt understand in your solution is “whatever”, is this a new filed name to give? Yes, that's the name of the field you want to create. See the grok filter documentation. > I have two csv lines like below, I need to create a new field with values ADP & Dianon, how I can achieve with grok? > Internal,/ADP\_WorkForce\_IVR,22907388,96732247 > Internal,/Dianon/LCLS/Orders,22907225,9673181, So what exactly are the rules? Extract everything between the forward slashes but only up until the first underscore? ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 11:29am UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/5 "2017-08-09T11:29:03Z")
Hi Magnus, oh ok. Yes, some CSV lines extract data between first two forward slashes and in some CSV lines I need to extract data after first forward slash and before underscore which comes after first forward slash. Please let me know how to achive the above in the same gork expression Thanks Somu ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 9, 2017, 11:43am UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/6 "2017-08-09T11:43:08Z")
``` ^/(?[^/_]+)/ ``` ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 12:59pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/7 "2017-08-09T12:59:42Z")
HI Magnus, it was there for a long time, hardly i have 5 lines in the csv file, here is the console output bin/logstash -f MBXLogstash20170709.conf ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Sending Logstash's logs to /apps/mft/ELK/logstash/logs which is now configured via log4j2.properties [2017-08-09T07:55:00,084][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://192.168.56.105:9200/](http://192.168.56.105:9200/)]}} [2017-08-09T07:55:00,098][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://192.168.56.105:9200/](http://192.168.56.105:9200/), :path=\>"/"} [2017-08-09T07:55:00,245][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>#Java::JavaNet::URI:0x14200553} [2017-08-09T07:55:00,247][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} [2017-08-09T07:55:00,314][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2017-08-09T07:55:00,332][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>[#Java::JavaNet::URI:0x1b4a3154]} [2017-08-09T07:55:00,392][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>250} [2017-08-09T07:55:00,581][INFO][logstash.pipeline] Pipeline main started [2017-08-09T07:55:00,653][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} Here is the complete logstash config $ cat MBXLogstash20170709.conf input { file { path =\> "/apps/mft/ELK/MBXLogFiles/test.log" start\_position =\> "beginning" } } filter { csv { separator =\> "," columns =\> ["Destination","MailboxName","BPId","MessageId","FileName","Bytes","Date","Time","Action"] } mutate { convert =\> { "Bytes" =\> "integer" } } grok { match =\> ["MailboxName", "^/(?[^/]+)/"] } } output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> ["192.168.56.105:9200"] index =\> "mbx201707log" document\_type =\> "mbx201707doctype" } } ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 1:03pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/8 "2017-08-09T13:03:55Z")
Could you please help on why it was not loading elastic, my configuration seems to be correct ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 9, 2017, 1:07pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/9 "2017-08-09T13:07:15Z")
I suspect it's tailing /apps/mft/ELK/MBXLogFiles/test.log and waiting for more data to be added to it. Read about sincedb in the file input documentation. Setting `sincedb_path` to /dev/null disables sincedb. ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 1:27pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/10 "2017-08-09T13:27:04Z")
Perfect magnus, I added sincedb now it is loading but I have one error, it is not able to get CMBP from the blow message, could you help "message" =\> "TP,/CMBP\_PremiereGlobalServices,22875803,96589231,18805956550\_2017\_4600\_0708\_213524.DAT.tmp,79643,2017-07-09,00:02:34,Transmitted,", "Date" =\> "2017-07-09", "tags" =\> [ [0] "\_grokparsefailure" ], ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 1:29pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/11 "2017-08-09T13:29:16Z")
here is my configuration with sincedb, as of now it able to extract data between first two forward slashes but not data between first forward slash and underscore(\_) input { file { path =\> "/apps/mft/ELK/MBXLogFiles/test.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," columns =\> ["Destination","MailboxName","BPId","MessageId","FileName","Bytes","Date","Time","Action"] } grok { match =\> ["MailboxName", "^/(?[^/\_]+)/"] } mutate { convert =\> { "Bytes" =\> "integer" } } } ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 1:43pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/12 "2017-08-09T13:43:38Z")
Magnus, I am trying to play in below url for grok, still no luck..if you have some thing in mind to fix let me know [http://grokconstructor.appspot.com/do/match#result](http://grokconstructor.appspot.com/do/match#result) ---
**Author:** ![somu411](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@somu411](https://discuss.elastic.co/u/somu411)\ **Post date:** [August 9, 2017, 1:54pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/13 "2017-08-09T13:54:38Z")
Magnus, it worked for both now, I removed extra forward slash at the end. Thanks a lot which really helps me testing the grok using online [http://grokconstructor.appspot.com](http://grokconstructor.appspot.com) /(?[^/\_]\*) Thanks Somu ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 6, 2017, 1:54pm UTC](https://discuss.elastic.co/t/grok-after-csv-filter/96303/14 "2017-09-06T13:54:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.