# Hypen "-" in grok fields

**URL:** <https://discuss.elastic.co/t/hypen-in-grok-fields/149981>\
**Category:** Logstash\
**Created:** [September 26, 2018, 10:23am UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981 "2018-09-26T10:23:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ssi](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@ssi](https://discuss.elastic.co/u/ssi)\
**Post date:** [September 26, 2018, 10:23am UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/1 "2018-09-26T10:23:49Z")

</div>

Hi im trying to apply the solution described here

> [@Grok Filter Parse a Hyphen as absent](https://discuss.elastic.co/t/grok-filter-parse-a-hyphen-as-absent/106030):
>
> Hi, I'm trying to parse a log which will insert a hyphen for some fields when there is no value: Is there an easy way with grok to treat the hyphen as if the value is absent? i.e. not add the hyphen as a value, but continue to parse the rest of the values correctly? 2 Example lines from log: 2017-11-01 12:03:47 - - - "-" 10.0.0.100 "-" - "-" - "VS: name" 2017-11-01 12:44:01 10.0.0.1 "/file.cfc?test=thing" 192.168.1.1:80 192.168.1.1:80 "P: pool1" 10.0.0.100 "thing1=24; thing2=5;" 200 "Mozil…

to the follwoing part of the Exchange message tracking logs

source\_context

MDB:04111c61-c212-4078-bf65-369a8cd3080c, Mailbox:257ff165-97e3-498b-8e11-b5b735da312b, Event:172773384, MessageClass:IPM.Note.MapiSubmitLAMProbe, CreationTime:2018-09-26T09:49:24.396Z, ClientType:Monitoring

to fix the "-" situation in the fields above magnus suggests this line in the grok filter  
(-|%{PATTERN:fieldname})

but i can not figure out how to apply this in a grok filter that works

below is my grok that i Thought would work

{WORD:MDB},%{SPACE}%{WORD:Mailbox},%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP\_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}

after findeing the post above i made the following changes

(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP\_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}

but im still hitting a compile error so it seams i getting the structure of the grok pattern wrong some how.

the entire current filter below

if [type] == "exchange" {  
csv {  
add\_tag =\> ['exh\_msg\_trk']  
columns =\> ['logdate', 'client\_ip', 'client\_hostname', 'server\_ip', 'server\_hostname', 'source\_context', 'connector\_id', 'source', 'event\_id', 'internal\_message\_id', 'message\_id', 'network\_message\_id', 'recipient\_address', 'recipient\_status', 'total\_bytes', 'recipient\_count', 'related\_recipient\_address', 'reference', 'message\_subject', 'sender\_address', 'return\_path', 'message\_info', 'directionality', 'tenant\_id', 'original\_client\_ip', 'original\_server\_ip', 'custom\_data']  
remove\_field =\> ["logdate"]  
}  
grok {  
match =\> ["message", "%{TIMESTAMP\_ISO8601:timestamp}"]  
}  
mutate {  
convert =\> ["total\_bytes", "integer"]  
convert =\> ["recipient\_count", "integer"]  
split =\> ["recipient\_address", ";"]  
split =\> ["source\_context", ";"]  
split =\> ["custom\_data", ";"]  
}  
grok {  
match =\> ["source\_context", "%{GREEDYDATA}%{WORD:ClientType}"]  
}  
date {  
match =\> ["timestamp", "ISO8601"]  
timezone =\> "Europe/London"  
remove\_field =\> ["timestamp"]  
}  
if "\_grokparsefailure" in [tags] {  
drop { }  
}  
}

the working filter should look like this ?

if [type] == "exchange" {  
csv {  
add\_tag =\> ['exh\_msg\_trk']  
columns =\> ['logdate', 'client\_ip', 'client\_hostname', 'server\_ip', 'server\_hostname', 'source\_context', 'connector\_id', 'source', 'event\_id', 'internal\_message\_id', 'message\_id', 'network\_message\_id', 'recipient\_address', 'recipient\_status', 'total\_bytes', 'recipient\_count', 'related\_recipient\_address', 'reference', 'message\_subject', 'sender\_address', 'return\_path', 'message\_info', 'directionality', 'tenant\_id', 'original\_client\_ip', 'original\_server\_ip', 'custom\_data']  
remove\_field =\> ["logdate"]  
}  
grok {  
match =\> ["message", "%{TIMESTAMP\_ISO8601:timestamp}"]  
}  
mutate {  
convert =\> ["total\_bytes", "integer"]  
convert =\> ["recipient\_count", "integer"]  
split =\> ["recipient\_address", ";"]  
split =\> ["source\_context", ";"]  
split =\> ["custom\_data", ";"]  
}  
grok {  
match =\> ["source\_context", "(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP\_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}"]  
}  
date {  
match =\> ["timestamp", "ISO8601"]  
timezone =\> "Europe/London"  
remove\_field =\> ["timestamp"]  
}  
if "\_grokparsefailure" in [tags] {  
drop { }  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 26, 2018, 11:04am UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/2 "2018-09-26T11:04:30Z")

</div>

What error message are you getting? For which line of input?

---

<div class="post-metadata">

**Author:** ![ssi](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@ssi](https://discuss.elastic.co/u/ssi)\
**Post date:** [September 26, 2018, 1:23pm UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/3 "2018-09-26T13:23:16Z")

</div>

im not getting any errors on the logstash.conf file if that's what your asking it passes the tests done with the -t switch.

[2018-09-26T15:17:28,278][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::FilterDelegator:0x35eba953 @metric\_events\_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @id="25e7e156a52a0839d5c78d0b9f25a0ab1c142dc4a88540e8b21694257124e1af", @klass=LogStash::Filters::Grok, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0x451d1d73 @metric=#\<LogStash::Instrument::Metric:0x71e91024 @collector=#\<LogStash::Instrument::Collector:0x29427b63 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x418bfe29 @store=#\<Concurrent:🗺0x00000000000fc4 entries=2 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x6fa220c0, @fast\_lookup=#\<Concurrent:🗺0x00000000000fc8 entries=119 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :filters, :"25e7e156a52a0839d5c78d0b9f25a0ab1c142dc4a88540e8b21694257124e1af", :events]\>, @filter=\<LogStash::Filters::Grok match=\>{"source\_context"=\>"(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP\_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}"}, id=\>"25e7e156a52a0839d5c78d0b9f25a0ab1c142dc4a88540e8b21694257124e1af", enable\_metric=\>true, periodic\_flush=\>false, patterns\_files\_glob=\>"\*", break\_on\_match=\>true, named\_captures\_only=\>true, keep\_empty\_captures=\>false, tag\_on\_failure=\>["\_grokparsefailure"], timeout\_millis=\>30000, tag\_on\_timeout=\>"\_groktimeout"\>\>", :error=\>"pattern %{INTEGER:Event} not defined", :thread=\>"#\<Thread:0x19d1e588 run\>"}  
[2018-09-26T15:17:28,398][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<Grok::PatternError: pattern %{INTEGER:Event} not defined\>, :backtrace=\>["E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "org/jruby/RubyKernel.java:1292:in`loop'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:in`block in register'", "org/jruby/RubyArray.java:1734:in `each'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:in`block in register'", "org/jruby/RubyHash.java:1343:in `each'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:in`register'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:340:in `register_plugin'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:351:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:351:in`register\_plugins'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:729:in `maybe_setup_out_plugins'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:361:in`start\_workers'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:288:in `run'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:248:in`block in start'"], :thread=\>"#\<Thread:0x19d1e588 run\>"}  
[2018-09-26T15:17:28,435][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, action\_result: false", :backtrace=\>nil}  
[2018-09-26T15:17:29,057][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

E:\FWLOG\logstash-6.3.2\logstash-6.3.2\bin\>

this is if I run the config

above shows it ( I think it shows) its related to the %{INTEGER:Event} statement but I'm in the dark as to how to fix it.

if u run the config test (-t swith) do not get any errors

E:\FWLOG\logstash-6.3.2\logstash-6.3.2\bin\>logstash -f logstash\_exchange\_fw\_loadbalancer\_grok.conf -t  
Sending Logstash's logs to E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logs which is now configured via log4j2.properties  
[2018-09-26T15:20:20,657][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
Configuration OK  
[2018-09-26T15:20:30,216][INFO][logstash.runner] Using config.test\_and\_exit mode. Config Validation Result: OK. Exiting Logstash

E:\FWLOG\logstash-6.3.2\logstash-6.3.2\bin\>

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 26, 2018, 1:24pm UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/4 "2018-09-26T13:24:25Z")

</div>

I don't believe there is a grok pattern named INTEGER, but there's one called INT.

---

<div class="post-metadata">

**Author:** ![ssi](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@ssi](https://discuss.elastic.co/u/ssi)\
**Post date:** [September 27, 2018, 7:55am UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/5 "2018-09-27T07:55:18Z")

</div>

thanks magnus, my mistake I fixed it, but it still seams as if the grok filter simply is not activated, at least that what i thought. Then i noticed the empty first space on the fields below (from the json output)

there is an empty space before the first M in mailbox line and the ", this indicates to me that the grok filter is doing its job.

What i want to achieve is having the fields seperated out / split so i can index them individually. I begining to thing this is not a grok issue at all?

```
"source_context": [
  "MDB:bc599da3-28ee-4fae-8ee1-dae15cf076dd",
  " Mailbox:cea54ccc-7e6b-4113-936b-652fc0286322",
  " Event:49815788",
  " MessageClass:IPM.Schedule.Meeting.Request",
  " CreationTime:2018-09-27T07:27:13.267Z",
  " ClientType:MOMT"

```

should i be looking at the mutate section instead?  
cut from logstash config

mutate {  
convert =\> ["total\_bytes", "integer"]  
convert =\> ["recipient\_count", "integer"]  
split =\> ["recipient\_address", ";"]  
split =\> ["source\_context", ";"]  
split =\> ["custom\_data", ";"]  
}

i tried to change the ";" to a "," in the split statement for "source\_context" as my understanding is that the "," will then be the qualifier for seperating the fileds and index them as individial?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 27, 2018, 8:25am UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/6 "2018-09-27T08:25:55Z")

</div>

The mutate filter's split option splits strings to arrays but `source_context` already is an array so I'm not sure what you're trying to do. Do you want to have each element of the `source_context` array in an event of its own? If so you should use the split filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2018, 8:25am UTC](https://discuss.elastic.co/t/hypen-in-grok-fields/149981/7 "2018-10-25T08:25:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.