Hypen "-" in grok fields


#1

Hi im trying to apply the solution described here

to the follwoing part of the Exchange message tracking logs

source_context

MDB:04111c61-c212-4078-bf65-369a8cd3080c, Mailbox:257ff165-97e3-498b-8e11-b5b735da312b, Event:172773384, MessageClass:IPM.Note.MapiSubmitLAMProbe, CreationTime:2018-09-26T09:49:24.396Z, ClientType:Monitoring

to fix the "-" situation in the fields above magnus suggests this line in the grok filter
(-|%{PATTERN:fieldname})

but i can not figure out how to apply this in a grok filter that works

below is my grok that i Thought would work

{WORD:MDB},%{SPACE}%{WORD:Mailbox},%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}

after findeing the post above i made the following changes

(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}

but im still hitting a compile error so it seams i getting the structure of the grok pattern wrong some how.

the entire current filter below

if [type] == "exchange" {
csv {
add_tag => [ 'exh_msg_trk' ]
columns => ['logdate', 'client_ip', 'client_hostname', 'server_ip', 'server_hostname', 'source_context', 'connector_id', 'source', 'event_id', 'internal_message_id', 'message_id', 'network_message_id', 'recipient_address', 'recipient_status', 'total_bytes', 'recipient_count', 'related_recipient_address', 'reference', 'message_subject', 'sender_address', 'return_path', 'message_info', 'directionality', 'tenant_id', 'original_client_ip', 'original_server_ip', 'custom_data']
remove_field => [ "logdate" ]
}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
}
mutate {
convert => [ "total_bytes", "integer" ]
convert => [ "recipient_count", "integer" ]
split => ["recipient_address", ";"]
split => [ "source_context", ";" ]
split => [ "custom_data", ";" ]
}
grok {
match => [ "source_context", "%{GREEDYDATA}%{WORD:ClientType}" ]
}
date {
match => [ "timestamp", "ISO8601" ]
timezone => "Europe/London"
remove_field => [ "timestamp" ]
}
if "_grokparsefailure" in [tags] {
drop { }
}
}

the working filter should look like this ?

if [type] == "exchange" {
csv {
add_tag => [ 'exh_msg_trk' ]
columns => ['logdate', 'client_ip', 'client_hostname', 'server_ip', 'server_hostname', 'source_context', 'connector_id', 'source', 'event_id', 'internal_message_id', 'message_id', 'network_message_id', 'recipient_address', 'recipient_status', 'total_bytes', 'recipient_count', 'related_recipient_address', 'reference', 'message_subject', 'sender_address', 'return_path', 'message_info', 'directionality', 'tenant_id', 'original_client_ip', 'original_server_ip', 'custom_data']
remove_field => [ "logdate" ]
}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
}
mutate {
convert => [ "total_bytes", "integer" ]
convert => [ "recipient_count", "integer" ]
split => ["recipient_address", ";"]
split => [ "source_context", ";" ]
split => [ "custom_data", ";" ]
}
grok {
match => [ "source_context", "(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}" ]
}
date {
match => [ "timestamp", "ISO8601" ]
timezone => "Europe/London"
remove_field => [ "timestamp" ]
}
if "_grokparsefailure" in [tags] {
drop { }
}
}


(Magnus Bäck) #2

What error message are you getting? For which line of input?


#3

im not getting any errors on the logstash.conf file if that's what your asking it passes the tests done with the -t switch.

[2018-09-26T15:17:28,278][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x35eba953 @metric_events_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric_events_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric_events_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @id="25e7e156a52a0839d5c78d0b9f25a0ab1c142dc4a88540e8b21694257124e1af", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x451d1d73 @metric=#<LogStash::Instrument::Metric:0x71e91024 @collector=#<LogStash::Instrument::Collector:0x29427b63 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x418bfe29 @store=#<Concurrent::map:0x00000000000fc4 entries=2 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x6fa220c0, @fast_lookup=#<Concurrent::map:0x00000000000fc8 entries=119 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :"25e7e156a52a0839d5c78d0b9f25a0ab1c142dc4a88540e8b21694257124e1af", :events]>, @filter=<LogStash::Filters::Grok match=>{"source_context"=>"(-|%{WORD:MDB}),%{SPACE}(-|%{WORD:Mailbox}),%{SPACE}%{INTEGER:Event},%{SPACE}%{WORD:MessageClass},%{SPACE}%{TIMESTAMP_ISO8601:CreationTime},%{SPACE}%{WORD:ClientType}"}, id=>"25e7e156a52a0839d5c78d0b9f25a0ab1c142dc4a88540e8b21694257124e1af", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>"*", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>["_grokparsefailure"], timeout_millis=>30000, tag_on_timeout=>"_groktimeout">>", :error=>"pattern %{INTEGER:Event} not defined", :thread=>"#<Thread:0x19d1e588 run>"}
[2018-09-26T15:17:28,398][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{INTEGER:Event} not defined>, :backtrace=>["E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1292:inloop'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:281:inblock in register'", "org/jruby/RubyArray.java:1734:in each'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:275:inblock in register'", "org/jruby/RubyHash.java:1343:in each'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.3/lib/logstash/filters/grok.rb:270:inregister'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:340:in register_plugin'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:351:inblock in register_plugins'", "org/jruby/RubyArray.java:1734:in each'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:351:inregister_plugins'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:729:in maybe_setup_out_plugins'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:361:instart_workers'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:288:in run'", "E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logstash-core/lib/logstash/pipeline.rb:248:inblock in start'"], :thread=>"#<Thread:0x19d1e588 run>"}
[2018-09-26T15:17:28,435][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2018-09-26T15:17:29,057][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

E:\FWLOG\logstash-6.3.2\logstash-6.3.2\bin>

this is if I run the config

above shows it ( I think it shows) its related to the %{INTEGER:Event} statement but I'm in the dark as to how to fix it.

if u run the config test (-t swith) do not get any errors

E:\FWLOG\logstash-6.3.2\logstash-6.3.2\bin>logstash -f logstash_exchange_fw_loadbalancer_grok.conf -t
Sending Logstash's logs to E:/FWLOG/logstash-6.3.2/logstash-6.3.2/logs which is now configured via log4j2.properties
[2018-09-26T15:20:20,657][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2018-09-26T15:20:30,216][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

E:\FWLOG\logstash-6.3.2\logstash-6.3.2\bin>


(Magnus Bäck) #4

I don't believe there is a grok pattern named INTEGER, but there's one called INT.


#5

thanks magnus, my mistake I fixed it, but it still seams as if the grok filter simply is not activated, at least that what i thought. Then i noticed the empty first space on the fields below (from the json output)

there is an empty space before the first M in mailbox line and the ", this indicates to me that the grok filter is doing its job.

What i want to achieve is having the fields seperated out / split so i can index them individually. I begining to thing this is not a grok issue at all?

"source_context": [
  "MDB:bc599da3-28ee-4fae-8ee1-dae15cf076dd",
  " Mailbox:cea54ccc-7e6b-4113-936b-652fc0286322",
  " Event:49815788",
  " MessageClass:IPM.Schedule.Meeting.Request",
  " CreationTime:2018-09-27T07:27:13.267Z",
  " ClientType:MOMT"

should i be looking at the mutate section instead?
cut from logstash config

mutate {
convert => [ "total_bytes", "integer" ]
convert => [ "recipient_count", "integer" ]
split => ["recipient_address", ";"]
split => [ "source_context", ";" ]
split => [ "custom_data", ";" ]
}

i tried to change the ";" to a "," in the split statement for "source_context" as my understanding is that the "," will then be the qualifier for seperating the fileds and index them as individial?


(Magnus Bäck) #6

The mutate filter's split option splits strings to arrays but source_context already is an array so I'm not sure what you're trying to do. Do you want to have each element of the source_context array in an event of its own? If so you should use the split filter.


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.