Grok filter

Sarthak_Mishra · June 22, 2021, 2:57pm

Hello,

I want to create a grok filter. e.g There is a column in my csv file with all the os system listed windows,ubuntu etc. If there is any typo in the column. I want logstash to create a new field with the name "is_os_correct" and have a false value for the the typo column.

input{

file

   {

    path=>"C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter{

csv

{

separator => ","

columns => ["Name","Age","System"]

}

grok { match => {"System" => "%{WORD:validSystem}" } }

if "_grokparsefailure" in [tags] {

  mutate {

    add_field => { "is_os_correct" => "false" }

  }

}

else

{

  mutate

{

    add_field => { "is_os_correct" => "true" }

  }

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

Badger · June 22, 2021, 11:15pm

You are only going to get a grok failure if the field contains characters outside of the group [a-zA-Z0-9_].

Do you have a list of system names that are acceptable? If so, using a translate filter might be better.

Sarthak_Mishra · June 24, 2021, 11:07am

HI Badger, can you please help me further on it

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

    separator => ","

    columns => ["Name","Age","System"]

}

translate {

    field => "System"

    destination => "realOS"

    fall_back => "not_found"

    dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

    match => [ %{WORD:System}]

}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

This is my error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logs which is now configured via log4j2.properties
[2021-06-24T16:32:17,029][INFO ][logstash.runner ] Log4j configuration path used is: C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\config\log4j2.properties
[2021-06-24T16:32:17,041][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T16:32:17,101][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-24T16:32:17,969][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T16:32:18,107][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{", "]" at line 22, column 20 (byte 467) after filter {\r\n csv {\r\n separator => ","\r\n columns => ["Name","Age","System"]\r\n }\r\n\r\n translate {\r\n field => "System"\r\n destination => "realOS"\r\n fall_back => "not_found"\r\n dictionary => ["Windows10","Kali","Ubuntu","CentOS"]\r\n }\r\n\r\n grok {\r\n match => [ ", :backtrace=>["C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:389:in block in converge_state'"]}
[2021-06-24T16:32:18,207][INFO ][logstash.runner ] Logstash shut down.
[2021-06-24T16:32:18,207][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.lib.bootstrap.environment.(C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

Sarthak_Mishra · June 24, 2021, 11:26am

Hi,
I still get an error can you help be debug it

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

separator => ","

columns => ["Name","Age","System"]

}

translate {

field => "System"

destination => "realOS"

fall_back => "not_found"

dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

match => [ "%{WORD:System}" ]

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

Cad · June 24, 2021, 11:29am

Sorry i give the answer to fast.

The match option need

a field to apply the regex.
one or more regex.

So it need to be like this :

grok {
match => { "field" => ["%{WORD:System}"]}
}

Or

grok {
match => { "field" => "%{WORD:System}"}
}

But i don't understand why you add a grok filter if the translate filter do the job.

Sarthak_Mishra · June 24, 2021, 12:04pm

I tried doing it with the only translate it did not work.So I added grok to check my luck. Can you please suggest something

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

separator => ","

columns => ["Name","Age","System"]

}

translate {

field => "System"

destination => "realOS"

fall_back => "not_found"

dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

match => { "field" => ["%{WORD:System}"]}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

"Using bundled JDK: ""
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logs which is now configured via log4j2.properties
[2021-06-24T17:33:21,874][INFO ][logstash.runner ] Log4j configuration path used is: C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\config\log4j2.properties
[2021-06-24T17:33:21,883][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T17:33:21,948][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-24T17:33:22,961][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T17:33:26,921][INFO ][org.reflections.Reflections] Reflections took 192 ms to scan 1 urls, producing 23 keys and 47 values
[2021-06-24T17:33:27,758][ERROR][logstash.filters.translate] Unknown setting 'fall_back' for translate
[2021-06-24T17:33:27,763][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:402
[2021-06-24T17:33:27,771][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[2021-06-24T17:33:27,781][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
[2021-06-24T17:33:27,788][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.lib.bootstrap.environment.(C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

Cad · June 24, 2021, 12:38pm

It is fallback instead of fall_back.

Try this

input {
  file {
    path => "C:/Users/kumar/Desktop/grok.csv"
    start_position => "beginning"
    sincedb_path => "NULL"
  }
}

filter {
  csv {
    separator => ","
    columns => ["Name","Age","System"]
  }

  translate {
    field => "System"
    destination => "realOS"
    fallback => "not_found"
    dictionary => ["Windows10","Kali","Ubuntu","CentOS"]
  }
}

output {
  elasticsearch{
    hosts => "http://localhost:9200/"
    index => "grok"
  }
  stdout{}
}

Sarthak_Mishra · June 24, 2021, 1:53pm

Thanks it has worked. Can i add something like fallback=not found for things that match the dictionary which say "found". If yes, can you please tell me how

Cad · June 24, 2021, 2:16pm

Take a look at this

Cad.

Sarthak_Mishra · June 25, 2021, 6:06pm

Hi again. the code was supposed to show "not_found" only for the system not in my dictionary. But it shows "not_found" even for some system names which are in my dictionary. Can you please help to identify where the algorithm lacks? Thanks for helping

Badger · June 25, 2021, 6:21pm

The dictionary should be a hash showing what value should replace the value in the [System] field if it matches. You have set it to an array, which logstash converts to a hash

dictionary => {
    "Windows10" => "Kali"
    "Ubuntu" => "CentOS"
}

So that if the [System] field contains "Windows10" that is replaced with "Kali".

Sarthak_Mishra · June 26, 2021, 6:15am

Sorry, I shall explain it differently. dictionary => ["Windows10","Kali","Ubuntu","CentOS"] in this dictionary, the system names which not matchh the ones in the dictionary should be named not_found. e.g Kali in the above snapshot says not found where else kali is mentioned in my dictionary.

Cad · June 26, 2021, 9:51am

Badger explained well why it's not working.
dictionary => ["Windows10","Kali","Ubuntu","CentOS"]
is converted by logstash in

dictionary => {
    "Windows10" => "Kali"
    "Ubuntu" => "CentOS"
}

In your case, i think a dictionnary like this give the result you want

translate{
  field => "System"
  destination => "realOS"
  fallback => "not_found"
  dictionary => {
      "Windows10" => "found"
      "Ubuntu" => "found"
      "Kali" => "found"
      "CentOS" => "found"
  }
}

It put found in realOS if System contains Windows10, Ubuntu, Kali or CentOS. Otherwise it put not_found in realOS.

Sarthak_Mishra · June 26, 2021, 1:50pm

Thanks, Badger and Cad for helping me. Really appreciate it.

system · July 24, 2021, 1:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Grok filter Logstash	11	323	May 4, 2021
Grok after CSV filter Logstash	13	5214	September 6, 2017
Grok add field after CSV Filter Logstash	1	390	December 9, 2019
Grok filter to retrieve simple words or character from text/string Logstash	8	4531	December 29, 2017
Csv filter plugin: how to deal with masked quotations? Logstash	3	1654	September 20, 2019

Grok filter

Related Topics