Grok filter

Sarthak_Mishra · June 22, 2021, 2:57pm

Hello,

I want to create a grok filter. e.g There is a column in my csv file with all the os system listed windows,ubuntu etc. If there is any typo in the column. I want logstash to create a new field with the name "is_os_correct" and have a false value for the the typo column.

input{

file

   {

    path=>"C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter{

csv

{

separator => ","

columns => ["Name","Age","System"]

}

grok { match => {"System" => "%{WORD:validSystem}" } }

if "_grokparsefailure" in [tags] {

  mutate {

    add_field => { "is_os_correct" => "false" }

  }

}

else

{

  mutate

{

    add_field => { "is_os_correct" => "true" }

  }

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

Badger · June 22, 2021, 11:15pm

You are only going to get a grok failure if the field contains characters outside of the group [a-zA-Z0-9_].

Do you have a list of system names that are acceptable? If so, using a translate filter might be better.

Sarthak_Mishra · June 24, 2021, 11:07am

HI Badger, can you please help me further on it

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

    separator => ","

    columns => ["Name","Age","System"]

}

translate {

    field => "System"

    destination => "realOS"

    fall_back => "not_found"

    dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

    match => [ %{WORD:System}]

}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

This is my error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logs which is now configured via log4j2.properties
[2021-06-24T16:32:17,029][INFO ][logstash.runner ] Log4j configuration path used is: C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\config\log4j2.properties
[2021-06-24T16:32:17,041][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T16:32:17,101][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-24T16:32:17,969][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T16:32:18,107][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{", "]" at line 22, column 20 (byte 467) after filter {\r\n csv {\r\n separator => ","\r\n columns => ["Name","Age","System"]\r\n }\r\n\r\n translate {\r\n field => "System"\r\n destination => "realOS"\r\n fall_back => "not_found"\r\n dictionary => ["Windows10","Kali","Ubuntu","CentOS"]\r\n }\r\n\r\n grok {\r\n match => [ ", :backtrace=>["C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:389:in block in converge_state'"]}
[2021-06-24T16:32:18,207][INFO ][logstash.runner ] Logstash shut down.
[2021-06-24T16:32:18,207][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.lib.bootstrap.environment.(C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

Sarthak_Mishra · June 24, 2021, 11:26am

Hi,
I still get an error can you help be debug it

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

separator => ","

columns => ["Name","Age","System"]

}

translate {

field => "System"

destination => "realOS"

fall_back => "not_found"

dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

match => [ "%{WORD:System}" ]

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

Cad · June 24, 2021, 11:29am

Sorry i give the answer to fast.

The match option need

a field to apply the regex.
one or more regex.

So it need to be like this :

grok {
match => { "field" => ["%{WORD:System}"]}
}

Or

grok {
match => { "field" => "%{WORD:System}"}
}

But i don't understand why you add a grok filter if the translate filter do the job.

Sarthak_Mishra · June 24, 2021, 12:04pm

I tried doing it with the only translate it did not work.So I added grok to check my luck. Can you please suggest something

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

separator => ","

columns => ["Name","Age","System"]

}

translate {

field => "System"

destination => "realOS"

fall_back => "not_found"

dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

match => { "field" => ["%{WORD:System}"]}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

"Using bundled JDK: ""
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logs which is now configured via log4j2.properties
[2021-06-24T17:33:21,874][INFO ][logstash.runner ] Log4j configuration path used is: C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\config\log4j2.properties
[2021-06-24T17:33:21,883][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T17:33:21,948][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-24T17:33:22,961][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T17:33:26,921][INFO ][org.reflections.Reflections] Reflections took 192 ms to scan 1 urls, producing 23 keys and 47 values
[2021-06-24T17:33:27,758][ERROR][logstash.filters.translate] Unknown setting 'fall_back' for translate
[2021-06-24T17:33:27,763][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:402
[2021-06-24T17:33:27,771][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[2021-06-24T17:33:27,781][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
[2021-06-24T17:33:27,788][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.lib.bootstrap.environment.(C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

Cad · June 24, 2021, 12:38pm

It is fallback instead of fall_back.

Try this

input {
  file {
    path => "C:/Users/kumar/Desktop/grok.csv"
    start_position => "beginning"
    sincedb_path => "NULL"
  }
}

filter {
  csv {
    separator => ","
    columns => ["Name","Age","System"]
  }

  translate {
    field => "System"
    destination => "realOS"
    fallback => "not_found"
    dictionary => ["Windows10","Kali","Ubuntu","CentOS"]
  }
}

output {
  elasticsearch{
    hosts => "http://localhost:9200/"
    index => "grok"
  }
  stdout{}
}

Sarthak_Mishra · June 24, 2021, 1:53pm

Thanks it has worked. Can i add something like fallback=not found for things that match the dictionary which say "found". If yes, can you please tell me how

Cad · June 24, 2021, 2:16pm

Take a look at this

Cad.

Sarthak_Mishra · June 25, 2021, 6:06pm

Hi again. the code was supposed to show "not_found" only for the system not in my dictionary. But it shows "not_found" even for some system names which are in my dictionary. Can you please help to identify where the algorithm lacks? Thanks for helping

Badger · June 25, 2021, 6:21pm

The dictionary should be a hash showing what value should replace the value in the [System] field if it matches. You have set it to an array, which logstash converts to a hash

dictionary => {
    "Windows10" => "Kali"
    "Ubuntu" => "CentOS"
}

So that if the [System] field contains "Windows10" that is replaced with "Kali".

Sarthak_Mishra · June 26, 2021, 6:15am

Sorry, I shall explain it differently. dictionary => ["Windows10","Kali","Ubuntu","CentOS"] in this dictionary, the system names which not matchh the ones in the dictionary should be named not_found. e.g Kali in the above snapshot says not found where else kali is mentioned in my dictionary.

Cad · June 26, 2021, 9:51am

Badger explained well why it's not working.
dictionary => ["Windows10","Kali","Ubuntu","CentOS"]
is converted by logstash in

dictionary => {
    "Windows10" => "Kali"
    "Ubuntu" => "CentOS"
}

In your case, i think a dictionnary like this give the result you want

translate{
  field => "System"
  destination => "realOS"
  fallback => "not_found"
  dictionary => {
      "Windows10" => "found"
      "Ubuntu" => "found"
      "Kali" => "found"
      "CentOS" => "found"
  }
}

It put found in realOS if System contains Windows10, Ubuntu, Kali or CentOS. Otherwise it put not_found in realOS.

Sarthak_Mishra · June 26, 2021, 1:50pm

Thanks, Badger and Cad for helping me. Really appreciate it.

system · July 24, 2021, 1:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.