Grok filter

Elastic Stack Logstash
1

Hello,

I want to create a grok filter. e.g There is a column in my csv file with all the os system listed windows,ubuntu etc. If there is any typo in the column. I want logstash to create a new field with the name "is_os_correct" and have a false value for the the typo column.

input{

file

   {

    path=>"C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter{

csv

{

separator => ","

columns => ["Name","Age","System"]

}

grok { match => {"System" => "%{WORD:validSystem}" } }

if "_grokparsefailure" in [tags] {

  mutate {

    add_field => { "is_os_correct" => "false" }

  }

}

else

{

  mutate

{

    add_field => { "is_os_correct" => "true" }

  }

}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

2

You are only going to get a grok failure if the field contains characters outside of the group [a-zA-Z0-9_].

Do you have a list of system names that are acceptable? If so, using a translate filter might be better.

1 Like
3

HI Badger, can you please help me further on it

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

    separator => ","

    columns => ["Name","Age","System"]

}

translate {

    field => "System"

    destination => "realOS"

    fall_back => "not_found"

    dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

    match => [ %{WORD:System}]

}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

This is my error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logs which is now configured via log4j2.properties
[2021-06-24T16:32:17,029][INFO ][logstash.runner ] Log4j configuration path used is: C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\config\log4j2.properties
[2021-06-24T16:32:17,041][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T16:32:17,101][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-24T16:32:17,969][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T16:32:18,107][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{", "]" at line 22, column 20 (byte 467) after filter {\r\n csv {\r\n separator => ","\r\n columns => ["Name","Age","System"]\r\n }\r\n\r\n translate {\r\n field => "System"\r\n destination => "realOS"\r\n fall_back => "not_found"\r\n dictionary => ["Windows10","Kali","Ubuntu","CentOS"]\r\n }\r\n\r\n grok {\r\n match => [ ", :backtrace=>["C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:389:in block in converge_state'"]}
[2021-06-24T16:32:18,207][INFO ][logstash.runner ] Logstash shut down.
[2021-06-24T16:32:18,207][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.lib.bootstrap.environment.(C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

5

Hi,
I still get an error can you help be debug it

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

separator => ","

columns => ["Name","Age","System"]

}

translate {

field => "System"

destination => "realOS"

fall_back => "not_found"

dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

match => [ "%{WORD:System}" ]

}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

6

Sorry i give the answer to fast.

The match option need

  • a field to apply the regex.
  • one or more regex.

So it need to be like this :

grok {
match => { "field" => ["%{WORD:System}"]}
}

Or

grok {
match => { "field" => "%{WORD:System}"}
}

But i don't understand why you add a grok filter if the translate filter do the job.

7

I tried doing it with the only translate it did not work.So I added grok to check my luck. Can you please suggest something

input {

file {

    path => "C:/Users/kumar/Desktop/grok.csv"

    start_position => "beginning"

    sincedb_path => "NULL"

}

}

filter {

csv {

separator => ","

columns => ["Name","Age","System"]

}

translate {

field => "System"

destination => "realOS"

fall_back => "not_found"

dictionary => ["Windows10","Kali","Ubuntu","CentOS"]

}

grok {

match => { "field" => ["%{WORD:System}"]}

}

}

output {

elasticsearch{

    hosts => "http://localhost:9200/"

    index => "grok"

}

stdout{}

}

"Using bundled JDK: ""
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logs which is now configured via log4j2.properties
[2021-06-24T17:33:21,874][INFO ][logstash.runner ] Log4j configuration path used is: C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\config\log4j2.properties
[2021-06-24T17:33:21,883][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T17:33:21,948][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-24T17:33:22,961][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T17:33:26,921][INFO ][org.reflections.Reflections] Reflections took 192 ms to scan 1 urls, producing 23 keys and 47 values
[2021-06-24T17:33:27,758][ERROR][logstash.filters.translate] Unknown setting 'fall_back' for translate
[2021-06-24T17:33:27,763][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:129
add at org/logstash/execution/ConvergeResultExt.java:57
converge_state at C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:402
[2021-06-24T17:33:27,771][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[2021-06-24T17:33:27,781][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in create'", "org/logstash/execution/ConvergeResultExt.java:57:in add'", "C:/elastic_stack/logstash-7.12.1-windows-x86_64/logstash-7.12.1/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
[2021-06-24T17:33:27,788][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at C_3a_.elastic_stack.logstash_minus_7_dot_12_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_12_dot_1.lib.bootstrap.environment.(C:\elastic_stack\logstash-7.12.1-windows-x86_64\logstash-7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

8

It is fallback instead of fall_back.

Try this

input {
  file {
    path => "C:/Users/kumar/Desktop/grok.csv"
    start_position => "beginning"
    sincedb_path => "NULL"
  }
}

filter {
  csv {
    separator => ","
    columns => ["Name","Age","System"]
  }

  translate {
    field => "System"
    destination => "realOS"
    fallback => "not_found"
    dictionary => ["Windows10","Kali","Ubuntu","CentOS"]
  }
}

output {
  elasticsearch{
    hosts => "http://localhost:9200/"
    index => "grok"
  }
  stdout{}
}
1 Like
10

Thanks it has worked. Can i add something like fallback=not found for things that match the dictionary which say "found". If yes, can you please tell me how

11

Take a look at this

Cad.

12

image
Hi again. the code was supposed to show "not_found" only for the system not in my dictionary. But it shows "not_found" even for some system names which are in my dictionary. Can you please help to identify where the algorithm lacks? Thanks for helping

13

The dictionary should be a hash showing what value should replace the value in the [System] field if it matches. You have set it to an array, which logstash converts to a hash

dictionary => {
    "Windows10" => "Kali"
    "Ubuntu" => "CentOS"
}

So that if the [System] field contains "Windows10" that is replaced with "Kali".

14

Sorry, I shall explain it differently. dictionary => ["Windows10","Kali","Ubuntu","CentOS"] in this dictionary, the system names which not matchh the ones in the dictionary should be named not_found. e.g Kali in the above snapshot says not found where else kali is mentioned in my dictionary.

15

Badger explained well why it's not working.
dictionary => ["Windows10","Kali","Ubuntu","CentOS"]
is converted by logstash in

dictionary => {
    "Windows10" => "Kali"
    "Ubuntu" => "CentOS"
}

In your case, i think a dictionnary like this give the result you want

translate{
  field => "System"
  destination => "realOS"
  fallback => "not_found"
  dictionary => {
      "Windows10" => "found"
      "Ubuntu" => "found"
      "Kali" => "found"
      "CentOS" => "found"
  }
}

It put found in realOS if System contains Windows10, Ubuntu, Kali or CentOS. Otherwise it put not_found in realOS.

16

Thanks, Badger and Cad for helping me. Really appreciate it.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.