Can't create a field with a variable from a grok match regex

[TheSmartMonkey]

Hi all,

I am currently using logstash, elasticsearch and kibana 6.3.0

My log are generated at a unique id path : /tmp/USER_DATA/FactoryContainer/images/(my unique id)/oar/oar_image_job(my unique id).stdout

What i want to do is to match this unique id and to create a field with this id.

I m a bit novice to logstash filter but I don't know why it doesn't want to use my uid and always return me %{uid} in my field or this Failed to execute action error

my filter :

    input {
      file {
        path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
        start_position => "beginning"
        add_field => { "data_source" => "oar-image-job" }
       }
    }
    
    filter {
        grok {
            match => ["path","%{UNIXPATH}%{NUMBER:uid}%{UNIXPATH}"]
        }
        mutate {
            add_field => [ "unique_id" => "%{uid}" ]
        }
    }
    
    output {
      if [data_source] == "oar-image-job" {
        elasticsearch {
            index => "oar-image-job-%{+YYYY.MM.dd}"
    	    hosts => ["localhost:9200"]
    	    }
        }
    }

the data_source field is to avoid this issue : When you put multiple config files in a directory for Logstash to use, they will all be concatenated

in the grok debugger %{UNIXPATH}%{NUMBER:uid}%{UNIXPATH} my path return me the good value

[Badger]

Avoid UNIXPATH. It is extremely expensive.

You know what the path looks like. Why not reference it?

grok { match => { "path" => [ "/images/%{DATA:id1}/oar/oar_images_job%{DATA:id2}.stdout" ] } }

[TheSmartMonkey]

Good point but it don't resolve my issue, other ideas ?

input {
  file {
    path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
    start_position => "beginning"
    add_field => { "data_source" => "oar-image-job" }
   }
}

filter {
    grok {
        match => { "path" => [ "/tmp/USER_DATA/FactoryContainer/images/%{DATA:id1}/oar/oar_image_job%{DATA:id2}.stdout" ] }
    }
    mutate {
        add_field => [ "unique_id" => "%{id1}" ]
    }
}

output {
  if [data_source] == "oar-image-job" {
    elasticsearch {
    	index => "oar-image-job-%{+YYYY.MM.dd}"
	hosts => ["localhost:9200"]
	}
    }
}

[Badger]

@TheSmartMonkey

Good point but it don't resolve my issue

What is your issue?

[TheSmartMonkey]

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, ,, ] at line 64, column 36 (byte 1305) after filter {\n grok {\n match => { "[path]" => [ "/tmp/USER_DATA/FactoryContainer/images/%{DATA:id1}/oar/oar_image_job%{DATA:id2}.stdout" ] }\n }\n mutate {\n add_field => [ "unique_id" ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

[Badger]

mutate {
    add_field => { "unique_id" => "%{id1}" }
}

Use braces, not brackets. It wants a hash, not an array.

[TheSmartMonkey]

Thanks a lot @Badger I just removed the mutate and changed the data variable because it add a field automatically don't need add_field

the correct filter:

input {
  file {
    path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
    start_position => "beginning"
    add_field => { "data_source" => "oar-image-job" }
   }
}

filter {
    grok {
        match => { "path" => [ "/tmp/USER_DATA/FactoryContainer/images/%{DATA:unique_id}/oar/oar_image_job%{DATA}.stdout" ] }
    }
  
}

output {
  if [data_source] == "oar-image-job" {
    elasticsearch {
    	index => "oar-image-job-%{+YYYY.MM.dd}"
	hosts => ["localhost:9200"]
	}
    }
}