Logstash Apache and Spring log files issue

Our task was Read both Apache and Springboot log file and add a tag name as given below, to differentiate the logs.

  • Apache logfile -> apacheLog
  • spring-boot logfile -> javaLog

Print the standard output and write it to /usr/share/logstash/output.txt file

we written code as below which is not working please help us -

input { 
  file { 
    type => "apache" 
    path => [ "/usr/share/logstash/logstash-tutorial.log" ] 
    start_position => "beginning"
    ignore_older => 0
  }
  file { 
    type => "java" 
    path => [ "/usr/share/logstash/Application-Log/springboot3.log" ] 
    codec => multiline {
      pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
      negate => "true"
      what => "previous"
    }
  }
}

filter {
  if [message] =~ "%{COMBINEDAPACHELOG}" {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    add_tag => ["apacheLog"]
  }
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["JavaLog"]
    }
  }

output {
   stdout {
     codec => 'rubydebug'
   }
    file {
       path => "/usr/share/logstash/output.txt"
    }
 }

i think logstash already adds tag witht the path in it.

our error si

[2020-10-12T07:19:46,485][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 33, column 11 (byte 694) after filter {\n if [message] =~ "%{COMBINEDAPACHELOG}" {\n grok {\n match => { "message" => "%{COMBINEDAPACHELOG}" }\n add_tag => ["apacheLog"]\n }\n if [message] =~ "\tat" {\n grok {\n match => ["message", "^(\tat)"]\n add_tag => ["JavaLog"]\n }\n }\n\noutput {\n stdout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2020-10-12T07:19:46,605][INFO ][logstash.inputs.metrics ] Monitoring License OK
[2020-10-12T07:19:48,676][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x2f0da7ca run>"}

We feel issue with

filter {
  if [message] =~ "%{COMBINEDAPACHELOG}" {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    add_tag => ["apacheLog"]
  }
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["JavaLog"]
    }
  }

The error you quoted is caused by a missing } to end the filter section, so logstash is trying to parse the output section as a filter configuration.

In filebeat, setting ignore_older to zero disables age based filtering. In logstash, it configures the input to ignore any files more then zero seconds old, which is all files, so the file input will not read anything.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.