Logstash error - Output

jgonzalez · June 4, 2020, 7:11pm

Hello All,

I have an error in my logstash config file and i have not able to find it. Please if someone can help me

This is the error:

[2020-06-04T15:00:07,633][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 115, column 9 (byte 3604) after filter{\n# Split the syslog part and Cisco tag out of the message\n grok {\n patterns_dir => "patterns"\n match => ["message", "%{CISCO_TAGGED_ASA}"]\n }\n\n# Replace the fw name if it is not in the correct format \nif [fw]=="10.60.1.4"{\n mutate{\n replace => { "fw" => "syslog101"}\n }\n}\n\n# Parse the date from the "timestamp" field to the "@timestamp" field\n date{\n match => [ "date","yyyy-MM-dd'T'HH:mm:ssZZ"]\n timezone => "America/New_York"\n target => "@timestamp"\n}\n\n# Clean up redundant fields if parsing was successful\n mutate {\n remove_field => ["message","input","prospector","host"]\n }\n\n grok {\n patterns_dir => "patterns"\n match => [\n "cisco_message", "%{CISCOFW106023}",\n "cisco_message", "%{CISCOFW302013_302014_302015_302016}",\n "cisco_message", "%{CISCOFTD430002_430003}",\n "cisco_message", "%{CISCOFW106001}",\n "cisco_message", "%{CISCOASA305013}",\n "cisco_message", "%{CISCOASA725001}",\n "cisco_message", "%{CISCOASA725007}",\n "cisco_message", "%{CISCOASA725002}",\n "cisco_message", "%{CISCOASA434002}",\n "cisco_message", "%{CISCOASA710003}",\n "cisco_message", "%{CISCOASA106016}",\n "cisco_message", "%{CISCOFW602303_602304}",\n "cisco_message", "%{CISCOASA722022_722023}",\n "cisco_message", "%{CISCOASA725003}",\n "cisco_message", "%{CISCOASA722033}",\n "cisco_message", "%{CISCOASA113009}",\n "cisco_message", "%{CISCOASA113019}",\n "cisco_message", "%{CISCOASA113008}",\n "cisco_message", "%{CISCOASA113004}",\n "cisco_message", "%{CISCOASA722037}",\n "cisco_message", "%{CISCOASA722051}",\n "cisco_message", "%{CISCOASA722032}",\n "cisco_message", "%{CISCOASA722028}",\n "cisco_message", "%{CISCOASA722034}",\n "cisco_message", "%{CISCOASA721018_721016}",\n "cisco_message", "%{CISCOASA716058_716059}",\n "cisco_message", "%{CISCOASA716002}",\n "cisco_message", "%{CISCOASA113039}",\n "cisco_message", "%{CISCOASA113012}",\n "cisco_message", "%{CISCOASA722012}",\n "cisco_message", "%{CISCOASA113005}",\n "cisco_message", "%{CISCOASA722011}",\n "cisco_message", "%{CISCOASA725006}",\n "cisco_message", "%{CISCOASA611102}",\n "cisco_message", "%{CISCOASA716039_716038}",\n "cisco_message", "%{CISCOASA113015}",\n "cisco_message", "%{CISCOASA722003_722001}",\n "cisco_message", "%{CISCOASA713119_713120}",\n "cisco_message", "%{CISCOASA713049}",\n "cisco_message", "%{CISCOASA113006}",\n "cisco_message", "%{CISCOASA113007}",\n "cisco_message", "%{CISCOFW106006_106007_106010}",\n "cisco_message", "%{CISCOFW302013_302014_302015_302016}",\n "cisco_message", "%{CISCOFW305011}",\n "cisco_message", "%{CISCOFW302020_302021}"]\n }\n\n# Mutate the indentifier\n if "SVC" in [identifier] or "AnyConnect" in [identifier] or "WebVPN" in [identifier] {\n mutate {\n replace => { "identifier" => "VPN"}\n }\n }\n\n# Mutate Deny\n if "denied" in [action]{\n mutate {\n replace => { "action" => "Deny"}\n }\n\n# Remove the cisco message after evaluate the information\n mutate {\n remove_field => ["cisco_message"]\n }\n}\n\noutput {\n # Avoid corrupt elements in Elasticsearch\n #if "_grokparsefailure" not in [tags] {\n elasticsearch {\n #hosts => "localhost:9200"\n # This rotate daily \n #index => "cisco-asa-%{+YYYY.MM.dd}"\n #manage_template => true\n #template => "/home/elkadmin/logstash-7.6.2/templates/cisco-asa.json"\n #template_name => "cisco-asa"\n #}\n #}\n # Standar Output\n stdout", :backtrace=>["/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:58:in compile_imperative'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:66:in compile_graph'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:28:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:27:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in initialize'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/java_pipeline.rb:43:in initialize'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/agent.rb:342:in block in converge_state'"]}

It is failing for anything after "output { " i tried to move some stuff
This is the pipeline that i used - prag-pipeline.conf

input {
  beats {
    port => 5044
  }
}

filter{
# Split the syslog part and Cisco tag out of the message
  grok {
    patterns_dir => "patterns"
    match => ["message", "%{CISCO_TAGGED_ASA}"]
  }

# Replace the fw name if it is not in the correct format
if [fw]=="10.60.1.4"{
  mutate{
    replace => { "fw" => "syslog101"}
  }
}

# Parse the date from the "timestamp" field to the "@timestamp" field
  date{
    match => [ "date","yyyy-MM-dd'T'HH:mm:ssZZ"]
    timezone => "America/New_York"
    target => "@timestamp"
}

# Clean up redundant fields if parsing was successful
  mutate {
    remove_field => ["message","input","prospector","host"]
  }

  grok {
    patterns_dir => "patterns"
    match => [
      "cisco_message", "%{CISCOFW106023}",
      "cisco_message", "%{CISCOFW302013_302014_302015_302016}",
      "cisco_message", "%{CISCOFTD430002_430003}",
      "cisco_message", "%{CISCOFW106001}",
      "cisco_message", "%{CISCOASA305013}",
      "cisco_message", "%{CISCOASA725001}",
      "cisco_message", "%{CISCOASA725007}",
      "cisco_message", "%{CISCOASA725002}",
      "cisco_message", "%{CISCOASA434002}",
      "cisco_message", "%{CISCOASA710003}",
      "cisco_message", "%{CISCOASA106016}",
      "cisco_message", "%{CISCOFW602303_602304}",
      "cisco_message", "%{CISCOASA722022_722023}",
      "cisco_message", "%{CISCOASA725003}",
      "cisco_message", "%{CISCOASA722033}",
      "cisco_message", "%{CISCOASA113009}",
      "cisco_message", "%{CISCOASA113019}",
      "cisco_message", "%{CISCOASA113008}",
      "cisco_message", "%{CISCOASA113004}",
      "cisco_message", "%{CISCOASA722037}",
      "cisco_message", "%{CISCOASA722051}",
      "cisco_message", "%{CISCOASA722032}",
      "cisco_message", "%{CISCOASA722028}",
      "cisco_message", "%{CISCOASA722034}",
      "cisco_message", "%{CISCOASA721018_721016}",
      "cisco_message", "%{CISCOASA716058_716059}",
      "cisco_message", "%{CISCOASA716002}",
      "cisco_message", "%{CISCOASA113039}",
      "cisco_message", "%{CISCOASA113012}",
      "cisco_message", "%{CISCOASA722012}",
      "cisco_message", "%{CISCOASA113005}",
      "cisco_message", "%{CISCOASA722011}",
      "cisco_message", "%{CISCOASA725006}",
      "cisco_message", "%{CISCOASA611102}",
      "cisco_message", "%{CISCOASA716039_716038}",
      "cisco_message", "%{CISCOASA113015}",
      "cisco_message", "%{CISCOASA722003_722001}",
      "cisco_message", "%{CISCOASA713119_713120}",
      "cisco_message", "%{CISCOASA713049}",
      "cisco_message", "%{CISCOASA113006}",
      "cisco_message", "%{CISCOASA113007}",
      "cisco_message", "%{CISCOFW106006_106007_106010}",
      "cisco_message", "%{CISCOFW302013_302014_302015_302016}",
      "cisco_message", "%{CISCOFW305011}",
      "cisco_message", "%{CISCOFW302020_302021}"]
    }

# Mutate the indentifier
  if "SVC" in [identifier] or "AnyConnect" in [identifier] or "WebVPN" in [identifier] {
    mutate {
      replace => { "identifier" => "VPN"}
    }
  }

# Mutate Deny
  if "denied" in [action]{
    mutate {
      replace => { "action" => "Deny"}
  }

# Remove the cisco message after evaluate the information
  mutate {
    remove_field => ["cisco_message"]
  }
}

output {
  # Avoid corrupt elements in Elasticsearch
  if "_grokparsefailure" not in [tags] {
    elasticsearch {
      hosts => ["http://10.70.4.142:9200","http://10.70.4.143:9200"]
      index => "cisco-asa-%{+YYYY.MM.dd}"
      manage_template => true
      template => "/home/elkadmin/logstash-7.6.2/templates/cisco-asa.json"
      template_name => "cisco-asa"
    }
  }
  # Standar Output
  #stdout{}
}

logstash.yml

# ------------ Pipeline Settings --------------
#  This defaults to the number of the host's CPU cores.
#
pipeline.workers: 16
# How many events to retrieve from inputs before sending to filters+workers
#
pipeline.batch.size: 125
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
pipeline.batch.delay: 50
# X-Pack Monitoring
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://10.70.4.142:9200", "http://10.70.4.143:9200"]
xpack.monitoring.collection.interval: 20s
xpack.monitoring.collection.pipeline.details.enabled: true

Badger · June 4, 2020, 7:49pm

No, it is not. The stdout output is uncommented in the error message. Please post an error message and the configuration file that causes it.

jgonzalez · June 4, 2020, 8:14pm

Hey Badger,

Thank you for your response, i updated a little bit the initial comment with the logstash.yml and the name of the pipeline that i put, but i don't know what do you mean exactly with the error and the config file that cause it.. i ran bin/logstash -f prag-pipeline.conf and it is not running well, i can put the whole error if you want but it a little long

ptamba · June 5, 2020, 2:51pm

this error here shows syntax error at line 115 of your config

jgonzalez · June 5, 2020, 4:58pm

I found it, the error was a "}" that I forgot to put, the strange thing was that it didn't fail in that line, but anyway thanks

    # Mutate Deny
      if "denied" in [action]{
    mutate {
      replace => { "action" => "Deny"}
      }

    # Remove the cisco message after evaluate the information

system · July 3, 2020, 4:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.