Logstash error - Output

jgonzalez · June 4, 2020, 7:11pm

Hello All,

I have an error in my logstash config file and i have not able to find it. Please if someone can help me

This is the error:

[2020-06-04T15:00:07,633][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 115, column 9 (byte 3604) after filter{\n# Split the syslog part and Cisco tag out of the message\n grok {\n patterns_dir => "patterns"\n match => ["message", "%{CISCO_TAGGED_ASA}"]\n }\n\n# Replace the fw name if it is not in the correct format \nif [fw]=="10.60.1.4"{\n mutate{\n replace => { "fw" => "syslog101"}\n }\n}\n\n# Parse the date from the "timestamp" field to the "@timestamp" field\n date{\n match => [ "date","yyyy-MM-dd'T'HH:mm:ssZZ"]\n timezone => "America/New_York"\n target => "@timestamp"\n}\n\n# Clean up redundant fields if parsing was successful\n mutate {\n remove_field => ["message","input","prospector","host"]\n }\n\n grok {\n patterns_dir => "patterns"\n match => [\n "cisco_message", "%{CISCOFW106023}",\n "cisco_message", "%{CISCOFW302013_302014_302015_302016}",\n "cisco_message", "%{CISCOFTD430002_430003}",\n "cisco_message", "%{CISCOFW106001}",\n "cisco_message", "%{CISCOASA305013}",\n "cisco_message", "%{CISCOASA725001}",\n "cisco_message", "%{CISCOASA725007}",\n "cisco_message", "%{CISCOASA725002}",\n "cisco_message", "%{CISCOASA434002}",\n "cisco_message", "%{CISCOASA710003}",\n "cisco_message", "%{CISCOASA106016}",\n "cisco_message", "%{CISCOFW602303_602304}",\n "cisco_message", "%{CISCOASA722022_722023}",\n "cisco_message", "%{CISCOASA725003}",\n "cisco_message", "%{CISCOASA722033}",\n "cisco_message", "%{CISCOASA113009}",\n "cisco_message", "%{CISCOASA113019}",\n "cisco_message", "%{CISCOASA113008}",\n "cisco_message", "%{CISCOASA113004}",\n "cisco_message", "%{CISCOASA722037}",\n "cisco_message", "%{CISCOASA722051}",\n "cisco_message", "%{CISCOASA722032}",\n "cisco_message", "%{CISCOASA722028}",\n "cisco_message", "%{CISCOASA722034}",\n "cisco_message", "%{CISCOASA721018_721016}",\n "cisco_message", "%{CISCOASA716058_716059}",\n "cisco_message", "%{CISCOASA716002}",\n "cisco_message", "%{CISCOASA113039}",\n "cisco_message", "%{CISCOASA113012}",\n "cisco_message", "%{CISCOASA722012}",\n "cisco_message", "%{CISCOASA113005}",\n "cisco_message", "%{CISCOASA722011}",\n "cisco_message", "%{CISCOASA725006}",\n "cisco_message", "%{CISCOASA611102}",\n "cisco_message", "%{CISCOASA716039_716038}",\n "cisco_message", "%{CISCOASA113015}",\n "cisco_message", "%{CISCOASA722003_722001}",\n "cisco_message", "%{CISCOASA713119_713120}",\n "cisco_message", "%{CISCOASA713049}",\n "cisco_message", "%{CISCOASA113006}",\n "cisco_message", "%{CISCOASA113007}",\n "cisco_message", "%{CISCOFW106006_106007_106010}",\n "cisco_message", "%{CISCOFW302013_302014_302015_302016}",\n "cisco_message", "%{CISCOFW305011}",\n "cisco_message", "%{CISCOFW302020_302021}"]\n }\n\n# Mutate the indentifier\n if "SVC" in [identifier] or "AnyConnect" in [identifier] or "WebVPN" in [identifier] {\n mutate {\n replace => { "identifier" => "VPN"}\n }\n }\n\n# Mutate Deny\n if "denied" in [action]{\n mutate {\n replace => { "action" => "Deny"}\n }\n\n# Remove the cisco message after evaluate the information\n mutate {\n remove_field => ["cisco_message"]\n }\n}\n\noutput {\n # Avoid corrupt elements in Elasticsearch\n #if "_grokparsefailure" not in [tags] {\n elasticsearch {\n #hosts => "localhost:9200"\n # This rotate daily \n #index => "cisco-asa-%{+YYYY.MM.dd}"\n #manage_template => true\n #template => "/home/elkadmin/logstash-7.6.2/templates/cisco-asa.json"\n #template_name => "cisco-asa"\n #}\n #}\n # Standar Output\n stdout", :backtrace=>["/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:58:in compile_imperative'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:66:in compile_graph'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:28:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/compiler.rb:27:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in initialize'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/java_pipeline.rb:43:in initialize'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/home/elkadmin/logstash-7.7.0/logstash-core/lib/logstash/agent.rb:342:in block in converge_state'"]}

It is failing for anything after "output { " i tried to move some stuff
This is the pipeline that i used - prag-pipeline.conf

input {
  beats {
    port => 5044
  }
}

filter{
# Split the syslog part and Cisco tag out of the message
  grok {
    patterns_dir => "patterns"
    match => ["message", "%{CISCO_TAGGED_ASA}"]
  }

# Replace the fw name if it is not in the correct format
if [fw]=="10.60.1.4"{
  mutate{
    replace => { "fw" => "syslog101"}
  }
}

# Parse the date from the "timestamp" field to the "@timestamp" field
  date{
    match => [ "date","yyyy-MM-dd'T'HH:mm:ssZZ"]
    timezone => "America/New_York"
    target => "@timestamp"
}

# Clean up redundant fields if parsing was successful
  mutate {
    remove_field => ["message","input","prospector","host"]
  }

  grok {
    patterns_dir => "patterns"
    match => [
      "cisco_message", "%{CISCOFW106023}",
      "cisco_message", "%{CISCOFW302013_302014_302015_302016}",
      "cisco_message", "%{CISCOFTD430002_430003}",
      "cisco_message", "%{CISCOFW106001}",
      "cisco_message", "%{CISCOASA305013}",
      "cisco_message", "%{CISCOASA725001}",
      "cisco_message", "%{CISCOASA725007}",
      "cisco_message", "%{CISCOASA725002}",
      "cisco_message", "%{CISCOASA434002}",
      "cisco_message", "%{CISCOASA710003}",
      "cisco_message", "%{CISCOASA106016}",
      "cisco_message", "%{CISCOFW602303_602304}",
      "cisco_message", "%{CISCOASA722022_722023}",
      "cisco_message", "%{CISCOASA725003}",
      "cisco_message", "%{CISCOASA722033}",
      "cisco_message", "%{CISCOASA113009}",
      "cisco_message", "%{CISCOASA113019}",
      "cisco_message", "%{CISCOASA113008}",
      "cisco_message", "%{CISCOASA113004}",
      "cisco_message", "%{CISCOASA722037}",
      "cisco_message", "%{CISCOASA722051}",
      "cisco_message", "%{CISCOASA722032}",
      "cisco_message", "%{CISCOASA722028}",
      "cisco_message", "%{CISCOASA722034}",
      "cisco_message", "%{CISCOASA721018_721016}",
      "cisco_message", "%{CISCOASA716058_716059}",
      "cisco_message", "%{CISCOASA716002}",
      "cisco_message", "%{CISCOASA113039}",
      "cisco_message", "%{CISCOASA113012}",
      "cisco_message", "%{CISCOASA722012}",
      "cisco_message", "%{CISCOASA113005}",
      "cisco_message", "%{CISCOASA722011}",
      "cisco_message", "%{CISCOASA725006}",
      "cisco_message", "%{CISCOASA611102}",
      "cisco_message", "%{CISCOASA716039_716038}",
      "cisco_message", "%{CISCOASA113015}",
      "cisco_message", "%{CISCOASA722003_722001}",
      "cisco_message", "%{CISCOASA713119_713120}",
      "cisco_message", "%{CISCOASA713049}",
      "cisco_message", "%{CISCOASA113006}",
      "cisco_message", "%{CISCOASA113007}",
      "cisco_message", "%{CISCOFW106006_106007_106010}",
      "cisco_message", "%{CISCOFW302013_302014_302015_302016}",
      "cisco_message", "%{CISCOFW305011}",
      "cisco_message", "%{CISCOFW302020_302021}"]
    }

# Mutate the indentifier
  if "SVC" in [identifier] or "AnyConnect" in [identifier] or "WebVPN" in [identifier] {
    mutate {
      replace => { "identifier" => "VPN"}
    }
  }

# Mutate Deny
  if "denied" in [action]{
    mutate {
      replace => { "action" => "Deny"}
  }

# Remove the cisco message after evaluate the information
  mutate {
    remove_field => ["cisco_message"]
  }
}

output {
  # Avoid corrupt elements in Elasticsearch
  if "_grokparsefailure" not in [tags] {
    elasticsearch {
      hosts => ["http://10.70.4.142:9200","http://10.70.4.143:9200"]
      index => "cisco-asa-%{+YYYY.MM.dd}"
      manage_template => true
      template => "/home/elkadmin/logstash-7.6.2/templates/cisco-asa.json"
      template_name => "cisco-asa"
    }
  }
  # Standar Output
  #stdout{}
}

logstash.yml

# ------------ Pipeline Settings --------------
#  This defaults to the number of the host's CPU cores.
#
pipeline.workers: 16
# How many events to retrieve from inputs before sending to filters+workers
#
pipeline.batch.size: 125
# How long to wait in milliseconds while polling for the next event
# before dispatching an undersized batch to filters+outputs
#
pipeline.batch.delay: 50
# X-Pack Monitoring
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://10.70.4.142:9200", "http://10.70.4.143:9200"]
xpack.monitoring.collection.interval: 20s
xpack.monitoring.collection.pipeline.details.enabled: true

Badger · June 4, 2020, 7:49pm

No, it is not. The stdout output is uncommented in the error message. Please post an error message and the configuration file that causes it.

jgonzalez · June 4, 2020, 8:14pm

Hey Badger,

Thank you for your response, i updated a little bit the initial comment with the logstash.yml and the name of the pipeline that i put, but i don't know what do you mean exactly with the error and the config file that cause it.. i ran bin/logstash -f prag-pipeline.conf and it is not running well, i can put the whole error if you want but it a little long

ptamba · June 5, 2020, 2:51pm

this error here shows syntax error at line 115 of your config

jgonzalez · June 5, 2020, 4:58pm

I found it, the error was a "}" that I forgot to put, the strange thing was that it didn't fail in that line, but anyway thanks

    # Mutate Deny
      if "denied" in [action]{
    mutate {
      replace => { "action" => "Deny"}
      }

    # Remove the cisco message after evaluate the information

system · July 3, 2020, 4:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash Config File Error2 Logstash	13	715	March 29, 2019
Can someone help with the problem? logstasherror Logstash	2	327	December 25, 2018
Logstash Configuration Error -**Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 14 Logstash	3	4558	July 11, 2018
Logstash error log Logstash	2	563	February 7, 2018
Logstash agent failed to execute action Logstash	2	11	November 2, 2024

Logstash error - Output

Related topics