Receive multiple Logstash inputs with TCP, UDP and Beats

Hi everyone, I am trying to get logs input into logstash using TCP, UDP and Beats. I have applications that drain syslog to logstash using tcp and udp and I also have an application that writes logs to files in a server. I have been trying to get those logs using Filebeat running in the server. However, I have found that TCP and Beats together don't work. When I configure Beats as an input, I receive no logs using TCP. When that Beats input is removed, the logs start coming in normally. Is there a way I could make it both work?

I have tried with TCP and Beats in different ports and also separating the two configuration into two config files and configuring the pipeline.yml. It still didn't work. Any ideas?

Hi,

Is it possible to share your pipeline config?

pipelines.yml

- pipeline.id: pcf
  path.config: "/etc/logstash/conf.d/syslog.conf"
- pipeline.id: vm
  path.config: "/etc/logstash/conf.d/vmlog.conf"

vmlog.conf

input {
	beats {
    port => 5044
    type => 3dlog
  }
}

filter {}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logs-%{+yyyy.MM.dd}"
  }
}

syslog.conf

input {
  tcp {
    port => 5041
    type => syslog
  }
  udp {
    port => 5041
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => {
        "message" => [
          "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:timestamp}|-) +(?:%{HOSTNAME:host_app}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) - \[(?:%{GREEDYDATA:syslog_tags})\"\] +%{TIMESTAMP_ISO8601:time} +%{LOGLEVEL:logLevel} +%{INT:logLevelNo} +%{GREEDYDATA:scheduling}] +%{GREEDYDATA:logMessage}",
          "(?:%{HOSTNAME:host_app}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{NOTSPACE}|) \[(?:%{GREEDYDATA:syslog_tags})\"\] +(?:%{HOSTNAME:app_host}) +(?:%{NOTSPACE}|) +(?:%{GREEDYDATA:logMessage})",
          "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:logMessage}"
        ]
      }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    kv {
      source => "syslog_tags"
      field_split => " "
      value_split => "="
      remove_char_key => "<>\[\],"
      remove_char_value => "<>\[\],"
      trim_key => "<>\[\],"
      trim_value => "<>\[\],"
      include_brackets => false
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        rename => [ "source_host", "syslog_hostname" ]
        rename =>[ "message", "syslog_message" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_tags", "syslog_timestamp" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-logs-%{+yyyy.MM.dd}"
  }
}

With this configuration Logstash starts right?
If so, can you check the server if the ports are listened to? If so, can you telnet ports from a remote machine?

Ok I found the issue. Logstash is not starting with vmlog.conf.

[2021-10-18T16:52:48,698][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:vm, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at line 5, column 3 (byte 56) after input {\n  beats {\n    port => 5044\n    type => 3dlog\n    ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391:in `block in converge_state'"]}
[2021-10-18T16:52:48,799][INFO ][logstash.runner          ] Logstash shut down.
[2021-10-18T16:52:48,808][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit

Can you tell me what's the issue here? I have edited and rewritten the file.

Ok alright found the issue. Looks like you can't have a number ('3') in the type in the input. Didn't see it anywhere and so didn't realize.

Great! I hope I could have helped you in some way.

Logstash logs speak a lot of things!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.