Receive multiple Logstash inputs with TCP, UDP and Beats

[sakshat]

Hi everyone, I am trying to get logs input into logstash using TCP, UDP and Beats. I have applications that drain syslog to logstash using tcp and udp and I also have an application that writes logs to files in a server. I have been trying to get those logs using Filebeat running in the server. However, I have found that TCP and Beats together don't work. When I configure Beats as an input, I receive no logs using TCP. When that Beats input is removed, the logs start coming in normally. Is there a way I could make it both work?

I have tried with TCP and Beats in different ports and also separating the two configuration into two config files and configuring the pipeline.yml. It still didn't work. Any ideas?

[joao (João Neto)]

Hi,

Is it possible to share your pipeline config?

[sakshat]

pipelines.yml

- pipeline.id: pcf
  path.config: "/etc/logstash/conf.d/syslog.conf"
- pipeline.id: vm
  path.config: "/etc/logstash/conf.d/vmlog.conf"

vmlog.conf

input {
	beats {
    port => 5044
    type => 3dlog
  }
}

filter {}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logs-%{+yyyy.MM.dd}"
  }
}

syslog.conf

input {
  tcp {
    port => 5041
    type => syslog
  }
  udp {
    port => 5041
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => {
        "message" => [
          "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:timestamp}|-) +(?:%{HOSTNAME:host_app}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) - \[(?:%{GREEDYDATA:syslog_tags})\"\] +%{TIMESTAMP_ISO8601:time} +%{LOGLEVEL:logLevel} +%{INT:logLevelNo} +%{GREEDYDATA:scheduling}] +%{GREEDYDATA:logMessage}",
          "(?:%{HOSTNAME:host_app}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{NOTSPACE}|) \[(?:%{GREEDYDATA:syslog_tags})\"\] +(?:%{HOSTNAME:app_host}) +(?:%{NOTSPACE}|) +(?:%{GREEDYDATA:logMessage})",
          "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:logMessage}"
        ]
      }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    kv {
      source => "syslog_tags"
      field_split => " "
      value_split => "="
      remove_char_key => "<>\[\],"
      remove_char_value => "<>\[\],"
      trim_key => "<>\[\],"
      trim_value => "<>\[\],"
      include_brackets => false
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        rename => [ "source_host", "syslog_hostname" ]
        rename =>[ "message", "syslog_message" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_tags", "syslog_timestamp" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-logs-%{+yyyy.MM.dd}"
  }
}

[joao (João Neto)]

With this configuration Logstash starts right?
If so, can you check the server if the ports are listened to? If so, can you telnet ports from a remote machine?

[sakshat]

Ok I found the issue. Logstash is not starting with vmlog.conf.

[2021-10-18T16:52:48,698][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:vm, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at line 5, column 3 (byte 56) after input {\n  beats {\n    port => 5044\n    type => 3dlog\n    ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391:in `block in converge_state'"]}
[2021-10-18T16:52:48,799][INFO ][logstash.runner          ] Logstash shut down.
[2021-10-18T16:52:48,808][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit

Can you tell me what's the issue here? I have edited and rewritten the file.

[sakshat]

Ok alright found the issue. Looks like you can't have a number ('3') in the type in the input. Didn't see it anywhere and so didn't realize.

[joao (João Neto)]

Great! I hope I could have helped you in some way.

Logstash logs speak a lot of things!