Hi everyone, I am trying to get logs input into logstash using TCP, UDP and Beats. I have applications that drain syslog to logstash using tcp and udp and I also have an application that writes logs to files in a server. I have been trying to get those logs using Filebeat running in the server. However, I have found that TCP and Beats together don't work. When I configure Beats as an input, I receive no logs using TCP. When that Beats input is removed, the logs start coming in normally. Is there a way I could make it both work?
I have tried with TCP and Beats in different ports and also separating the two configuration into two config files and configuring the pipeline.yml. It still didn't work. Any ideas?
joao
(João Neto)
October 18, 2021, 7:57pm
2
Hi,
Is it possible to share your pipeline config?
pipelines.yml
- pipeline.id: pcf
path.config: "/etc/logstash/conf.d/syslog.conf"
- pipeline.id: vm
path.config: "/etc/logstash/conf.d/vmlog.conf"
vmlog.conf
input {
beats {
port => 5044
type => 3dlog
}
}
filter {}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logs-%{+yyyy.MM.dd}"
}
}
syslog.conf
input {
tcp {
port => 5041
type => syslog
}
udp {
port => 5041
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => {
"message" => [
"%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:timestamp}|-) +(?:%{HOSTNAME:host_app}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) - \[(?:%{GREEDYDATA:syslog_tags})\"\] +%{TIMESTAMP_ISO8601:time} +%{LOGLEVEL:logLevel} +%{INT:logLevelNo} +%{GREEDYDATA:scheduling}] +%{GREEDYDATA:logMessage}",
"(?:%{HOSTNAME:host_app}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{NOTSPACE}|) \[(?:%{GREEDYDATA:syslog_tags})\"\] +(?:%{HOSTNAME:app_host}) +(?:%{NOTSPACE}|) +(?:%{GREEDYDATA:logMessage})",
"%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:logMessage}"
]
}
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
kv {
source => "syslog_tags"
field_split => " "
value_split => "="
remove_char_key => "<>\[\],"
remove_char_value => "<>\[\],"
trim_key => "<>\[\],"
trim_value => "<>\[\],"
include_brackets => false
}
if !("_grokparsefailure" in [tags]) {
mutate {
rename => [ "source_host", "syslog_hostname" ]
rename =>[ "message", "syslog_message" ]
}
}
mutate {
remove_field => [ "syslog_hostname", "syslog_message", "syslog_tags", "syslog_timestamp" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "app-logs-%{+yyyy.MM.dd}"
}
}
joao
(João Neto)
October 18, 2021, 8:47pm
4
With this configuration Logstash starts right?
If so, can you check the server if the ports are listened to? If so, can you telnet ports from a remote machine?
Ok I found the issue. Logstash is not starting with vmlog.conf.
[2021-10-18T16:52:48,698][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:vm, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at line 5, column 3 (byte 56) after input {\n beats {\n port => 5044\n type => 3dlog\n ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391:in `block in converge_state'"]}
[2021-10-18T16:52:48,799][INFO ][logstash.runner ] Logstash shut down.
[2021-10-18T16:52:48,808][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
Can you tell me what's the issue here? I have edited and rewritten the file.
Ok alright found the issue. Looks like you can't have a number ('3') in the type in the input. Didn't see it anywhere and so didn't realize.
joao
(João Neto)
October 18, 2021, 9:06pm
7
Great! I hope I could have helped you in some way.
Logstash logs speak a lot of things!
1 Like
system
(system)
Closed
November 15, 2021, 9:06pm
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.