Logstash server NOT seeing files shipped by rsyslog from remote client?

Elastic Stack Logstash
1

Logstash is not seeing files sent over from a remote client (VM running rsyslog) on my ELK server.

I've been following the great article How to ship logs with Rsyslog and Logstash and am at the point where I'm shipping logs from my client VM to my ELK server. I verify this using tcpdump utility.

$ tcpdump -i eth0 -n host 153.65.199.92 and port 514 -P in -p -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:01:00.244844 IP (tos 0x0, ttl 61, id 20061, offset 0, flags [DF], proto TCP (6), length 60)

I use the author's textLogTemplate file in my /etc/rsyslog.d01-tr-error-log.conf file, like so

input(type="imfile"
    File="/usr/share/tomcat/dist/logs/trm-error.log"
    Facility="local3"
    Tag="trm-error-logs:"
    readMode="2"
    escapeLF="on"
)

if $programname == 'trm-error-logs:' then {
    action(
        type="omfwd"
        Target="10.128.13.4"
        Port="514"
        Protocol="tcp"
        template="textLogTemplate"
    )
    stop
}

...and have the following lines in my /etc/rsyslog.conf file

# Template for non json logs, just sends the message wholesale with extra
# # furniture. 
template(name="textLogTemplate"
  type="list") {
  constant(value="{ ")
  constant(value="\"type\":\"")
  property(name="programname")
  constant(value="\", ")
  constant(value="\"host\":\"")
  property(name="%HOSTNAME%")
  constant(value="\", ")
  constant(value="\"timestamp\":\"")
  property(name="timestamp" dateFormat="rfc3339")
  constant(value="\", ")
  constant(value="\"@version\":\"1\", ")
  constant(value="\"role\":\"app-server\", ")
  constant(value="\"sourcefile\":\"")
  property(name="$!metadata!filename")
  constant(value="\", ")
  constant(value="\"message\":\"")
  property(name="rawmsg" format="json")
  constant(value="\"}\n")
}

Here's my /etc/logstash/conf.d/03-trm-error-syslog.conf file

input {
  tcp {
    port => 514
  }
}

filter {
  if [type] == "trm-error-logs:" {
    mutate {
      strip => "message"
    }
    grok {
      match => {
        "message" => [
          "%{TIMESTAMP_ISO8601:logdate}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}:%{DATA:thread}%{SPACE}:%{SPACE}%{NUMBER:thread_pool}%{SPACE}\[(?<classname>[^\]]+)\]%{SPACE}%{GREEDYDATA:msgbody}"
        ]
      }
      add_field => [ "received_from", "%{host}" ]
    }
  }
}

output {
  file {
    path => "/tmp/trm-error.log"
  }
#  elasticsearch {
#    hosts => ["localhost:9200"]
#    index => "logstash-trm"
#  }
}

I expect to see /tmp/trm-error.log file created, but I don't. I've tested my filter using the Grok debugger and it works.

Any clues?

2

I simplified my logstash *.conf file to look like this

input {
  syslog {
    type => syslog
    port => 5514
    facility_labels => ["local3", "local4"]
  }
}

output {
  stdout { }
}

When I start logstash, I see the following, so I know logstash is listening on port 5514.

Starting syslog udp listener {:address=>"0.0.0.0:5514", :level=>:info}
Starting syslog tcp listener {:address=>"0.0.0.0:5514", :level=>:info}

However, I do this on the remote client, but logstash does not output the log on stdout

logger --port 5514 --server 10.128.13.4 "Hello sucker" -p local3.info

What gives?

3

I tracked this down to a firewall issue. I turned the firewall off, but did not disable it so when we restarted the server, it was back on.