Logstash listening on port 5000 and is able to connect with Filebeat but not parsing logs

Elastic Stack Logstash
1

Hii I have this architecture with me -
Filebeat ships logs to logstash(listening on port 5000) . Logstash parses logs (i can see on my console too) and pushes them to elasticsearch. Now when I am running my Filebeat , it is able to connect to Logstash as it says "sending logs to logstash" . Further I am doing "tcpdump port 5000" on my Logstash server to see whether I am receiving any traffic or not . And the thing is that I am indeed receiving traffic at port 5000. But I am not seeing Logstash parsing any log and pushing to elasticsearch . What can be the issue ??? First I thought of some communication problem between Logstash and Filebeat due to certificate and key but then I thought this cannot be the case as Filebeat is able to connect to Logstash .If there would have been a certificate problem Filebeat could not have connected to Logstash right ????

2

this is my Logstash configuration file . I am posting input { } , filter {} , and output{} separately.

input {

    beats {
            codec =>  multiline {
                            pattern => "^\["
                            what => "previous"
                            negate => true
             }
            ssl => true
            port => 5000
            ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
            ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    }

}

3

Have you looked in the Logstash logs to confirm that it's getting the events and seeing whether it has any problems sending the logs to ES? Upping the log level with --verbose or --debug may provide additional insights.

4

filter {
grok {
match => { "message" => "%{CISCO_REASON}:%{ISO8601_SECOND},%{ISO8601_SECOND}%{DATA}%{LOGLEVEL:loglevel}"}
match => { "message" => "%{SYSLOG5424SD} %{JAVACLASS:loglevel}" }

    }
    geoip {
            source => "clientip"
    }

    date {
            match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
    }

}

5

where can i see the logstash logs ??? i am working on ubuntu .

6

I don't see anything when I go to /var/log/logstash

7

How are you starting Logstash?

8

i am starting logstash by using the command -
/opt/logstash/bin/logstash -f /etc/logstash/logstash.conf

9

@magnusbaek can you specify the path for logstash logs . Doing -
tail /var/log/logstash/logstash.log
is of no use as the directory /var/log/logstash does not contain anything

10

Then Logstash will log to your terminal. I suggest you save the output to a file (e.g. with the -l flag) since enabling verbose output will produce a lot of text.

11

@manusbaeck sorry don't get you . Do you want me to put the output of --debug into a file ?

12

These kind of probelms are faced when there is a problem in the configuration. Do you see any problem with the configuration ?

13

Do you want me to put the output of --debug into a file ?

Yes, if you want to be able to inspect the data you're going to want to have it in a file.

14

so how do I need to do that ?? can you tell me the command ? will it be like -
/opt/logstash/bin/logstash -f /etc/logstash/logstash.conf -l /path/to/file

15

@magnusbaeck what do you think can be the problem???

17

@magnusbaeck Hey I am seeing a peculiar behavior . When I removed the codec plugin from input {} things worked fine . I need that codec plugin in order to merge stack traces of laravel logs with the log itself so that logstash does not treat the stack trace as separate logs . Any help with this ???? This should not happen , don't you think so ??? So where should I put my codec multiline ?

18

If you have multiple different types of logs coming in and the multiline codec is not able to handle some of them, e.g. because each line does not start with the pattern the codec is looking for, you may need to specify multiple inputs and separate them.

19

I don't think you can use the multiline codec with the beats input. The event arrives from Filebeat in structured form but the multiline codec assumes line-based input. You may be able to use the multiline filter instead, but ideally you should join the lines on the Filebeat side. That will be possible in Filebeat 1.1 which will be released soon. There are pre-release binaries to download or you can build the source yourself.

1 Like