Filebeat vs Logstash

Elastic Stack Logstash
81

What we have created so far is this:

You send many different logs. If it is an apache log then filter it and add a tag that implies this is an apache log from whatever server.
In the output we say "is this tagged as apache log?". If so then write all the tagged apache events,from all sources, to a single file.
We never checked for other logs.

Why were you seeing events before but after my fix you did not?
Becouse you never checked for the tag and just dumped everything out.
Now that you are checking for the tag , and NOTHING gets tagged since they are not coming from apache, you see nothing.

Please try to read and understand what i am trying to explain here.

82

I feel like you do not want to help yourself.
This will be my last post i already gave you EVERYTHING you need to understand and solver your problem.
Your error is clear. There is a special character missing.

output {
   if "apache_logs" in [tags] {
        file {
       path => "/home/ubuntu/apache.log"
  }
}   
}  # <- this was missing on output. Please count curly braces even if other write it to help you.

It even tells you where exactly it was :

"LogStash::ConfigurationError", :message=>"Expected one of #, else, if, ", ', } at line 23, column 1 (byte 336) after output {\n if "apache_logs" in [tags] {\n file {\n path => "/home/ubuntu/apache.log"\n }\n}\n",

83

sorry its there, while copying it missed.

84

config file --> /usr/share/logstash/apache.conf

# INPUT HERE
  input {
      beats {
                port => 5044
        }
 }
# FILTER HERE
 filter{
        if [source]=="/var/log/apache2/error.log"
        {
        mutate {
            remove_tag => [ "beats_input_codec_plain_applied" ]
            add_tag => [ "apache_logs" ]
         }
        }
            if [source]=="/var/log/apache2/access.log"
        {
        mutate {
            remove_tag => [ "beats_input_codec_plain_applied" ]
            add_tag => [ "apache_logs" ]
         }
        }
    }
    # OUTPUT HERE
    output {
       if "apache_logs" in [tags] {
                file {
               path => "/home/ubuntu/apache.log"
                    codec => "json"
      }
    }

output --> /home/ubuntu/apache.log

42,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:01 +0000] \"GET /favicon.ico HTTP/1.1\" 404 504 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T06:33:06.508Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","fileset":{"name":"access","module":"apache2"},"offset":569,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:00 +0000] \"GET /icons/ubuntu-logo.png HTTP/1.1\" 200 3623 \"http://35.154.56.205/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T06:33:06.590Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","offset":365,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:00 +0000] \"GET / HTTP/1.1\" 200 3525 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T06:33:06.507Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","fileset":{"name":"access","module":"apache2"},"offset":365,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:00 +0000] \"GET / HTTP/1.1\" 200 3525 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T06:33:06.508Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","fileset":{"name":"access","module":"apache2"},"offset":742,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:01 +0000] \"GET /favicon.ico HTTP/1.1\" 404 504 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T06:33:06.590Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","offset":569,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:00 +0000] \"GET /icons/ubuntu-logo.png HTTP/1.1\" 200 3623 \"http://35.154.56.205/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T06:33:06.590Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","offset":742,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:06:33:01 +0000] \"GET /favicon.ico HTTP/1.1\" 404 504 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-223"},"@timestamp":"2018-04-17T07:00:21.558Z","source":"/var/log/apache2/access.log","host":"ip-192-168-2-223","fileset":{"name":"access","module":"apache2"},"offset":940,"@version":"1","message":"14.142.151.180 - - [17/Apr/2018:07:00:13 +0000] \"GET / HTTP/1.1\" 200 3525 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15\"","prospector":{"type":"log"}}{"tags":["apache_logs"],"beat":{"version":"6.2.3","hostname":"ip-192-168-2-223","name":"ip-192-168-2-2

Thanks for support
@sancroth

85

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.