Sending logs through logstash using email output plugin?

Elastic Stack Logstash
1

Hi

  [2017-01-14 10:48:06,848][WARN ][index.search.slowlog.query] [yaswanth] [bank][0] took[27.8ms], took_millis[27], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[], extra_source[], 
    [2017-01-14 10:48:06,851][WARN ][index.search.slowlog.query] [yaswanth] [bank][3] took[12.7ms], took_millis[33], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[], extra_source[],

Using below config i am sending the logs to my email using logstash:

    input {
  file {
    path => "C:\Users\571952\Desktop\pica.txt"
    start_position => "beginning"
  }
}

filter {
    grok {
       match => [ "message", 

"\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:QUERY}\]%{SPACE}\[%{DATA:QUE

RY1}\]%{SPACE}\[%{DATA:INDEX-NAME}\]\[%{DATA:SHARD}\]%{SPACE}took\[%{DATA:TOOK}\],%{SPACE}

took_millis\[%{DATA:TOOKM}\], types\[%{DATA:types}\], stats\[%{DATA:stats}\], search_type

\[%{DATA:search_type}\], total_shards\[%{NUMBER:total_shards}\], source\[%{DATA:source_query}\], 

extra_source\[%{DATA:extra_source}\],"]
    }

  
    mutate {
        convert => { "TOOKM" => "integer" }
    }

    
    if [TOOKM] > 15 {
        
    } else {
        drop { }
    }
}
output {
email {
  to => "zyx@gmail.com"
  body => "Here is the event line that occured: %{message}"
  address => "smtp.gmail.com"
  port => 587
  username => "xyz@gmail.com"
  password => "mypass"
}

 stdout { codec => rubydebug }
}

I am getting error like this

    ←[31mSomething happen while delivering an email {:exception=>#<Errno::ECONNREFUS
ED: Connection refused - Connection refused>, :level=>:error}←[0m
←[31mSomething happen while delivering an email {:exception=>#<Errno::ECONNREFUS
ED: Connection refused - Connection refused>, :level=>:error}←[0m
←[31mSomething happen while delivering an email {:exception=>#<Errno::ECONNREFUS
ED: Connection refused - Connection refused>, :level=>:error}←[0m

FYI - I am running the logstash in windows .

Do i need to install any SMTP server in my windows machine to send the mail or logstash alone will do that for me as per the above configuration?

Thanks

2

I would not say that you need a SMTP server running on localhost.

Have you checked that you can "telnet smtp.gmail.com 587" from the host running Logstash?

Moreover you should set up the domain parameter, which is localhost by default.

Additionally, have you enabled 2nd factor auth on gmail?

If none of this helped I would suggest using a sniffer (like wireshark) to triple-check where is logstash trying to connect on port 587

3

Thanks @Xavy

Now it is showing different error like this:

Something happen while delivering an email {:exception=>#<Net::SMTPAuthenticationError: 530 5.7.0 Must issue a STARTTLS command first. e187sm3531196wmf.31 - gsmtp
>, :level=>:error}
Something happen while delivering an email {:exception=>#<Net::SMTPAuthenticationError: 530 5.7.0 Must issue a STARTTLS command first. j124sm3264630wmg.13 - gsmtp
>, :level=>:error}
4

Hello @Yaswanth

As long as you're using port 587, which is (on standard) a TLS port, you must setup the following config:

use_tls
Value type is boolean
Default value is false
Enables TLS when communicating with the server

to true

5

Thanks @Xavy

It got worked but when i seen the output in the mail it is not the same as stdout in the screen

STDOUT:

         "message" => "[2017-04-25 04:40:05,240][TRACE][index.search.slowlog.query] [edata-0] [data-apr-2017][4] took[20.6ms], took_millis[20], types[details], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{\"query\":{\"bool\":{\"must\":[{\"terms\":{\"articleId\":[316249486]}}]}}}], extra_source[],",
        "@version" => "1",
      "@timestamp" => "2017-05-10T13:19:17.000Z",
            "path" => "/home/itadmin/logstash/logstash-2.4.1/slowlogs.txt",
            "host" => "kibana",
       "TIMESTAMP" => "2017-04-25 04:40:05,240",
           "LEVEL" => "TRACE",
           "QUERY" => "index.search.slowlog.query",
          "QUERY1" => "edata-0",
      "INDEX-NAME" => "data-apr-2017",
           "SHARD" => "4",
            "TOOK" => "20.6ms",
           "TOOKM" => 20,
           "types" => "details",
     "search_type" => "QUERY_THEN_FETCH",
    "total_shards" => "5",
    "source_query" => "{\"query\":{\"bool\":{\"must\":[{\"terms\":{\"articleId\":[316249486]}}]}}}"
}
{
         "message" => "[2017-04-25 05:47:02,335][TRACE][index.search.slowlog.query] [edata-0] [data-apr-2017][4] took[20.8ms], took_millis[20], types[details], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{\"query\":{\"bool\":{\"must\":[{\"terms\":{\"articleId\":[316252085]}}]}}}], extra_source[],",
        "@version" => "1",
      "@timestamp" => "2017-05-10T13:19:17.001Z",
            "path" => "/home/itadmin/logstash/logstash-2.4.1/slowlogs.txt",
            "host" => "kibana",
       "TIMESTAMP" => "2017-04-25 05:47:02,335",
           "LEVEL" => "TRACE",
           "QUERY" => "index.search.slowlog.query",
          "QUERY1" => "edata-0",
      "INDEX-NAME" => "data-apr-2017",
           "SHARD" => "4",
            "TOOK" => "20.8ms",
           "TOOKM" => 20,
           "types" => "details",
     "search_type" => "QUERY_THEN_FETCH",
    "total_shards" => "5",
    "source_query" => "{\"query\":{\"bool\":{\"must\":[{\"terms\":{\"articleId\":[316252085]}}]}}}"

But in my mail in am getting only the first message field

Here is the event line that occured: [2017-04-25 04:40:05,240][TRACE][index.search.slowlog.query] [esndata-0] [uatmmemediacontent-apr-2017][4] took[20.6ms], took_millis[20], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"assignedArticleList.assignedArticleId":[316249486]}}]}}}], extra_source[],

Is there any field limit for that?

Thanks

6

Hello again @Yaswanth:
I've not tested this particular output plugin, but I would expect that it sent an email per each event happening (per each json entry)

Therfore, I would expect that, for the stdout you showed us, you should be receiving two emails.

Could it be that the second had been moved to spam folder by Gmail antispam engine?

Regards

7

@Xavy What you had said is exactly correct ? One came to mail and other went to spam how can i send all the events in one mail?

THANKS

8

@Yaswanth:

I don't think that's something the plugin is intended to do, as per documentation:

Send email when an output is received. Alternatively, you may include or exclude the email output execution using conditionals.

Therefore you could avoid some emails being sent by using conditionals, but I don't think you may group several events on a per event number or a per time basis into a single email.

9

@Xavy Means your saying it cant be possible to send multiple messages in one mail?

Thanks

10

Unfortunately I suspect so. Keep in mind that output plugins are event based (each time an event occur, it generates an output with its content)
Perhaps a devel could confirm me or make me wrong, bu t I would say that you're asking for is not feasible for now

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.