Imap plugin crashes logstash when there are attachments

Elastic Stack Logstash
1

My issue is the same as was reported about a year ago here:

https://logstash.jira.com/browse/LOGSTASH-2080

logstash silently (other than the two below log entries) dies when it tries to read an e-mail with an attachment.

{:timestamp=>"2015-05-27T11:58:52.925000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::IMAP type=>"mail", host=>"redacted", user=>"redacted", folder=>"INBOX", content_type=>"text/plain">\n Error: Can not decode an entire message, try calling #decoded on the various fields and body or parts if it is a multipart message.", :level=>:error}
{:timestamp=>"2015-05-27T11:58:55.792000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::IMAP type=>"mail", host=>"redacted", user=>"redacted", folder=>"INBOX", content_type=>"text/plain">\n Error: Can not decode an entire message, try calling #decoded on the various fields and body or parts if it is a multipart message.", :level=>:error}

Anyone have a work-around?

2

And I'm actually not sure it's a problem with an attachment - might just be a "large" message. It's difficult to tell which mail it's failing on.

3

I also see errors like the following that cause logstash to crash:

{:timestamp=>"2015-06-05T09:10:48.800000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::IMAP type=>"mail", host=>"redacted", user=>"redacted", folder=>"mail/dba-sqljobs", content_type=>"text/plain">\n Error: Bad file descriptor - Bad file descriptor", :level=>:error}
{:timestamp=>"2015-06-05T10:59:06.962000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::IMAP type=>"mail", host=>"redacted", user=>"redacted", folder=>"mail/dba-sqljobs", content_type=>"text/plain">\n Error: Bad file descriptor - Bad file descriptor", :level=>:error}

And in both cases, it takes two to cause logstash to die but it seems they can be an arbitrary amount of time between them.

4

Hi,

I have this problem with fresh installation.
Do you know how I can solve this problem ?

Thanks.

5

I didn't actually find a solution but my workaround is:

  1. Try and avoid the e-mails with attachments that cause the crash - I moved the mails that I wanted to ingest to a separate imap folder while avoiding the mails that I know cause problems. I used imapfilter - https://github.com/lefcha/imapfilter

  2. I wrote a little cron script that detects when logstash crashes and restarts. Here's the script:

#!/bin/sh
ls_status=service logstash status
if [ "x$ls_status" != "xlogstash is running" ]
then
service logstash start;
echo "date - logstash not running - restarted" >> /tmp/logstash.log;
fi

Not pretty and don't think I'd use this for "production" but good enough for testing.