Issue with Ruby and logstash-input-imap

Afer configuring the impa plugin on logstash and running the following command:

bin/logstash -f email_log.conf --path.data /var/lib/logstash/imap --config.reload.automatic

I get the following error:

[ERROR] 2018-06-06 13:33:11.795 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 13, column 20 (byte 214) after output {\n stdout { codec ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

Checking at line 13 on the script task.rb I can see the following :slight_smile:

class Task

... the full code output is:

require "thread"
require "stud/interval"

module Stud

A Task spawns a thread to execute the given block. execution completion and result retrieval is

done using the Task#wait method. A Task is run once and the thread exists upon block completion.

A task and its underlying thread are not reusable.

Task does not provide a mean to force-interrupt a running task, it only provides the #stop!

method to signal the task for a stop request. The task or code block can use the #stop? method

to check for a stop request. Note that the #stop! and #stop? methods are thread safe.

class Task
# provide access to the underlying thread if ever needed.
attr_reader :thread

def initialize(*args, &block)
  # A queue to receive the result of the block
  # TODO(sissel): Don't use a queue, just store it in an instance variable.
  @queue = Queue.new

  @thread = Thread.new(@queue, *args) do |queue, *args|
    begin
      result = block.call(*args)
      queue << [:return, result]
    rescue => e
      queue << [:exception, e]
    end
  end # thread
end # def initialize

# wait waits for the task thread to complete and return the block return value
# if the block raises an exception, this exception is propagated in this
# wait method.
# @return [Object, Exception] block return value
def wait
  @thread.join
  reason, result = @queue.pop

  if reason == :exception
    #raise StandardError.new(result)
    raise result
  else
    return result
  end
end # def wait

# stop! requests the task to stop. the Thread#wakeup method is also
# called so that a sleeping task is waked up and has a chance to verify
# the stop request using the #stop? method. also see Stud.stop!
def stop!
  Stud.stop!(@thread)
end

# stop? returns true if this task stop! has been called
# See Stud.stop?
# @return [Boolean] true if the stop! has been called
def stop?
  Stud.stop?(@thread)
end
alias_method :interrupted?, :stop?

end # class Task
end # module Stud

Is anybody able to assist with this error please, as the error notification seems to be pointing to the wrong place, else there may be a bug with this module on Elasticsearch 6.2.0

It is noticing the error at line 13 of email_log.conf. Can you post that file?

Thank you for the pointer, I noticed that there was an extraneous > in that line which I have since taken out and am restarting all the processes

Alexa

I have edited and restarted the processes however this error is still showing. the email_log.conf file is:

#email_log.conf
input {
imap {
host => "..net"
password => "
***"
user => "test-imap@
.net"
port => 143
check_interval => 10
folder => "Inbox"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => "emails"
document_type => "email"
hosts => "localhost:9200"
}
}

Exactly what error message do you get with that file?

It has changed, now the error (I have everything set to perform stdout to console) is:

[ERROR] 2018-06-07 11:51:03.882 [[main]<imap] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::IMAP host=>"bandal.owta.net", password=>, user=>"test-imap@owta.net", port=>143, check_interval=>10, folder=>"Inbox", id=>"c5687598a52bbd85958d432fd18433530c150d94801456b1fbb544308bbb558a", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_c35559f2-eaa6-4958-9316-495b3cc47dbd", enable_metric=>true, charset=>"UTF-8">, secure=>true, verify_cert=>true, fetch_count=>50, lowercase_headers=>true, delete=>false, expunge=>false, strip_attachments=>false, content_type=>"text/plain">
Error: Unrecognized SSL message, plaintext connection?
Exception: OpenSSL::SSL::SSLError
Stack: org/jruby/ext/openssl/SSLSocket.java:218:in connect' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/imap.rb:1492:instart_tls_session'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/imap.rb:1085:in initialize' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:61:inconnect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:76:in check_mail' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:69:inblock in run'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in interval' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:68:inrun'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in inputworker' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:inblock in start_input'

The complaint is about ssl. The imap mailbox is starttls and I am not employing (currently) any form of ssl on the stack

SSL is on by default. Set 'secure => false' for the imap plugin.

Thank you so much for the pointers. I had noted that it was doing this by default and have now set it to false, along with trying the config with verify_cert set and not set:

#email_log.conf
input {
imap {
host => "bandal.owta.net"
password => "GrabFiles"
user => "test-imap@owta.net"
port => 143
secure => false
verify_cert => true
check_interval => 300
folder => "Inbox"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => "emails"
document_type => "email"
hosts => "localhost:9200"
}

It is however still erroring on trying to connect to the imap mailbox:

[ERROR] 2018-06-07 12:41:30.925 [[main]<imap] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::IMAP host=>"bandal.owta.net", password=>, user=>"test-imap@owta.net", port=>143, secure=>false, verify_cert=>true, check_interval=>300, folder=>"Inbox", id=>"51191afd186cc52fe992d9de46d6ee3524721585ad3ccb5b742a60728a54bda0", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_feafede3-8759-414b-8e4e-d2408c204c06", enable_metric=>true, charset=>"UTF-8">, fetch_count=>50, lowercase_headers=>true, delete=>false, expunge=>false, strip_attachments=>false, content_type=>"text/plain">
Error: cleartext logins disabled
Exception: Net::IMAP::NoResponseError
Stack: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/imap.rb:1196:in get_tagged_response' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/imap.rb:1250:in block in send_command'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:214:in mon_synchronize' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/imap.rb:1232:in send_command'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/imap.rb:436:in login' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:62:in connect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:76:in check_mail' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:69:in block in run'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in interval' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:68:in run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in inputworker' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in block in start_input'

Alexa

I think that is telling you that you have to use SSL. The IMAP server is configured not to let you send credentials across the network unencrypted. So you need to switch secure to be true, and change the port to the IMAP SSL port. Or just not set the port if your IMAP server listens on 993.

I am checking just this with the administrator of the mail server as we speak

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.