Connecting logstash to swift-ceph


(Jin Nguyen) #1

Hi all,

I find out a custom plugin here https://github.com/jedipunkz/logstash-output-plugin-swift to connect to swift Openstack. But what I need is connecting to swift on Ceph storage using ruby API. This is the code I modified:

encoding: utf-8

require "logstash/outputs/base"
require "logstash/namespace"
class LogStash::Outputs::Swift < LogStash::Outputs::Base
config_name "swift"
milestone 2
config :auth_url, :validate => :string, :require => true
config :auth_user, :validate => :string, :require => true
#config :auth_tenant, :validate => :string, :require => true
config :auth_api_key, :validate => :string, :require => true
config :swift_account, :validate => :string, :require => false
config :swift_container, :validate => :string, :require => true
config :swift_object_key_format, :validate => :string, :require => false
public
def register
require 'fog'
require 'zlib'
require 'time'
require 'tempfile'
#require 'open3'
@conn = CloudFiles::Connection.new :auth_url => @auth_url,
:username => @auth_user,
:api_key => @auth_api_key
@conn.change_account @swift_account if @swift_account
end # register
public
def receive(event)
#require 'fog'
require 'zlib'
require 'time'
require 'tempfile'
require 'open3'
return unless output?(event)
i = 0
# @swift_object_key_format = "%{path}%{time_slice}_%{index}.%{file_extension}"
# swift_path = @swift_object_key_format.gsub(%r(%{[^}]+})) { |expr|
# values_for_swift_object_key[expr[2...expr.size-1]]
swift_path = "testobject"
@mime_type = 'application/x-gzip'
tmp = Tempfile.new("swift-")
begin
w = Zlib::GzipWriter.new(tmp)
chunk.write_to(w)
w.close
File.open(tmp.path) do |file|
@conn.put_object(@swift_container, swift_path, file, {:content_type => @mime_type})
end # File.open
rescue => e
@logger.warn("Failed to send event to Swift", :evnet => event, :exception => e,
:backtrace => e.backtrace)
ensure
w.close rescue nil
end # begin
end # def write
end # class LogStash::Outputs::Mongodb

But when I run the conf logstash, it occurs error like this:
/opt/logstash/bin/logstash -f ../test_swift.conf
swift plugin is using the 'milestone' method to declare the version of the plugin this method is deprecated in favor of declaring the version inside the gemspec. {:level=>:warn}
Default settings used: Filter workers: 2
The error reported is:
Unable to connect to http://s3.mydomain.net/auth

This is my test_swift.conf file:

input {
stdin {
codec => "json"
}
}

output {
swift {
auth_url => "http://s3.mydomain.net/auth/v1.0"
auth_user => "vinhn4:swift"
auth_api_key => "KsG4/Ye/5tP0jgTfeD4n5KHlr5BQJEfc4hTuj0kd"
swift_container => "test"
}
}


(Mark Walkom) #2

You may want to raise an issue against the original plugin creator's repo.


(Jin Nguyen) #3

I've connected to swift with above plugin with some modifies in code. It's connect but "Failed to send event to Swift"
This is error "Failed to send event to Swift {:evnet=>#<LogStash::Event:0x692a56d6 @metadata_accessors=#<LogStash::Util::Accessors:0x76cbe9c4 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"", "@version"=>"1", "@timestamp"=>"2015-11-19T04:19:46.519Z", "host"=>"0.0.0.0"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x35e1a2e7 @store={"message"=>"", "@version"=>"1", "@timestamp"=>"2015-11-19T04:19:46.519Z", "host"=>"0.0.0.0"}, @lut={"host"=>[{"message"=>"", "@version"=>"1", "@timestamp"=>"2015-11-19T04:19:46.519Z", "host"=>"0.0.0.0"}, "host"]}>>, :exception=>#<NameError: undefined local variable or method chunk' for #<LogStash::Outputs::Swift:0x2bb684c3>>, :backtrace=>["/opt/logstash/vendor/local_gems/bc2a61ae/logstash-output-swift-2.0.0/lib/logstash/outputs/swift.rb:52:inreceive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/outputs/base.rb:80:in handle'", "(eval):22:inoutput_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:252:in outputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:169:instart_outputs'"], :level=>:warn}"


(system) #4