Logstash works fine when ran localy but when reading data from a NAS and pushing to staging Elasticsearch it errors out


(Shah) #1

I am getting following error while running logstash which works fine on my local localstash -ElasticSearch instance but fails while reading data from a NAS and pushing to Production instance of elasticsearch.Following is my logstash config.

I even tried updating Heaps size as LS_HEAP_SIZE=2048m in the config. Any help is appreciated.

Error log

{:timestamp=>"2016-02-09T08:23:04.350000-0800", :message=>"Flushing buffer at interval", :instance=>"#  <LogStash::Outputs::ElasticSearch::Buffer:0x7006afbb @stopping=#<Concurrent::AtomicBoolean:0x46fb5772>, @last_flush=2016-02-09 08:23:03 -0800, @flush_thread=#<Thread:0x1f2a21c6 run>, @max_size=500, 
@operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x70f5e26>, @submit_proc=#<Proc:0x357dfa7c@/softwares/logstash/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/common.rb:55>, @flush_interval=1, @logger=#
<Cabin::Channel:0x56e56093 @subscriber_lock=#<Mutex:0x6d277f76>, @data={}, @metrics=#<Cabin::Metrics:0x3c2153f6 @channel=#<Cabin::Channel:0x56e56093 ...>, @metrics={}, @metrics_lock=#
<Mutex:0x300f5a51>>, @subscribers={12840=>#<Cabin::Outputs::IO:0x21e9753f @lock=#
<Mutex:0x66108c7d>, @io=#<IO:fd 1>>}, @level=:debug>, @buffer=[], @operations_mutex=#
<Mutex:0x61c7fe53>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}

Config FIle

input {
	file {
		path => ["/net/smccscscss/somepath/syslogs/*.log"]		
		start_position => "beginning"
		type => "syslogs"
		tags => "syslogs"
	}
}
filter {
		if "syslogs" in [tags]{		
			multiline {	
				patterns_dir => "./patterns"			
				pattern => "^\[level="
				what => "previous"
				add_tag => "multilinesyslog"
				negate => true
				}
			grok {
				patterns_dir => "./patterns"
				match => ["path","/net/smccscscss/%{DATA:sjptjobid}/syslogs/%{DATA:sjptlogtype}.log" ]
				add_tag => "pathsyslogs"
				}
			grok {
				patterns_dir => "./patterns"
				match => {"message" => "%{SJPTSYSLOG}"} 
				add_tag => "msgsyslogs"}
		}
		
		if "fastdebuglogs" in [tags]{
			grok {
			match => ["path","/net/smccscscss//%{DATA:sjptjobid}/logs/%{DATA:sjptlogtype}.log"]
			add_tag => "processedjobidsys"
			}}
	}	
output {
	elasticsearch{
		index => "sjpt-%{type}"
		manage_template => false
		hosts => "somemachine:9200"
		}
	stdout{ }
}

(system) #2