EXCEPTION : java.lang.OutOfMemoryError: GC overhead limit exceeded

Elastic Stack Logstash
1

Hi,
I'm indexing data from sql server to Elasticsearch using JDBC plugin. While running a particular query, I'm facing this issue.

Pasted image650×713 52.4 KB

Have tried increasing ES_MIN_MEM to 10g, ES_MAX_MEM to 10g, ES_HEAP_SIZE to 10g. But still facing the same issue. Can anyone please help me out.

Regards,
Sanjay Reddy.

2

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them :slight_smile:

What version are you on? What's your config look like?

3

@warkolm Sorry for posting picture.

I'm using logstash version 2.3.2. and my config looks like this:

input {
  jdbc {
jdbc_driver_library => "Q:/softwares/sqljdbc-1.2.0.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://10.151.2.188;databaseName=central"
jdbc_user => "sanjay"
jdbc_password => "sanjay@123"

statement => "select * from dbo.sanjay"

 jdbc_paging_enabled => "true"
 jdbc_page_size => "5000000"
 }
}
filter{
grok {
    patterns_dir => "Q:/softwares/ElasticSearch/logstash-1.3.3-flatjar/patterns"
    match => ["AdmitDate", "%{YEAR:al_year}-%{MONTHNUM:al_month}-%{MONTHDAY:al_monthday} %{TIME:al_time}" ]
    add_field => [ "LogTime", "%{al_year}-%{al_month}-%{al_monthday} %{al_time}" ]
}
date 
{
    match => [ "LogTime", "YYYY-MM-dd HH:mm:ss.SSS"] 
}
mutate 
{
	convert => [ "Age", "integer" ]
	convert => [ "PatientType", "integer" ]
	convert => [ "AmountPaid", "float" ]
}	
}
output {
  stdout {
   # codec => rubydebug
 codec => "json"
}
 elasticsearch { 
   hosts => "localhost:9200"
   index => "central"
   document_type => "central"

template => "Q:/softwares/ElasticSearch/logstash-2.3.2/bin/elasticsearch-template-central.json"
template_name => "central"
  }
}

And I'm getting the following error:

io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 4
Pipeline main started
Exception in thread "Ruby-0-Thread-11: Q:/softwares/ElasticSearch/logstash-2.3.2
/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.2-java/lib/logstash/pipeline.rb:
444" Exception in thread "Ruby-0-Thread-6: Q:/softwares/ElasticSearch/logstash-2
.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/l
ogstash/outputs/elasticsearch/buffer.rb:78"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler i
n thread "Ruby-0-Thread-6: Q:/softwares/ElasticSearch/logstash-2.3.2/vendor/bund
le/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/
elasticsearch/buffer.rb:78"
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Ruby-0-Thread-3: Q:/softwares/ElasticSearch/logstash-2.3.2/
vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:22" java.lang.OutOfMem
oryError: GC overhead limit exceeded
Exception in thread "[main]<jdbc" java.lang.OutOfMemoryError: GC overhead limit
exceeded

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler i
n thread "[main]>worker0"
Exception in thread "[main]>worker0" Exception in thread "[main]>worker3" Except
ion in thread "[main]>worker2" java.lang.OutOfMemoryError: GC overhead limit exc
eeded
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler i
n thread "[main]>worker2"

java.lang.ArrayIndexOutOfBoundsException: -1
        at org.jruby.runtime.ThreadContext.popRubyClass(ThreadContext.java:702)
        at org.jruby.runtime.ThreadContext.postYield(ThreadContext.java:1266)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler i
n thread "[main]>worker3"
Exception in thread "[main]>worker1" java.lang.OutOfMemoryError: GC overhead lim
it exceeded

Not able to index anything. Please help me out :frowning:

4

I'd reduce that and see how it goes.

5

Thanks @warkolm for the suggestion.
When I reduce the size to 50000, it is working.
But I have more than 50,00,000 records.
Can you please suggest any other way... :frowning:

6

Any other way what?

7

I mean Can I index all 50,000,000 records at a time???
If yes, please let me know how to do that :slight_smile:

8

No, that is not really a sane idea.
What is wrong with streaming them?

9

Ok I'll stream it.
In order to stream it, I'm thinking to stream it for every 5,00,000 records in 10 logstash instances at a time inorder to index 5,000,000 records.
will it reduce the indexing speed???