16000 threads, then death

Elastic Stack Logstash
1

Hi !

I'm currently facing an issue with logstash making it impossible to use :
When it starts, logstash creates some threads (like 10 or 20) to initialize, and right after the pipelines are set up, it creates threads without stopping, up to 16k threads, which leads to the java.lang.OutOfMemoryError: unable to create new native thread error, and its death.

Here are my config files:

#logstash.yml
path.data: /var/lib/logstash
http.host: "0.0.0.0"
path.logs: /var/log/logstash
log.level: warn
xpack.monitoring.elasticsearch.url: ["https://elasticsearch"]
xpack.monitoring.elasticsearch.ssl.ca: "/etc/logstash/keys/cert.pem"
xpack.monitoring.enabled: true
pipeline.workers: 4
pipeline.batch.size: 100

pipelines :

input {
        http {
                id => "telegraf_in_http"
                        port => 9990
                        host => "0.0.0.0"
        }
}

filter {
        split {
                id => "telegraf_filter_split"
                        field => "metrics"
        }
        ruby {
                id => "telegraf_filter_ruby_move"
                        code => "event.get('[metrics][fields]').each {|k, v|
                        event.set('[' + event.get('[metrics][name]') + '][' + k + ']', v)
                        event.set('hostname', event.get('[metrics][tags][host]'))
        }"
        }
        ruby {
                id => "telegraf_filter_ruby_remove"
                        code => "
                        event.remove('[metrics][fields]')
                        event.remove('[metrics][tags][host]')"
        }
        mutate {
                id => "telegraf_filter_mutate"
                        remove_field => [ '[headers]',
                                     '[metrics][tags][org.label-schema.url]',
                                     '[metrics][tags][org.label-schema.description]',
                                     '[metrics][tags][org.label-schema.vendor]',
                                     '[metrics][tags][org.label-schema.docker.schema-version]',
                                     '[procstat][cpu_time_idle]',
                                        '[cpu][usage_guest]',
                                        '[cpu][usage_guest_nice]',
                                        '[cpu][usage_steal]'
                                             ]
        }
        ruby {
                code => "
                        event.to_hash.keys.each { |k|
                                if k.start_with?('org')
                                        event.remove(k)
                                                end
                        }
                "
        }

if [metrics][name]
{
        if [metrics][name] == "docker_container_blkio" or [metrics][name] == "sqlserver_waitstats" or [metrics][name] == "sqlserver_memory_clerks" {
                drop { }
        }
}
}
output {
        http {
                id => "telegraf_out_http_internal"
                        http_method => "put"
                        url => "http://127.0.0.1:9991"
        }
        http {
                id => "telegraf_out_http_external"
                        http_method => "put"
                        url => "http://127.0.0.1:9992"
        }
}

2nd pipeline

input {
        http {
                id => "internal_in_http"
                port => 9991
                host => "127.0.0.1"

        }
}
filter {
        mutate {
                id => "internal_filter_mutate"
                remove_field => [ "[headers]" ]
        }
}
output {
        elasticsearch {
                id => "internal_out_es"
                cacert => "/cert.pem"
                hosts => ["https://ES:443"]
                index => "logstash-rbx-%{+YYYY.MM.dd}-%{[metrics][name]}"
        }
}

3rd :

input {
        http {
                id => "mirror_in_http"
                port => 9992
                host => "127.0.0.1"
        }
}
filter {
        mutate {
                id => "mirror_filter_mutate"
                remove_field => ["headers"]
        }
}
output {
        http {
                id => "mirror_out_http"
                cacert => "/cert.pem"
                        http_method => "post"
                        url => "https://<OTHER ES>:8080/"
        }
}

It seems that if I remove to the last pipelines and keep only the main first one, everything runs smoothly. However, I need those 2 pipelines because if I specify 2 outputs in one pipeline, and one of those outputs is down, the other won't receive documents until the first one is back online.
Having such configuration ensure me (if logstash is doing fine) that the documents will be outputed directly to their destination if it's up.

Am I missing something ?

Thanks in advance,

Cyril

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.