High CPU usage docker container running Logstash 2.0

Elastic Stack Logstash
1

I am running the official logstash2.0 (using marathon via mesos) docker and the process is using a lot of CPU. Heres what the top on the process and its threads (below).
The process eventually dies failing to connect to elastic search saying it's not available which isn't the case or it runs out of memory or it gets a GC error even though I have
"JAVA_OPTS":"-Xmx12000m -XX:-UseGCOverheadLimit" set.
If anyone has any ideas, that would be great?

The logstash pipeline is
Input: s3
filters csv, date, mutate
output: elasticsearch

Part of my input pipe moves the files after processing, but this fails to happen if the process dies. I would think it would move the processed records because I can see them coming into elastic search?
Cheers,

  • danny

    Threads: 47 total, 0 running, 47 sleeping, 0 stopped, 0 zombie
    %Cpu(s): 70.4 us, 0.4 sy, 0.0 ni, 29.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
    KiB Mem: 65965436 total, 19961468 used, 46003968 free, 672448 buffers
    KiB Swap: 0 total, 0 used, 0 free. 16578204 cached Mem

     PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND

    14510 root 20 0 5193412 836396 15640 S 87.2 1.3 10:33.31 java
    14517 root 20 0 5193412 836396 15640 S 86.8 1.3 10:33.20 java
    14509 root 20 0 5193412 836396 15640 S 86.2 1.3 10:33.29 java
    14518 root 20 0 5193412 836396 15640 S 86.2 1.3 10:33.43 java
    14511 root 20 0 5193412 836396 15640 S 85.8 1.3 10:33.42 java
    14512 root 20 0 5193412 836396 15640 S 85.8 1.3 10:32.82 java
    14513 root 20 0 5193412 836396 15640 S 85.8 1.3 10:33.37 java
    14514 root 20 0 5193412 836396 15640 S 85.8 1.3 10:32.71 java
    14519 root 20 0 5193412 836396 15640 S 85.8 1.3 10:33.70 java
    14521 root 20 0 5193412 836396 15640 S 85.8 1.3 10:32.75 java
    14515 root 20 0 5193412 836396 15640 S 85.5 1.3 10:32.70 java
    14516 root 20 0 5193412 836396 15640 S 85.5 1.3 10:33.47 java
    14520 root 20 0 5193412 836396 15640 S 85.5 1.3 10:33.25 java
    14617 root 20 0 5193412 836396 15640 S 4.3 1.3 6:48.91 |worker
    14522 root 20 0 5193412 836396 15640 S 2.3 1.3 0:23.40 java
    14624 root 20 0 5193412 836396 15640 S 2.0 1.3 4:03.51 >output
    14616 root 20 0 5193412 836396 15640 S 1.0 1.3 1:34.17 <s3
    14539 root 20 0 5193412 836396 15640 S 0.3 1.3 0:00.53 java
    14621 root 20 0 5193412 836396 15640 S 0.3 1.3 0:04.96 LogStash::Runne
    14494 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
    14508 root 20 0 5193412 836396 15640 S 0.0 1.3 0:03.40 LogStash::Runne
    14523 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.22 java
    14524 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.22 java
    14525 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
    14526 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.95 java
    14527 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.93 java
    14528 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.50 java
    14529 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.82 java
    14530 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.79 java
    14531 root 20 0 5193412 836396 15640 S 0.0 1.3 0:06.11 java
    14532 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.10 java
    14533 root 20 0 5193412 836396 15640 S 0.0 1.3 0:06.67 java
    14534 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.55 java
    14535 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.47 java
    14536 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.53 java
    14537 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.57 java
    14538 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
    14604 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
    14605 root 20 0 5193412 836396 15640 S 0.0 1.3 0:02.80 LogStash::Runne
    14607 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
    14610 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
    14611 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
    14618 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.15 LogStash::Runne
    14619 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 <s3
    14620 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 <s3
    14622 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
    14623 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne

2

This is an official image provided by Docker, not us :smile:

However you should check the LS logs, there may be a problem causing it to not start correctly, and if something like upstart is restarting the process, you can see this sort of behaviour.

3

Warkolm,
it's actually a job started by marathon, which is a mesos framework that loads the job on a slave. I have provisioned it for 13GB and 12core limits but it still happens.
However there is light and the end of this tunnel.
I noticed that stdout was also part of the output plugin. I removed that and it has been running fine.
Thanks for the input.
I guess this was user error.
Cheers,

  • danny