High CPU usage docker container running Logstash 2.0

danwald · November 20, 2015, 6:34pm

I am running the official logstash2.0 (using marathon via mesos) docker and the process is using a lot of CPU. Heres what the top on the process and its threads (below).
The process eventually dies failing to connect to elastic search saying it's not available which isn't the case or it runs out of memory or it gets a GC error even though I have
"JAVA_OPTS":"-Xmx12000m -XX:-UseGCOverheadLimit" set.
If anyone has any ideas, that would be great?

The logstash pipeline is
Input: s3
filters csv, date, mutate
output: elasticsearch

Part of my input pipe moves the files after processing, but this fails to happen if the process dies. I would think it would move the processed records because I can see them coming into elastic search?
Cheers,

danny

Threads: 47 total, 0 running, 47 sleeping, 0 stopped, 0 zombie
%Cpu(s): 70.4 us, 0.4 sy, 0.0 ni, 29.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65965436 total, 19961468 used, 46003968 free, 672448 buffers
KiB Swap: 0 total, 0 used, 0 free. 16578204 cached Mem
```
 PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
```
14510 root 20 0 5193412 836396 15640 S 87.2 1.3 10:33.31 java
14517 root 20 0 5193412 836396 15640 S 86.8 1.3 10:33.20 java
14509 root 20 0 5193412 836396 15640 S 86.2 1.3 10:33.29 java
14518 root 20 0 5193412 836396 15640 S 86.2 1.3 10:33.43 java
14511 root 20 0 5193412 836396 15640 S 85.8 1.3 10:33.42 java
14512 root 20 0 5193412 836396 15640 S 85.8 1.3 10:32.82 java
14513 root 20 0 5193412 836396 15640 S 85.8 1.3 10:33.37 java
14514 root 20 0 5193412 836396 15640 S 85.8 1.3 10:32.71 java
14519 root 20 0 5193412 836396 15640 S 85.8 1.3 10:33.70 java
14521 root 20 0 5193412 836396 15640 S 85.8 1.3 10:32.75 java
14515 root 20 0 5193412 836396 15640 S 85.5 1.3 10:32.70 java
14516 root 20 0 5193412 836396 15640 S 85.5 1.3 10:33.47 java
14520 root 20 0 5193412 836396 15640 S 85.5 1.3 10:33.25 java
14617 root 20 0 5193412 836396 15640 S 4.3 1.3 6:48.91 |worker
14522 root 20 0 5193412 836396 15640 S 2.3 1.3 0:23.40 java
14624 root 20 0 5193412 836396 15640 S 2.0 1.3 4:03.51 >output
14616 root 20 0 5193412 836396 15640 S 1.0 1.3 1:34.17 <s3
14539 root 20 0 5193412 836396 15640 S 0.3 1.3 0:00.53 java
14621 root 20 0 5193412 836396 15640 S 0.3 1.3 0:04.96 LogStash::Runne
14494 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
14508 root 20 0 5193412 836396 15640 S 0.0 1.3 0:03.40 LogStash::Runne
14523 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.22 java
14524 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.22 java
14525 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
14526 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.95 java
14527 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.93 java
14528 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.50 java
14529 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.82 java
14530 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.79 java
14531 root 20 0 5193412 836396 15640 S 0.0 1.3 0:06.11 java
14532 root 20 0 5193412 836396 15640 S 0.0 1.3 0:05.10 java
14533 root 20 0 5193412 836396 15640 S 0.0 1.3 0:06.67 java
14534 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.55 java
14535 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.47 java
14536 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.53 java
14537 root 20 0 5193412 836396 15640 S 0.0 1.3 0:01.57 java
14538 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
14604 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 java
14605 root 20 0 5193412 836396 15640 S 0.0 1.3 0:02.80 LogStash::Runne
14607 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
14610 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
14611 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
14618 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.15 LogStash::Runne
14619 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 <s3
14620 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 <s3
14622 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne
14623 root 20 0 5193412 836396 15640 S 0.0 1.3 0:00.00 LogStash::Runne

warkolm · November 21, 2015, 5:56am

This is an official image provided by Docker, not us

However you should check the LS logs, there may be a problem causing it to not start correctly, and if something like upstart is restarting the process, you can see this sort of behaviour.

danwald · November 21, 2015, 10:26am

Warkolm,
it's actually a job started by marathon, which is a mesos framework that loads the job on a slave. I have provisioned it for 13GB and 12core limits but it still happens.
However there is light and the end of this tunnel.
I noticed that stdout was also part of the output plugin. I removed that and it has been running fine.
Thanks for the input.
I guess this was user error.
Cheers,

danny