HUGE Transaction Log

Elastic Stack Elasticsearch
1

I'm running ES 2.0.0. During a 48 hour test I have transaction logs that are almost 60Gb and growing. It seems to not be flushing. I could try a manual flush but don't understand why I would need to do that. My understanding is flushing happens automatically whenever the translog is too big (default 512Mb) or after 30 minutes.

We're running a single node. Our index is named "report" I see five folders in esearch/data/iTE/nodes/0/indices/report.

root@ite60-henry-perf02:/data/esearch/data/iTE/nodes/0/indices/report# ls
0  1  2  3  4  _state

root@ite60-henry-perf02:/data/esearch/data/iTE/nodes/0/indices/report# du -h .
1.4G    ./3/index
8.0K    ./3/_state
14G     ./3/translog
16G     ./3
1.4G    ./2/index
8.0K    ./2/_state
14G     ./2/translog
16G     ./2
1.4G    ./1/index
8.0K    ./1/_state
14G     ./1/translog
16G     ./1
1.5G    ./0/index
8.0K    ./0/_state
189M    ./0/translog
1.6G    ./0
8.0K    ./_state
1.5G    ./4/index
8.0K    ./4/_state
14G     ./4/translog
16G     ./4
63G     .

Inside the translog folders I see one huge log file. For example:

root@ite60-henry-perf02:/data/esearch/data/iTE/nodes/0/indices/report# ls -l 1/translog
total 14523980
-rwxrwxrwx 1 esearch esearch          20 Feb  1 05:30 translog-15.ckp
-rwxrwxrwx 1 esearch esearch          43 Nov 21 05:22 translog-15.tlog
-rwxr-xr-x 1 esearch esearch          20 Feb  6 04:03 translog-16.ckp
-rwxrwxrwx 1 esearch esearch          43 Feb  1 05:30 translog-16.tlog
-rwxr-xr-x 1 esearch esearch          20 Feb  6 05:30 translog-17.ckp
-rw-r--r-- 1 esearch esearch          43 Feb  6 04:03 translog-17.tlog
-rw-r--r-- 1 esearch esearch 14872489859 Feb  7 19:26 translog-18.tlog
-rwxrwxrwx 1 esearch esearch          20 Nov 15 18:39 translog-1.ckp
-rwxrwxrwx 1 esearch esearch          20 Nov 15 19:08 translog-2.ckp
-rwxrwxrwx 1 esearch esearch          20 Feb  7 19:26 translog.ckp

Over time those big files keeps growing.

Appreciate any pointers on what to look at or what to set.

Thanks.

2

I flushed (POST /report/_flush). Took 14s, cleared 56Gb and rotated translog (now have translog-19, which again continues to grow.

3

Looks like this is related to 15814 and fixed by 15830

What is best way to pick up that fix?

4

Upgrade. 2.0.0 is a long time ago in Elasticsearch terms. It is the first release of the last major version.

5

I understand. We're in a development cycle. Don't want to upgrade just on the hope stuff is fixed. This is the first time we noticed the problem. But it looks like upgrading to 2.2+ will fix, right?

6

Yes, 2.2+ would fix it. If you are upgrading you might want to get to a more modern version though.

7

Already tested, ES 5 breaks us. We use Jest and they seem to have issues, but might be that we're behind there too. For now a customer situation so ES 5 not an option. (Unlike the entire rest of the world it is still necessary for us to deliver software.)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.