Container logs not parsed fully by fluentd

[Ayush_Mathur]

fluentd version: 0.12.47
elastic version: 5.4.2

I have recently configured logging driver to rotate the container log files once it reaches 2G size and keep last 10 log files.

When I push bulk messages to the application, I can see multiple log files being generated (around 9 rotated and 1 current) in /var/lib/docker/container/{container-id} path.

However, when I check for the logs in Kibana, it looks like the logs are missing when log file is rotated. In addition to this, the logs are never parsed till the end of rotated log files no matter if fluentd is restarted again or not.

My current driver configuration is:

OPTIONS=' --selinux-enabled --selinux-enabled --log-driver=json-file --log-opt max-size=2G --log-opt max-file=10'

The files present in container-id directory:

-r--------. 1 root root  385M May  4 15:14 test-json.log
-r--------. 1 root root 1908M May  4 15:14 test-json.log.1
-r--------. 1 root root 1908M May  4 15:12 test-json.log.2
-r--------. 1 root root 1908M May  4 15:11 test-json.log.3
-r--------. 1 root root 1908M May  4 15:09 test-json.log.4
-r--------. 1 root root 1908M May  4 15:07 test-json.log.5
-r--------. 1 root root 1908M May  4 15:05 test-json.log.6
-r--------. 1 root root 1908M May  4 15:04 test-json.log.7
-r--------. 1 root root 1908M May  4 15:02 test-json.log.8
-r--------. 1 root root 1908M May  4 15:00 test-json.log.9

Where the files are rotated as :-

1. Logs are pushed to test-json.log
2. on hitting 2G size, file is renamed to test-json.log.1 and test-json.log is created
3. on hitting 2G again, test-json.log.1 is renamed to test-json.log2, test-json.log is renamed to test-json.log.1 and new test-json.log is created, and so on.
4. In step 2 and 3, the logs pushed towards the end are not parsed by fluentd and are missing in Kibana.
5. In short, the fluentd only parse the current file: test-json.log and don't even care about the rotated files.

The fluentd throttle-config is default from master without any modifications: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/generate_throttle_configs.rb

Can please someone guide me on how to fix this log rotation to parse all the logs from rotated files before actually picking up the new file ?