Logstash calls flush method multiple times per filter plugin

I am writing some logstash filter plugin and found one weird behaviour. It seems the LS somehow calls flush method multiple times even if I use 1 worker and 1 pipeline called "def". I did some tests and here are parts of the code and the result:

      def register
        @esde_uuid = SecureRandom.uuid

        @last_flush = Concurrent::AtomicFixnum.new(0)
        th = Thread.current.object_id
        @logger.warn("MASTER THREAD ID: #{th}")
      end

      def flush(options = {})
        return if @refresh_interval == 0

        # Add 5 seconds to @last_flush counter
        # since this method is called every 5 seconds.
        @last_flush.update { |v| v + 5 }

        return unless @last_flush.value >= @refresh_interval

        euuid = @esde_uuid
        uuid = SecureRandom.uuid
        lf = @last_flush.value
        th = Thread.current.object_id
        @logger.warn("TH #{th} FLUSH! #{euuid} - #{uuid} : #{lf}")

        # Reset counter since metrics were flushed
        @last_flush.value = 0

        return
      end

      private

      def periodic_flush
        true
      end

@refresh_interval is set to 10 in plugin config but each 5 seconds I got following multiple lines instead of ONE per 10 second.

[2021-12-16T01:12:14,835][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3adde33a-66fc-40d6-a9a1-bb473c7756c4 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - cd9e8f67-e86c-448a-8259-36ce15fb761c : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 9bd1b5c3-c843-41f5-8293-5b53de985685 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 387f652f-92dc-481d-9e9f-91ab8e5623fa : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 8278ce82-41d5-44b9-b3e7-3bdd6677e4a4 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 137a65f2-b19d-4fbe-a75e-f20c15eb4b8f : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 5c5b28b5-9a94-4b98-ad9f-041999838620 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - f09389ec-d32c-41e6-be3d-0f3c7f061e43 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1010a008-20c2-4f8f-b534-cdcffe1494aa : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 909687e9-ae9c-40fe-b703-28450fbd074a : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0a03bcdf-33c8-4d5e-a607-2b4d18d1e97f : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - c6408aa0-4951-45d6-8c92-3d10453f5440 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - d37a3519-84ac-49d5-add9-667593d2af85 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 8a44d9f4-3eac-421d-944f-19c6f2701995 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3f38d6af-98b1-4193-b211-50d13e22976c : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - a92f4d9a-a6e5-4f36-93eb-4da67a8af958 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - b38b4103-c52e-4f8e-97a6-c9d8be837a15 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3e6c97cc-08aa-4677-a1c3-856477f4ceb4 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1a40f369-8b3f-4a07-b8b0-c29a75422a77 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - fde0b854-e1cf-4d60-8566-29a463fde526 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 66fd0d74-155e-4155-a242-cb488bb2f9df : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0177a041-c92b-46b4-9db9-ea974092d703 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - c9610e99-7f85-4b69-b3cd-902ba5d3049e : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 720edf27-e536-45e0-bde8-02c67d9ec4e6 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 9489ccd5-e3d5-40b3-92c6-f2c85f68e533 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0c2b1bcc-b34c-4892-bff7-cb4af771d90d : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 14e579dd-7ce7-4d33-a9f2-aca3b0da08ee : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 7079efc6-154c-4753-acdf-7caaeababff3 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 62e34c69-2569-463e-8918-933b05c308e2 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 6138d68f-ac63-4f27-a37a-7e126b2eb440 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 55ac10d1-30e3-45c6-9f47-14bdd4eec7f9 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1e54199f-933b-4a98-8d33-9fbf99fb0957 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1cd0abf8-ca2e-4aae-b869-119a86c8c97e : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - e7e90036-209c-4be3-9668-20c003a14d11 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - e10b0ddc-b3e3-4c1f-959c-f120ce7876a2 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 37a8ebcd-cd3e-49e6-bbef-cb3d2565c39c : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - cb99c39b-eee5-41b3-b655-559194dbd90b : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - bafff4cd-c6fc-45a2-9511-251080eec860 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 17c3b38f-1204-4022-a6b6-6c1d03375a1b : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - c45edf4e-9633-406f-8fbf-6fd2006e16c1 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 7b69b06d-f8a5-4950-83b5-632694ff90b3 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0ce22cda-bb02-4823-996a-5347fabeac8f : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 8037fe5d-0c5f-40b8-a631-5fe2a8e94942 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3190dba7-dabc-490e-8dad-723d91684b14 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3bb723b0-1690-46e3-b763-b1b34fdedfce : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - eb1416a1-9b75-4fc3-ae24-47e21905a823 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - a51c1298-731a-4cad-9338-abea4a2317b1 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3b5ac7d3-263a-4ba0-809c-d74fc776ae97 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1f94d7fe-a81d-4154-bb13-0a535ef38ec4 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - ac9126c4-fb32-4b91-a917-14afa77312b2 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - b7a38358-9a7c-4915-a4a8-1390f966aefe : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 67a73ec6-e81c-42f8-b85e-fc15d5966690 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 06321216-4c35-4717-a5ec-c075e1dd3ad8 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 89c5616a-689a-4fb8-a464-fb1c27c466ba : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3104fa04-2fbd-4caa-91dc-85ab33c7bdeb : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - bf5f4f1b-55eb-475d-8222-7307eb64b75a : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - f2a46ae6-55d0-4f38-a291-c2ae7d4a66c7 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - a74d5423-0286-4dcd-ac47-b0b30a55f251 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 9e2d8add-61ed-4854-bf90-d39d4625ca66 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - afc6b0b5-c6fc-484d-a569-a291bb9db424 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 7b83d88b-9fc3-40ea-a81f-14677de9c098 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 42537649-9ee2-45d6-a5f8-a03eac974841 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test    ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 6431ff1d-559e-47d4-a9d3-2d7ea84ed1d5 : 10

Logstash 7.15.2
Why does it behaves like this?

edit: I found the number of flush executions is related to the number of ingesting events, do not know why

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.