I am writing some logstash filter plugin and found one weird behaviour. It seems the LS somehow calls flush method multiple times even if I use 1 worker and 1 pipeline called "def". I did some tests and here are parts of the code and the result:
def register
@esde_uuid = SecureRandom.uuid
@last_flush = Concurrent::AtomicFixnum.new(0)
th = Thread.current.object_id
@logger.warn("MASTER THREAD ID: #{th}")
end
def flush(options = {})
return if @refresh_interval == 0
# Add 5 seconds to @last_flush counter
# since this method is called every 5 seconds.
@last_flush.update { |v| v + 5 }
return unless @last_flush.value >= @refresh_interval
euuid = @esde_uuid
uuid = SecureRandom.uuid
lf = @last_flush.value
th = Thread.current.object_id
@logger.warn("TH #{th} FLUSH! #{euuid} - #{uuid} : #{lf}")
# Reset counter since metrics were flushed
@last_flush.value = 0
return
end
private
def periodic_flush
true
end
@refresh_interval is set to 10 in plugin config but each 5 seconds I got following multiple lines instead of ONE per 10 second.
[2021-12-16T01:12:14,835][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3adde33a-66fc-40d6-a9a1-bb473c7756c4 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - cd9e8f67-e86c-448a-8259-36ce15fb761c : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 9bd1b5c3-c843-41f5-8293-5b53de985685 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 387f652f-92dc-481d-9e9f-91ab8e5623fa : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 8278ce82-41d5-44b9-b3e7-3bdd6677e4a4 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 137a65f2-b19d-4fbe-a75e-f20c15eb4b8f : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 5c5b28b5-9a94-4b98-ad9f-041999838620 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - f09389ec-d32c-41e6-be3d-0f3c7f061e43 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1010a008-20c2-4f8f-b534-cdcffe1494aa : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 909687e9-ae9c-40fe-b703-28450fbd074a : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0a03bcdf-33c8-4d5e-a607-2b4d18d1e97f : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - c6408aa0-4951-45d6-8c92-3d10453f5440 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - d37a3519-84ac-49d5-add9-667593d2af85 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 8a44d9f4-3eac-421d-944f-19c6f2701995 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3f38d6af-98b1-4193-b211-50d13e22976c : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - a92f4d9a-a6e5-4f36-93eb-4da67a8af958 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - b38b4103-c52e-4f8e-97a6-c9d8be837a15 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3e6c97cc-08aa-4677-a1c3-856477f4ceb4 : 10
[2021-12-16T01:12:14,836][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1a40f369-8b3f-4a07-b8b0-c29a75422a77 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - fde0b854-e1cf-4d60-8566-29a463fde526 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 66fd0d74-155e-4155-a242-cb488bb2f9df : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0177a041-c92b-46b4-9db9-ea974092d703 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - c9610e99-7f85-4b69-b3cd-902ba5d3049e : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 720edf27-e536-45e0-bde8-02c67d9ec4e6 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 9489ccd5-e3d5-40b3-92c6-f2c85f68e533 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0c2b1bcc-b34c-4892-bff7-cb4af771d90d : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 14e579dd-7ce7-4d33-a9f2-aca3b0da08ee : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 7079efc6-154c-4753-acdf-7caaeababff3 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 62e34c69-2569-463e-8918-933b05c308e2 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 6138d68f-ac63-4f27-a37a-7e126b2eb440 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 55ac10d1-30e3-45c6-9f47-14bdd4eec7f9 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1e54199f-933b-4a98-8d33-9fbf99fb0957 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1cd0abf8-ca2e-4aae-b869-119a86c8c97e : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - e7e90036-209c-4be3-9668-20c003a14d11 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - e10b0ddc-b3e3-4c1f-959c-f120ce7876a2 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 37a8ebcd-cd3e-49e6-bbef-cb3d2565c39c : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - cb99c39b-eee5-41b3-b655-559194dbd90b : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - bafff4cd-c6fc-45a2-9511-251080eec860 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 17c3b38f-1204-4022-a6b6-6c1d03375a1b : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - c45edf4e-9633-406f-8fbf-6fd2006e16c1 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 7b69b06d-f8a5-4950-83b5-632694ff90b3 : 10
[2021-12-16T01:12:14,837][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 0ce22cda-bb02-4823-996a-5347fabeac8f : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 8037fe5d-0c5f-40b8-a631-5fe2a8e94942 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3190dba7-dabc-490e-8dad-723d91684b14 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3bb723b0-1690-46e3-b763-b1b34fdedfce : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - eb1416a1-9b75-4fc3-ae24-47e21905a823 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - a51c1298-731a-4cad-9338-abea4a2317b1 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3b5ac7d3-263a-4ba0-809c-d74fc776ae97 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 1f94d7fe-a81d-4154-bb13-0a535ef38ec4 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - ac9126c4-fb32-4b91-a917-14afa77312b2 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - b7a38358-9a7c-4915-a4a8-1390f966aefe : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 67a73ec6-e81c-42f8-b85e-fc15d5966690 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 06321216-4c35-4717-a5ec-c075e1dd3ad8 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 89c5616a-689a-4fb8-a464-fb1c27c466ba : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 3104fa04-2fbd-4caa-91dc-85ab33c7bdeb : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - bf5f4f1b-55eb-475d-8222-7307eb64b75a : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - f2a46ae6-55d0-4f38-a291-c2ae7d4a66c7 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - a74d5423-0286-4dcd-ac47-b0b30a55f251 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 9e2d8add-61ed-4854-bf90-d39d4625ca66 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - afc6b0b5-c6fc-484d-a569-a291bb9db424 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 7b83d88b-9fc3-40ea-a81f-14677de9c098 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 42537649-9ee2-45d6-a5f8-a03eac974841 : 10
[2021-12-16T01:12:14,838][WARN ][logstash.filters.test ][def] TH 2052 FLUSH! 162c4747-c964-467c-8c6c-a79c8ab81314 - 6431ff1d-559e-47d4-a9d3-2d7ea84ed1d5 : 10
Logstash 7.15.2
Why does it behaves like this?
edit: I found the number of flush executions is related to the number of ingesting events, do not know why