Logstash doesn't deliver input to output

Elastic Stack Logstash
1

I don't understand why logstash doesn't deliver input data to output.
logstash is running in docker, alongside kibana and elastic.
logstash.conf

input {
  file {
    mode => "read"
    codec => "json_lines"
    add_field => {
      "[@metadata][source]" => "modsec"
    }
    path => "/usr/share/logstash/ingest_data/modsec/*.log"
    file_completed_action => "log"
    file_completed_log_path => "/usr/share/logstash/ingest_data/logstash_completed.log"
  }
}
  
filter {
  if "[@metadata][source]" == "modsec" {
    date {
      id => "modsec_date_parse"
      match => [ "[transaction][time_stamp]", "EEE MMM dd HH:mm:ss yyyy" ]
      timezone => "Asia/Jerusalem"
      remove_field => [ "[transaction][time_stamp]" ]
      enable_metric => true
      target => "@timestamp"
    }
    mutate {
      lowercase => [ "fieldname" ]
    }
  }
}
  
output {
  elasticsearch {
    index => "logstash-%{+YYYY.MM.dd}"
    hosts=> "${ELASTIC_HOSTS}"
    user=> "${ELASTIC_USER}"
    password=> "${ELASTIC_PASSWORD}"
    cacert=> "certs/ca/ca.crt"
  }
  file {
    id => "logstash_output"
    enable_metric => true
    path => "/usr/share/logstash/ingest_data/logstash_output"
    codec => line {
      format => "custom format: %{message}"
    }
  }
}

logs:

[2024-09-29T19:16:56,073][DEBUG][org.logstash.execution.AbstractPipelineExt] Flow metric registered: `worker_utilization` in namespace `[:stats, :pipelines, :main, :plugins, :outputs, :logstash_output, :flow]`
[2024-09-29T19:16:56,073][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main"}
[2024-09-29T19:16:56,094][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x446b43ac /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-09-29T19:16:56,770][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.67}
[2024-09-29T19:16:56,781][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_dc0fc26390d66ab73c5c25095c694398", :path=>["/usr/share/logstash/ingest_data/modsec/*.log"]}
[2024-09-29T19:16:56,785][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-09-29T19:16:56,789][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x446b43ac /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-09-29T19:16:56,797][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2024-09-29T19:16:56,800][INFO ][filewatch.observingread  ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] START, creating Discoverer, Watch with file and sincedb collections
[2024-09-29T19:16:56,807][DEBUG][filewatch.sincedbcollection][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] open: reading from /usr/share/logstash/data/plugins/inputs/file/.sincedb_dc0fc26390d66ab73c5c25095c694398
[2024-09-29T19:16:56,810][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2024-09-29T19:16:56,835][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Received line {:path=>"/usr/share/logstash/ingest_data/modsec/audit.log", :text=>"{\"transaction\":{\"client_ip\":\"5.33.164.194\",\"time_stamp\":\"Sun Sep 29 19:11:18 2024\",\"server_id\":\"cfbe5a4abbf6f2effb9d89132e930fcb2e9c6ddb\",\"client_port\":60906,\"host_ip\":\"172.18.0.2\",\"host_port\":443,\"unique_id\":\"172763707882.783232\",\"request\":{\"method\":\"GET\",\"http_version\":2.0,\"uri\":\"/api/endpoints/2/docker/containers/395aabd7692a7cbfbc01bf4574c56939cdf884d5258b8f171c34ce8d0db2829e/logs?since=0&stderr=1&stdout=1&tail=100&timestamps=0\",\"headers\":{\"sec-fetch-site\":\"same-origin\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0\",\"referer\":\"https://portainer.local/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"cookie\":\"portainer_api_key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJqZWx1bSIsInJvbGUiOjEsInNjb3BlIjoiZGVmYXVsdCIsImZvcmNlQ2hhbmdlUGFzc3dvcmQiOmZhbHNlLCJleHAiOjE3Mjc2NjE5NDQsImlhdCI6MTcyNzYzMzE0NH0.NlRLu_WpmTnXXdAn6FVY2k6Q1sKbZqe9P56Qc8YjeTo; _gorilla_csrf=MTcyNzYzMzE0NHxJbVZNZWsxMloyWkpRa1ozV0hodlVtUkVja2RTUm01eFVYbHpTa05LZG5BdlpFWkRSbmhxWVhsSFdXczlJZ289fFZwtEwEGa5acoWBjbH9kkUAogaqxklaCnqtmsIT4Mqm\",\"te\":\"trailers\",\"accept-language\":\"en-US,en;q=0.5\",\"x-csrf-token\":\"i5NTCflIW6WKDpAg6lFytf4dwvoYEqpJPqUPf5l5HdbzL5+3/oBf+Z3IFH3k4OOjhI0IOFo0UDZK9Yq5r8sEXw==\",\"accept\":\"application/json, text/plain, */*\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"host\":\"portainer.local\"}},\"response\":{\"body\":\"\",\"http_code\":200,\"headers\":{\"X-Csrf-Token\":\"sG9B3VkyYmcTibuxEjnP37mxgRSIv37a2q52sk7eb1zI041jXvpmOwRPP+wciF7JwyFL1sqZhKWu/vN0eGx21Q==\",\"Vary\":\"Cookie\",\"Docker-Experimental\":\"false\",\"Content-Type\":\"application/vnd.docker.multiplexed-stream\",\"Api-Version\":\"1.47\",\"X-Xss-Protection\":\"1; mode=block\",\"Ostype\":\"linux\",\"Connection\":\"close\",\"X-Content-Type-Options\":\"nosniff\",\"Date\":\"Sun, 29 Sep 2024 19:11:18 GMT\",\"Server\":\"nginx\",\"Strict-Transport-Security\":\"max-age=31536000\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.13 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"DetectionOnly\",\"components\":[\"OWASP_CRS/4.7.0\\\"\"]},\"messages\":[{\"message\":\"Remote Command Execution: Direct Unix Command Execution\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `(?i)(?:^|b[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?u[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?s[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)? (4180 characters omitted)' against variable `ARGS_NAMES:stdout' (Value: `stdout' )\",\"reference\":\"o0,6v117,6o0,6v126,6\",\"ruleId\":\"932260\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\",\"lineNumber\":\"519\",\"data\":\"Matched Data: stdout found within ARGS_NAMES:stdout: stdout\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-shell\",\"platform-unix\",\"attack-rce\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/152/248/88\",\"PCI/6.5.2\"],\"maturity\":\"0\",\"accuracy\":\"0\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 10)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `10' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"222\",\"data\":\"\",\"severity\":\"0\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"anomaly-evaluation\",\"OWASP_CRS\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}"}
[2024-09-29T19:16:56,846][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Start periodic runner
[2024-09-29T19:16:56,854][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@id = "json_lines_2085c679-0acf-4135-aad9-9778b871f31e"
[2024-09-29T19:16:56,855][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@enable_metric = true
[2024-09-29T19:16:56,856][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@charset = "UTF-8"
[2024-09-29T19:16:56,856][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@delimiter = "\n"
[2024-09-29T19:16:56,859][INFO ][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2024-09-29T19:16:56,861][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Received line {:path=>"/usr/share/logstash/ingest_data/modsec/audit.log", :text=>"{\"transaction\":{\"client_ip\":\"5.33.164.194\",\"time_stamp\":\"Sun Sep 29 19:11:21 2024\",\"server_id\":\"cfbe5a4abbf6f2effb9d89132e930fcb2e9c6ddb\",\"client_port\":60906,\"host_ip\":\"172.18.0.2\",\"host_port\":443,\"unique_id\":\"172763708118.613418\",\"request\":{\"method\":\"GET\",\"http_version\":2.0,\"uri\":\"/api/endpoints/2/docker/containers/395aabd7692a7cbfbc01bf4574c56939cdf884d5258b8f171c34ce8d0db2829e/logs?since=0&stderr=1&stdout=1&tail=100&timestamps=0\",\"headers\":{\"sec-fetch-site\":\"same-origin\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0\",\"referer\":\"https://portainer.local/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"cookie\":\"portainer_api_key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJqZWx1bSIsInJvbGUiOjEsInNjb3BlIjoiZGVmYXVsdCIsImZvcmNlQ2hhbmdlUGFzc3dvcmQiOmZhbHNlLCJleHAiOjE3Mjc2NjE5NDQsImlhdCI6MTcyNzYzMzE0NH0.NlRLu_WpmTnXXdAn6FVY2k6Q1sKbZqe9P56Qc8YjeTo; _gorilla_csrf=MTcyNzYzMzE0NHxJbVZNZWsxMloyWkpRa1ozV0hodlVtUkVja2RTUm01eFVYbHpTa05LZG5BdlpFWkRSbmhxWVhsSFdXczlJZ289fFZwtEwEGa5acoWBjbH9kkUAogaqxklaCnqtmsIT4Mqm\",\"te\":\"trailers\",\"accept-language\":\"en-US,en;q=0.5\",\"x-csrf-token\":\"sG9B3VkyYmcTibuxEjnP37mxgRSIv37a2q52sk7eb1zI041jXvpmOwRPP+wciF7JwyFL1sqZhKWu/vN0eGx21Q==\",\"accept\":\"application/json, text/plain, */*\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"host\":\"portainer.local\"}},\"response\":{\"body\":\"\",\"http_code\":200,\"headers\":{\"X-Csrf-Token\":\"xZasp5sYtlbFErP057XTiBSdH+Vd5x2NYRmN/eKIDaW9KmAZnNCyCtLUN6npBEKebg3VJx/B5/IVSQg71DoULA==\",\"Vary\":\"Cookie\",\"Docker-Experimental\":\"false\",\"Content-Type\":\"application/vnd.docker.multiplexed-stream\",\"Api-Version\":\"1.47\",\"X-Xss-Protection\":\"1; mode=block\",\"Ostype\":\"linux\",\"Connection\":\"close\",\"X-Content-Type-Options\":\"nosniff\",\"Date\":\"Sun, 29 Sep 2024 19:11:21 GMT\",\"Server\":\"nginx\",\"Strict-Transport-Security\":\"max-age=31536000\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.13 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"DetectionOnly\",\"components\":[\"OWASP_CRS/4.7.0\\\"\"]},\"messages\":[{\"message\":\"Remote Command Execution: Direct Unix Command Execution\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `(?i)(?:^|b[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?u[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?s[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)? (4180 characters omitted)' against variable `ARGS_NAMES:stdout' (Value: `stdout' )\",\"reference\":\"o0,6v117,6o0,6v126,6\",\"ruleId\":\"932260\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\",\"lineNumber\":\"519\",\"data\":\"Matched Data: stdout found within ARGS_NAMES:stdout: stdout\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-shell\",\"platform-unix\",\"attack-rce\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/152/248/88\",\"PCI/6.5.2\"],\"maturity\":\"0\",\"accuracy\":\"0\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 10)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `10' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"222\",\"data\":\"\",\"severity\":\"0\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"anomaly-evaluation\",\"OWASP_CRS\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}"}
[2024-09-29T19:16:56,865][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Received line {:path=>"/usr/share/logstash/ingest_data/modsec/audit.log", :text=>"{\"transaction\":{\"client_ip\":\"5.33.164.194\",\"time_stamp\":\"Sun Sep 29 19:11:48 2024\",\"server_id\":\"cfbe5a4abbf6f2effb9d89132e930fcb2e9c6ddb\",\"client_port\":60906,\"host_ip\":\"172.18.0.2\",\"host_port\":443,\"unique_id\":\"172763710894.080921\",\"request\":{\"method\":\"GET\",\"http_version\":2.0,\"uri\":\"/api/endpoints/2/docker/containers/395aabd7692a7cbfbc01bf4574c56939cdf884d5258b8f171c34ce8d0db2829e/logs?since=0&stderr=1&stdout=1&tail=100&timestamps=0\",\"headers\":{\"sec-fetch-site\":\"same-origin\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0\",\"referer\":\"https://portainer.local/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"cookie\":\"portainer_api_key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJqZWx1bSIsInJvbGUiOjEsInNjb3BlIjoiZGVmYXVsdCIsImZvcmNlQ2hhbmdlUGFzc3dvcmQiOmZhbHNlLCJleHAiOjE3Mjc2NjE5NDQsImlhdCI6MTcyNzYzMzE0NH0.NlRLu_WpmTnXXdAn6FVY2k6Q1sKbZqe9P56Qc8YjeTo; _gorilla_csrf=MTcyNzYzMzE0NHxJbVZNZWsxMloyWkpRa1ozV0hodlVtUkVja2RTUm01eFVYbHpTa05LZG5BdlpFWkRSbmhxWVhsSFdXczlJZ289fFZwtEwEGa5acoWBjbH9kkUAogaqxklaCnqtmsIT4Mqm\",\"te\":\"trailers\",\"accept-language\":\"en-US,en;q=0.5\",\"x-csrf-token\":\"PIXPTSDaPPSE3/3xZ/agGVlX2yIpQmvk+Y3Ke+aU4elEOQPzJxI4qJMZeaxpRzEPI8cR4GtkkZuN3U+90Cb4YA==\",\"accept\":\"application/json, text/plain, */*\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"host\":\"portainer.local\"}},\"response\":{\"body\":\"\",\"http_code\":200,\"headers\":{\"X-Csrf-Token\":\"8P3MBQOONTpaWJfDhehmrTqlyWn1Yp9CFqMSDRSVCx6IQQC7BEYxZk2eE56LWfe7QDUDq7dEZT1i85fLIicSlw==\",\"Vary\":\"Cookie\",\"Docker-Experimental\":\"false\",\"Content-Type\":\"application/vnd.docker.multiplexed-stream\",\"Api-Version\":\"1.47\",\"X-Xss-Protection\":\"1; mode=block\",\"Ostype\":\"linux\",\"Connection\":\"close\",\"X-Content-Type-Options\":\"nosniff\",\"Date\":\"Sun, 29 Sep 2024 19:11:48 GMT\",\"Server\":\"nginx\",\"Strict-Transport-Security\":\"max-age=31536000\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.13 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"DetectionOnly\",\"components\":[\"OWASP_CRS/4.7.0\\\"\"]},\"messages\":[{\"message\":\"Remote Command Execution: Direct Unix Command Execution\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `(?i)(?:^|b[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?u[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?s[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)? (4180 characters omitted)' against variable `ARGS_NAMES:stdout' (Value: `stdout' )\",\"reference\":\"o0,6v117,6o0,6v126,6\",\"ruleId\":\"932260\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\",\"lineNumber\":\"519\",\"data\":\"Matched Data: stdout found within ARGS_NAMES:stdout: stdout\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-shell\",\"platform-unix\",\"attack-rce\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/152/248/88\",\"PCI/6.5.2\"],\"maturity\":\"0\",\"accuracy\":\"0\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 10)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `10' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"222\",\"data\":\"\",\"severity\":\"0\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"anomaly-evaluation\",\"OWASP_CRS\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}"}
[2024-09-29T19:16:56,878][DEBUG][filewatch.sincedbcollection][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] writing sincedb (delta since last write = 1727637416)
[2024-09-29T19:16:56,884][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] handle_deletable_path {:path=>"/usr/share/logstash/ingest_data/modsec/audit.log"}
[2024-09-29T19:16:56,890][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Received line {:path=>"/usr/share/logstash/ingest_data/modsec/audit2.log", :text=>"{\"transaction\":{\"client_ip\":\"5.33.164.194\",\"time_stamp\":\"Sun Sep 29 19:14:45 2024\",\"server_id\":\"cfbe5a4abbf6f2effb9d89132e930fcb2e9c6ddb\",\"client_port\":60906,\"host_ip\":\"172.18.0.2\",\"host_port\":443,\"unique_id\":\"172763728575.845058\",\"request\":{\"method\":\"GET\",\"http_version\":2.0,\"uri\":\"/api/endpoints/2/docker/containers/395aabd7692a7cbfbc01bf4574c56939cdf884d5258b8f171c34ce8d0db2829e/logs?since=0&stderr=1&stdout=1&tail=100&timestamps=0\",\"headers\":{\"sec-fetch-site\":\"same-origin\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0\",\"referer\":\"https://portainer.local/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"cookie\":\"portainer_api_key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJqZWx1bSIsInJvbGUiOjEsInNjb3BlIjoiZGVmYXVsdCIsImZvcmNlQ2hhbmdlUGFzc3dvcmQiOmZhbHNlLCJleHAiOjE3Mjc2NjE5NDQsImlhdCI6MTcyNzYzMzE0NH0.NlRLu_WpmTnXXdAn6FVY2k6Q1sKbZqe9P56Qc8YjeTo; _gorilla_csrf=MTcyNzYzMzE0NHxJbVZNZWsxMloyWkpRa1ozV0hodlVtUkVja2RTUm01eFVYbHpTa05LZG5BdlpFWkRSbmhxWVhsSFdXczlJZ289fFZwtEwEGa5acoWBjbH9kkUAogaqxklaCnqtmsIT4Mqm\",\"te\":\"trailers\",\"accept-language\":\"en-US,en;q=0.5\",\"x-csrf-token\":\"BAyh68tKnZ48Z2DI3FU/ddl6Wl8ng46wIGP2V5S32X58sG1VzIKZwiuh5JXS5K5jo+qQnWWldM9UM3ORogXA9w==\",\"accept\":\"application/json, text/plain, */*\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"host\":\"portainer.local\"}},\"response\":{\"body\":\"\",\"http_code\":200,\"headers\":{\"X-Csrf-Token\":\"+rOr8Qq6m77PseI8cSSLvWiTkSH4JhUR1w5n4Kd4H3uCD2dPDXKf4th3ZmF/lRqrEgNb47oA726jXuImkcoG8g==\",\"Vary\":\"Cookie\",\"Docker-Experimental\":\"false\",\"Content-Type\":\"application/vnd.docker.multiplexed-stream\",\"Api-Version\":\"1.47\",\"X-Xss-Protection\":\"1; mode=block\",\"Ostype\":\"linux\",\"Connection\":\"close\",\"X-Content-Type-Options\":\"nosniff\",\"Date\":\"Sun, 29 Sep 2024 19:14:46 GMT\",\"Server\":\"nginx\",\"Strict-Transport-Security\":\"max-age=31536000\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.13 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"DetectionOnly\",\"components\":[\"OWASP_CRS/4.7.0\\\"\"]},\"messages\":[{\"message\":\"Remote Command Execution: Direct Unix Command Execution\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `(?i)(?:^|b[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?u[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?s[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)? (4180 characters omitted)' against variable `ARGS_NAMES:stdout' (Value: `stdout' )\",\"reference\":\"o0,6v117,6o0,6v126,6\",\"ruleId\":\"932260\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\",\"lineNumber\":\"519\",\"data\":\"Matched Data: stdout found within ARGS_NAMES:stdout: stdout\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-shell\",\"platform-unix\",\"attack-rce\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/152/248/88\",\"PCI/6.5.2\"],\"maturity\":\"0\",\"accuracy\":\"0\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 10)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `10' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"222\",\"data\":\"\",\"severity\":\"0\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"anomaly-evaluation\",\"OWASP_CRS\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}"}
[2024-09-29T19:16:56,904][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@id = "json_lines_2085c679-0acf-4135-aad9-9778b871f31e"
[2024-09-29T19:16:56,906][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@enable_metric = true
[2024-09-29T19:16:56,907][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@charset = "UTF-8"
[2024-09-29T19:16:56,907][DEBUG][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] config LogStash::Codecs::JSONLines/@delimiter = "\n"
[2024-09-29T19:16:56,909][INFO ][logstash.codecs.jsonlines][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2024-09-29T19:16:56,910][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Received line {:path=>"/usr/share/logstash/ingest_data/modsec/audit2.log", :text=>"{\"transaction\":{\"client_ip\":\"5.33.164.194\",\"time_stamp\":\"Sun Sep 29 19:14:48 2024\",\"server_id\":\"cfbe5a4abbf6f2effb9d89132e930fcb2e9c6ddb\",\"client_port\":60906,\"host_ip\":\"172.18.0.2\",\"host_port\":443,\"unique_id\":\"172763728814.485956\",\"request\":{\"method\":\"GET\",\"http_version\":2.0,\"uri\":\"/api/endpoints/2/docker/containers/395aabd7692a7cbfbc01bf4574c56939cdf884d5258b8f171c34ce8d0db2829e/logs?since=0&stderr=1&stdout=1&tail=100&timestamps=0\",\"headers\":{\"sec-fetch-site\":\"same-origin\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0\",\"referer\":\"https://portainer.local/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"cookie\":\"portainer_api_key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJqZWx1bSIsInJvbGUiOjEsInNjb3BlIjoiZGVmYXVsdCIsImZvcmNlQ2hhbmdlUGFzc3dvcmQiOmZhbHNlLCJleHAiOjE3Mjc2NjE5NDQsImlhdCI6MTcyNzYzMzE0NH0.NlRLu_WpmTnXXdAn6FVY2k6Q1sKbZqe9P56Qc8YjeTo; _gorilla_csrf=MTcyNzYzMzE0NHxJbVZNZWsxMloyWkpRa1ozV0hodlVtUkVja2RTUm01eFVYbHpTa05LZG5BdlpFWkRSbmhxWVhsSFdXczlJZ289fFZwtEwEGa5acoWBjbH9kkUAogaqxklaCnqtmsIT4Mqm\",\"te\":\"trailers\",\"accept-language\":\"en-US,en;q=0.5\",\"x-csrf-token\":\"+rOr8Qq6m77PseI8cSSLvWiTkSH4JhUR1w5n4Kd4H3uCD2dPDXKf4th3ZmF/lRqrEgNb47oA726jXuImkcoG8g==\",\"accept\":\"application/json, text/plain, */*\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"host\":\"portainer.local\"}},\"response\":{\"body\":\"\",\"http_code\":200,\"headers\":{\"X-Csrf-Token\":\"bGkKC9WDDGs+Cxyt9PeWX+LOtSInWra0Qqg5gHJHNRoU1ca10ksINynNmPD6RgdJmF5/4GV8TMs2+LxGRPUskw==\",\"Vary\":\"Cookie\",\"Docker-Experimental\":\"false\",\"Content-Type\":\"application/vnd.docker.multiplexed-stream\",\"Api-Version\":\"1.47\",\"X-Xss-Protection\":\"1; mode=block\",\"Ostype\":\"linux\",\"Connection\":\"close\",\"X-Content-Type-Options\":\"nosniff\",\"Date\":\"Sun, 29 Sep 2024 19:14:49 GMT\",\"Server\":\"nginx\",\"Strict-Transport-Security\":\"max-age=31536000\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.13 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"DetectionOnly\",\"components\":[\"OWASP_CRS/4.7.0\\\"\"]},\"messages\":[{\"message\":\"Remote Command Execution: Direct Unix Command Execution\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `(?i)(?:^|b[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?u[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?s[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)? (4180 characters omitted)' against variable `ARGS_NAMES:stdout' (Value: `stdout' )\",\"reference\":\"o0,6v117,6o0,6v126,6\",\"ruleId\":\"932260\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\",\"lineNumber\":\"519\",\"data\":\"Matched Data: stdout found within ARGS_NAMES:stdout: stdout\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-shell\",\"platform-unix\",\"attack-rce\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/152/248/88\",\"PCI/6.5.2\"],\"maturity\":\"0\",\"accuracy\":\"0\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 10)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `10' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"222\",\"data\":\"\",\"severity\":\"0\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"anomaly-evaluation\",\"OWASP_CRS\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}"}
[2024-09-29T19:16:56,919][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] Received line {:path=>"/usr/share/logstash/ingest_data/modsec/audit2.log", :text=>"{\"transaction\":{\"client_ip\":\"5.33.164.194\",\"time_stamp\":\"Sun Sep 29 19:14:57 2024\",\"server_id\":\"cfbe5a4abbf6f2effb9d89132e930fcb2e9c6ddb\",\"client_port\":60906,\"host_ip\":\"172.18.0.2\",\"host_port\":443,\"unique_id\":\"172763729749.516967\",\"request\":{\"method\":\"GET\",\"http_version\":2.0,\"uri\":\"/api/endpoints/2/docker/containers/395aabd7692a7cbfbc01bf4574c56939cdf884d5258b8f171c34ce8d0db2829e/logs?since=0&stderr=1&stdout=1&tail=100&timestamps=0\",\"headers\":{\"sec-fetch-site\":\"same-origin\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0\",\"referer\":\"https://portainer.local/\",\"accept-encoding\":\"gzip, deflate, br, zstd\",\"cookie\":\"portainer_api_key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidXNlcm5hbWUiOiJqZWx1bSIsInJvbGUiOjEsInNjb3BlIjoiZGVmYXVsdCIsImZvcmNlQ2hhbmdlUGFzc3dvcmQiOmZhbHNlLCJleHAiOjE3Mjc2NjE5NDQsImlhdCI6MTcyNzYzMzE0NH0.NlRLu_WpmTnXXdAn6FVY2k6Q1sKbZqe9P56Qc8YjeTo; _gorilla_csrf=MTcyNzYzMzE0NHxJbVZNZWsxMloyWkpRa1ozV0hodlVtUkVja2RTUm01eFVYbHpTa05LZG5BdlpFWkRSbmhxWVhsSFdXczlJZ289fFZwtEwEGa5acoWBjbH9kkUAogaqxklaCnqtmsIT4Mqm\",\"te\":\"trailers\",\"accept-language\":\"en-US,en;q=0.5\",\"x-csrf-token\":\"tfyvdkwu9bFzwxDCEtDqHEmJ6/88UBnFx36LtUjoYzLNQGPIS+bx7WQFlJ8cYXsKMxkhPX5247qzLg5zflp6uw==\",\"accept\":\"application/json, text/plain, */*\",\"sec-fetch-mode\":\"cors\",\"sec-fetch-dest\":\"empty\",\"host\":\"portainer.local\"}},\"response\":{\"body\":\"\",\"http_code\":200,\"headers\":{\"X-Csrf-Token\":\"Wex5wG7A4OhSEyX5qriMdgtrbHuHNye2Xa1v8RQ/MLEhULV+aQjktEXVoaSkCR1gcfumucUR3ckp/eo3Io0pOA==\",\"Vary\":\"Cookie\",\"Docker-Experimental\":\"false\",\"Content-Type\":\"application/vnd.docker.multiplexed-stream\",\"Api-Version\":\"1.47\",\"X-Xss-Protection\":\"1; mode=block\",\"Ostype\":\"linux\",\"Connection\":\"close\",\"X-Content-Type-Options\":\"nosniff\",\"Date\":\"Sun, 29 Sep 2024 19:14:58 GMT\",\"Server\":\"nginx\",\"Strict-Transport-Security\":\"max-age=31536000\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.13 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.3\",\"secrules_engine\":\"DetectionOnly\",\"components\":[\"OWASP_CRS/4.7.0\\\"\"]},\"messages\":[{\"message\":\"Remote Command Execution: Direct Unix Command Execution\",\"details\":{\"match\":\"Matched \\\"Operator `Rx' with parameter `(?i)(?:^|b[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?u[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)?\\\\$[!#\\\\(\\\\*\\\\-0-9\\\\?@_a-\\\\{]*)?\\\\x5c?s[\\\\\\\"'\\\\)\\\\[\\\\x5c]*(?:(?:(?:\\\\|\\\\||&&)[\\\\s\\\\x0b]*)? (4180 characters omitted)' against variable `ARGS_NAMES:stdout' (Value: `stdout' )\",\"reference\":\"o0,6v117,6o0,6v126,6\",\"ruleId\":\"932260\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf\",\"lineNumber\":\"519\",\"data\":\"Matched Data: stdout found within ARGS_NAMES:stdout: stdout\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-shell\",\"platform-unix\",\"attack-rce\",\"paranoia-level/1\",\"OWASP_CRS\",\"capec/1000/152/248/88\",\"PCI/6.5.2\"],\"maturity\":\"0\",\"accuracy\":\"0\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 10)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:BLOCKING_INBOUND_ANOMALY_SCORE' (Value: `10' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/modsecurity/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"222\",\"data\":\"\",\"severity\":\"0\",\"ver\":\"OWASP_CRS/4.7.0\",\"rev\":\"\",\"tags\":[\"anomaly-evaluation\",\"OWASP_CRS\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}"}
[2024-09-29T19:16:56,921][DEBUG][logstash.inputs.file     ][main][2b2622e24c2a9f5d1cc7f117b36fecf61b1605d666db7e17d1fabdd75387734d] handle_deletable_path {:path=>"/usr/share/logstash/ingest_data/modsec/audit2.log"}
[2024-09-29T19:16:58,087][DEBUG][logstash.outputs.file    ][main] Starting flush cycle
2

Hi @evgeniy

Most likely wrong codec

NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec.

Try just

codec => "json"

Yeah it's confusing.

Also, if logstash has already read the file, it won't read it again...

1 Like
3

To expand on what Stephen said, the file input reads newline delimited lines from the file, strips off the newline, then feeds the lines to the codec. The json_lines codec collects those lines until it gets a newline. But the file input removes the newlines, so the codec will never get one. It just keeps accumulating.

I think the developers are considering changing this.

Looking at the github history a change was recently made (August) to limit the accumulated text to 20MB before throwing an exception, then that limit was increased to 512 MB, and I think that limit is still being discussed.

Note that your logstash install will not reflect the current github code base until a new version of the codec is packaged/released (not sure what the right word for that is).

I've never seen a use case where I needed to consume a single line of JSON that was 40 MB, but the discussion the elastic developers are having tells me that they need to support it.

And, yeah, change the codec to "json".

2 Likes
4

@stephenb , @Badger , guys, you're great! It worked! Thank you very much!!!