Processing large CSV by file module

Elastic Stack Logstash
1

Hi, I'm facing an issue with decoding my CSV files after that when I've changes the approach to upload file. My first solution has been worked through input on multiply port to logstash.
What can I observer that logstash are shifting data not read correctly columns in order thus we got undesired effect. Looks like it's getting lost in the fields, doesn't keep order hence spouts decoding errors.
I'm using logstash in 8.2.1 with 16 workes on 8GB of mem. Files have an average capacity ~68MB
I don't see any side effect when I'm processing only one file.
Do You have any idea for configuration?
Thanks for You insight.

Of course the same results I can see on older logstash, one things is not related -> statistics when I saw a huge count of fail why logstash doesn't show such behavior under stats?

bash-4.2$ curl -XGET 'localhost:9600/_node/stats/pipelines?pretty'
{
  "host" : "9da12cd83ec0",
  "version" : "7.16.3",
  "http_address" : "0.0.0.0:9600",
  "id" : "8271ea02-bf31-45df-96b8-7e182b60f2eb",
  "name" : "9da12cd83ec0",
  "ephemeral_id" : "a870f636-ad60-49e6-a99d-05417e49ebc3",
  "status" : "green",
  "snapshot" : false,
  "pipeline" : {
    "workers" : 16,
    "batch_size" : 2000,
    "batch_delay" : 50
  },
  "pipelines" : {
    "main" : {
      "events" : {
        "duration_in_millis" : 4356256,
        "filtered" : 375341,
        "in" : 375341,
        "out" : 375341,
        "queue_push_duration_in_millis" : 214073
      },
      "plugins" : {
        "inputs" : [ {
          "id" : "1c445ccd66431ff2b5bb0d73d71a828edeccf9a939c9d118f8d93f078769f419",
          "events" : {
            "out" : 375341,
            "queue_push_duration_in_millis" : 214073
          },
          "name" : "file"
        } ],
        "codecs" : [ {
          "id" : "plain_3e5451d6-9737-4659-99f8-6f53db7ae0af",
          "decode" : {
            "duration_in_millis" : 0,
            "out" : 0,
            "writes_in" : 0
          },
          "name" : "plain",
          "encode" : {
            "duration_in_millis" : 0,
            "writes_in" : 0
          }
        }, {
          "id" : "rubydebug_93f76b54-a791-44d3-9ad4-2b6cc6ed5da5",
          "decode" : {
            "duration_in_millis" : 0,
            "out" : 0,
            "writes_in" : 0
          },
          "name" : "rubydebug",
          "encode" : {
            "duration_in_millis" : 216059,
            "writes_in" : 375328
          }
        } ],
        "filters" : [ {
          "id" : "044130fde2e6b0202cb7a60ab3752418b8ee4854c4ae84fda2158afef8083ec8",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 2516,
            "out" : 2516
          },
          "name" : "translate"
        }, {
          "id" : "f59f912273c5c1a4b861f05911eb18361e96e220050688ad5eb255b1fdc02603",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "cf28799c5975d56ed648acffc0a4dd387f4a12fb06ffc93a34d85924cc77ace4",
          "events" : {
            "duration_in_millis" : 5594,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "ruby"
        }, {
          "id" : "fee738e542ea9328f1603c881d1ca70c6e92ec194f3113890fea2cfa40b1fa21",
          "events" : {
            "duration_in_millis" : 410,
            "in" : 67622,
            "out" : 67622
          },
          "name" : "translate"
        }, {
          "id" : "2034d91a960ca65251752a4e70fbfc99ca573aa33aa122fb4830cf23ffb0975c",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 106,
            "out" : 106
          },
          "name" : "ruby"
        }, {
          "id" : "dbf6c1ac30e626576b9feb52c2d1d4bb0a5437e2cc29c03ddca203c28bf59512",
          "events" : {
            "duration_in_millis" : 7669,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "mutate"
        }, {
          "id" : "e28a1566c2399c451fb057934312323ae6eabad9a9fabed405d2a7cfa20d961e",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 4,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "90ddd7c0964035b712deafe5ffa416b7bae0bbd5944ac59df367fbbc44e60335",
          "events" : {
            "duration_in_millis" : 1194,
            "in" : 91301,
            "out" : 91301
          },
          "name" : "ruby"
        }, {
          "id" : "9942e81c44c8e61aa1b899aa19edd5e855c5d0c8d4289dbbb472369c5e16de56",
          "events" : {
            "duration_in_millis" : 486,
            "in" : 50008,
            "out" : 50008
          },
          "name" : "ruby"
        }, {
          "id" : "c35db69b16c88ba26c943fdc3d6a0254bbe332f43cd2dbf0325a69e2c491260d",
          "events" : {
            "duration_in_millis" : 11600,
            "in" : 99636,
            "out" : 99636
          },
          "name" : "mutate"
        }, {
          "id" : "84a020d13d896952b17012236cb7c4ce09ad1a9b167856faa00b08443793baa8",
          "events" : {
            "duration_in_millis" : 6346,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "mutate"
        }, {
          "id" : "a4366d54f0789ba412171c37705bcf9dbdc11cebb294e24e54e8591c890d9508",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 541,
            "out" : 541
          },
          "name" : "translate"
        }, {
          "id" : "8413b8c8bb14f827e8022ee09a9d4947c4b11a20050ba21703bf0eb11c43de4b",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 2,
            "out" : 2
          },
          "name" : "translate"
        }, {
          "id" : "cfe7c2cbfbac7300d4d174fccd23c86abcf9692362d9dedac96b8ce4b7f6a40b",
          "events" : {
            "duration_in_millis" : 603,
            "in" : 91301,
            "out" : 91301
          },
          "name" : "translate"
        }, {
          "id" : "8caaa134e4e74940ccf079ea979c4861ae6731b6251423f065fa503a587ffe68",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "mutate"
        }, {
          "id" : "dff17c5076f0a41608510b3d976ae7db2facd283893cac7b5a276da29559607f",
          "events" : {
            "duration_in_millis" : 17277,
            "in" : 275692,
            "out" : 275692
          },
          "name" : "mutate"
        }, {
          "id" : "56ab73447b306391736d20ef21b20b476745b785086bbd1ef53017d8fa905bdd",
          "events" : {
            "duration_in_millis" : 609,
            "in" : 91301,
            "out" : 91301
          },
          "name" : "translate"
        }, {
          "id" : "18976d77b1541a6f2ded211238df80d445031193a766a712de14d99d950fd098",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 1302,
            "out" : 1302
          },
          "name" : "ruby"
        }, {
          "id" : "e0b7c8f4c71133af2178b26d7fa76223c342b3c4c91e1e995ed67cf034bf8d4c",
          "events" : {
            "duration_in_millis" : 5,
            "in" : 3954,
            "out" : 3954
          },
          "name" : "mutate"
        }, {
          "id" : "a2610575a770fa30f963ef245925aa3ad64a04ec202f91b7285484e405cc9dd7",
          "events" : {
            "duration_in_millis" : 4922,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "mutate"
        }, {
          "id" : "1d9655caa6fe7870fa79488ca29b8f1d05b4943568e0b76991fb3d68242b1c58",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 541,
            "out" : 541
          },
          "name" : "ruby"
        }, {
          "id" : "036a6af77827f1efc2e1729a1a6e10260bfddb89abcad62b60f02db795155d79",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 580,
            "out" : 580
          },
          "name" : "ruby"
        }, {
          "id" : "fdc8f27e72137021076685d1cee9849f01e7dcd1078a2bb12992051ffddbad4c",
          "events" : {
            "duration_in_millis" : 310,
            "in" : 31974,
            "out" : 31974
          },
          "name" : "ruby"
        }, {
          "id" : "5643655f1686d01f320ed36cc643986be060bad7f857bb6ddee54a3ef3459421",
          "events" : {
            "duration_in_millis" : 967,
            "in" : 11603,
            "out" : 11603
          },
          "name" : "translate"
        }, {
          "id" : "8993fad65c7c4219a26cac368690fc8bdded6b7000c3dcdcb44cd93ce4266752",
          "events" : {
            "duration_in_millis" : 1683,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "mutate"
        }, {
          "id" : "b0a3894eee185efc91a16c22aa68051eb4276fcbd822711e703ee94b39a500cc",
          "events" : {
            "duration_in_millis" : 379,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "translate"
        }, {
          "id" : "bea2026e09c53a9f8e1006c6ec3b56786538fbb119b323c39f0c62c42a12a34f",
          "matches" : 375328,
          "events" : {
            "duration_in_millis" : 23043,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "date"
        }, {
          "id" : "82d6948e83b13f06faad18c0560a8b043ba39b7115f5a4c6482426eca04ef331",
          "events" : {
            "duration_in_millis" : 733,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "mutate"
        }, {
          "id" : "46291df465ead2692ae1cd84a75514b262892e90730312274a7fa79a06350c4c",
          "events" : {
            "duration_in_millis" : 39,
            "in" : 13865,
            "out" : 13865
          },
          "name" : "translate"
        }, {
          "id" : "d872b595aec8b24b8ac5e189a0041a3e6d3d5d887849f900231354e693aef896",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "mutate"
        }, {
          "id" : "a545adf27ad4c5e3bfdc87019e438c5761ae4c51b3f9c29d8a95bd75e5fa208a",
          "events" : {
            "duration_in_millis" : 954,
            "in" : 91301,
            "out" : 91301
          },
          "name" : "mutate"
        }, {
          "id" : "e3e360b1be0f1b556ec49a133e172776a9ffbdc59197ee3559c5f3b698162e83",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 575,
            "out" : 575
          },
          "name" : "translate"
        }, {
          "id" : "116c62c382153da3a36e426d1c8aa11d59adba1046dbeb072d5cac66f714c2cb",
          "events" : {
            "duration_in_millis" : 6072,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "mutate"
        }, {
          "id" : "f44e16af60f0f49b99993bdd007129b9c09d2a9ad5cb9b754ca8cae28eeffc5f",
          "events" : {
            "duration_in_millis" : 131,
            "in" : 13865,
            "out" : 13865
          },
          "name" : "mutate"
        }, {
          "id" : "88b4629fccb20bf51e9a508af467a53266d67e480df3e1e592e7e2fd69f6946d",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "mutate"
        }, {
          "id" : "66aa1d8761e831b0c333ef3a5afd648b0c5736f21c7d44fec1f9ff6867c3c804",
          "events" : {
            "duration_in_millis" : 2,
            "in" : 3954,
            "out" : 3954
          },
          "name" : "translate"
        }, {
          "id" : "0d9720b5751ce434d6abbef978b55de585df8b131e30ac47ed3f90953489ed6f",
          "events" : {
            "duration_in_millis" : 1,
            "in" : 6227,
            "out" : 6227
          },
          "name" : "ruby"
        }, {
          "id" : "619b3f4f1a5c39bf752fbeed6fc1be7e9edd0277addfca89d72d4ddd7ede81e7",
          "events" : {
            "duration_in_millis" : 131027,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "csv"
        }, {
          "id" : "143430914bfc705011dbcfa2075154da44eb7e48366cc44a70422b75a6ebf8c2",
          "events" : {
            "duration_in_millis" : 33337,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "ruby"
        }, {
          "id" : "76b4e0aaac566c94aade2b44ebb38560955be0dcaa5ded02e169691814fed39c",
          "events" : {
            "duration_in_millis" : 50,
            "in" : 11642,
            "out" : 11642
          },
          "name" : "translate"
        }, {
          "id" : "d50f8e0ef5f089eaad1a4612a574501892d8eb9c6a04e946d536cbe00179169e",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 4,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "77ee44ef0c01c5d1baff2b05039f5e89912b65b4e819e42ab26ce71d05a093eb",
          "events" : {
            "duration_in_millis" : 98,
            "in" : 18143,
            "out" : 18143
          },
          "name" : "ruby"
        }, {
          "id" : "bccf832b5473cff0baf079061e1ef272745da961f176bd6a2782d103c0aa5494",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 3954,
            "out" : 3954
          },
          "name" : "mutate"
        }, {
          "id" : "cae3962075f04bdcacf17000edb653ef05bc1506a128d724a8483264bef4b4e7",
          "events" : {
            "duration_in_millis" : 5422,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "mutate"
        }, {
          "id" : "243435b5cea391637d11ef340d6e6483889441e7d85d3093464591b5816fc0b2",
          "events" : {
            "duration_in_millis" : 1,
            "in" : 3272,
            "out" : 3272
          },
          "name" : "ruby"
        }, {
          "id" : "794f70e5d1add583d5cd87edccaf806565a622b7dc22aff30e8406745e6aed18",
          "events" : {
            "duration_in_millis" : 38432,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "ruby"
        }, {
          "id" : "236bab924f6228e6b37880b27bed467de6d8a5c07ff1c25a49e6f2a1ee3a338d",
          "events" : {
            "duration_in_millis" : 12,
            "in" : 4,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "fbfb62c3bf5d186d88bc223fb19d465e76b0de93f15293d3be224e97c061a6c9",
          "events" : {
            "duration_in_millis" : 800,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "translate"
        }, {
          "id" : "8d0228381de681a0b2dd14a50fbfdfd502fccc4dd01b72173d6f3226012f91b7",
          "events" : {
            "duration_in_millis" : 20516,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "ruby"
        }, {
          "id" : "3fbdf1cc8f5045099fc5a05e1edf0bcaf36afef95c392fdd005a423896da61ee",
          "events" : {
            "duration_in_millis" : 45,
            "in" : 10479,
            "out" : 10479
          },
          "name" : "translate"
        }, {
          "id" : "a528f8f555a27447b5b29e33a0c1d3372295ee2fce005849aa1b2734a23479a2",
          "events" : {
            "duration_in_millis" : 6,
            "in" : 10479,
            "out" : 10479
          },
          "name" : "mutate"
        }, {
          "id" : "ee1cc532779ccee1b27ec7bc5540c513c64b9920f729e8cd72bb7974766fb678",
          "events" : {
            "duration_in_millis" : 504,
            "in" : 73518,
            "out" : 73518
          },
          "name" : "ruby"
        }, {
          "id" : "ab1ade4fecec0d3d57ad4f27ba229de38dc2d48656290fed296880470fe2ef34",
          "events" : {
            "duration_in_millis" : 776,
            "in" : 90542,
            "out" : 90542
          },
          "name" : "translate"
        }, {
          "id" : "8bf0df845a8c74fe595a032f7d8fb214e7b0b3433c9caa521ead66b5ec9a339b",
          "events" : {
            "duration_in_millis" : 672,
            "in" : 71416,
            "out" : 71416
          },
          "name" : "ruby"
        }, {
          "id" : "5c7c083f7d830f942d365bc4e81eb5d7ab6ef58f2bf7021a474f65267f8d93bd",
          "events" : {
            "duration_in_millis" : 10511,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "ruby"
        }, {
          "id" : "d075d6b09b49fa3304b63d28528659879cc17e371baedc4daf020667d4e5699b",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 32,
            "out" : 32
          },
          "name" : "translate"
        }, {
          "id" : "2b89721a62472aed7e673bcf36636b04bd5b95d80611c556fa8abb2976664485",
          "events" : {
            "duration_in_millis" : 42,
            "in" : 13865,
            "out" : 13865
          },
          "name" : "translate"
        }, {
          "id" : "d9330148e935bbfe37aa1ff1a3b77e6210e67f97a280a98c1f2f486bcb9ffee5",
          "events" : {
            "duration_in_millis" : 745,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "translate"
        }, {
          "id" : "fc044a4cae592fbb0fad0c7cc94b08c0d43ac3f211ec5438516c24e38dc7e18d",
          "events" : {
            "duration_in_millis" : 222,
            "in" : 42997,
            "out" : 42997
          },
          "name" : "translate"
        }, {
          "id" : "be8e02f092b6c65647e1b1da3d9fdd2d49b47a95d0bb41db4f1116052d82e502",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 72,
            "out" : 72
          },
          "name" : "translate"
        }, {
          "id" : "3cc261ab2b891327e06bd15bbbe99306665d987db88e8038c73ca860d6ac15bb",
          "events" : {
            "duration_in_millis" : 53,
            "in" : 10479,
            "out" : 10479
          },
          "name" : "mutate"
        }, {
          "id" : "ddc4a948410b7fc476f9fb0865efb336e9fe3ae0807499b0f30755b36265b199",
          "events" : {
            "duration_in_millis" : 1639,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "mutate"
        }, {
          "id" : "6537a08234c1f530c14cdc3c642fece164dcedb27d152fea5b9a1093eb7764e6",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 220,
            "out" : 220
          },
          "name" : "ruby"
        }, {
          "id" : "e9918b568cf0b72485ffb59b8f5fa80d31af24dbb0481f8bb6437899b9dedf6c",
          "events" : {
            "duration_in_millis" : 670,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "translate"
        }, {
          "id" : "6b836885e310600a617e4177ed1778541a2262dc8c91e5b99a7b01c672c9db17",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "179b094debaf57d12bf2cbaf52443110705b678e2b5c1917eb1a8cc4342e128f",
          "events" : {
            "duration_in_millis" : 432,
            "in" : 73507,
            "out" : 73507
          },
          "name" : "mutate"
        }, {
          "id" : "322113022403c723cba85b560b40e89710a0e052eaea091e0500ac18d8b32932",
          "events" : {
            "duration_in_millis" : 814,
            "in" : 110267,
            "out" : 110267
          },
          "name" : "ruby"
        }, {
          "id" : "61ef513021af45bfb54c9c5a47caf38d3f2dabd1d554a02c2fb190498df0e55b",
          "events" : {
            "duration_in_millis" : 11542,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "mutate"
        }, {
          "id" : "7a6de7d6f3e91899c1b0179e1f9c47a8449de4f1ec450cfe595cb86c8c4ad571",
          "events" : {
            "duration_in_millis" : 37,
            "in" : 13865,
            "out" : 13865
          },
          "name" : "mutate"
        }, {
          "id" : "ecaa1001dcc63f796e1e4eebccd584322322a8c2a6470c4551ab2bc110ffc3ba",
          "events" : {
            "duration_in_millis" : 92,
            "in" : 31448,
            "out" : 31448
          },
          "name" : "translate"
        }, {
          "id" : "62415d506fcb46e963aed2808ef0271f77a1f1fb9879207f26fa9fbdca189b9e",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "24e291fc17c92b1efd77748b8a1eb0981e3f7f0ed5ba635753df245cc4f48fb4",
          "events" : {
            "duration_in_millis" : 3730,
            "in" : 218973,
            "out" : 218973
          },
          "name" : "mutate"
        }, {
          "id" : "f75b5b4602ab93079ddd1a3af93f85ec46208b5ae4398799e1bd20ea67e515b4",
          "events" : {
            "duration_in_millis" : 941,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "mutate"
        }, {
          "id" : "251b8f508132bf3b6daac0774fe1c563b891bdeadcbf3399062b94602fb69043",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 2745,
            "out" : 2745
          },
          "name" : "ruby"
        }, {
          "id" : "30eff25023b00a2a66d5c12f16ef6bbca1a05c2606a1bb83e496ab45884c27ef",
          "events" : {
            "duration_in_millis" : 3239,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "mutate"
        }, {
          "id" : "a4f4e5a6ff390b5df16da9938b8c34fe696f872c955572de15d0ddbc1e8aa928",
          "events" : {
            "duration_in_millis" : 2,
            "in" : 3954,
            "out" : 3954
          },
          "name" : "translate"
        }, {
          "id" : "d21b07fc3b02f346bc5d07b0bd034080da4d5ae11d45f18b874bca79ce8ac5d7",
          "events" : {
            "duration_in_millis" : 322,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "translate"
        }, {
          "id" : "e98ca0aded2b98e324eed07b4cc1494c1b0ae03f9a68f6d70d670165d3290302",
          "events" : {
            "duration_in_millis" : 670,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "mutate"
        }, {
          "id" : "02ea936e0b9352f323b07c60418abd62a6ac9554d130f8ef15449466676e7d10",
          "events" : {
            "duration_in_millis" : 104,
            "in" : 10479,
            "out" : 10479
          },
          "name" : "mutate"
        }, {
          "id" : "a796c17ff8917da2100c5813371c812ecf8cf3c8cf4cfae2aaa6ce8350f2b67b",
          "events" : {
            "duration_in_millis" : 16,
            "in" : 3954,
            "out" : 3954
          },
          "name" : "mutate"
        }, {
          "id" : "904151d7b62cea8bf587ceb81ad6bd8f655c209370f340cb108a861b83da055c",
          "events" : {
            "duration_in_millis" : 10,
            "in" : 1,
            "out" : 0
          },
          "name" : "drop"
        }, {
          "id" : "bfea11244612c88067f9133e902a66396113dc3f9c80590d1e4315acc927c748",
          "events" : {
            "duration_in_millis" : 65,
            "in" : 6991,
            "out" : 6991
          },
          "name" : "ruby"
        }, {
          "id" : "316de99ab45272bcadd189e497ac15be29cedd190758900fa9f7643a60d576cd",
          "events" : {
            "duration_in_millis" : 1628,
            "in" : 90542,
            "out" : 90542
          },
          "name" : "ruby"
        }, {
          "id" : "3a3bc1e6134ef323a1d3bde833ea46bcc0de104d829fd5addfe738ea4cecbc30",
          "events" : {
            "duration_in_millis" : 3709,
            "in" : 218973,
            "out" : 218973
          },
          "name" : "mutate"
        }, {
          "id" : "167693f428317ca23ccae115069cabd6003b17e449bb41e189226facd58a8ee5",
          "events" : {
            "duration_in_millis" : 751,
            "in" : 115305,
            "out" : 115305
          },
          "name" : "translate"
        }, {
          "id" : "0a3237b29ceea8b4b0e77364911873fe4df6be0a9f06374a086676f6a8038aad",
          "events" : {
            "duration_in_millis" : 144,
            "in" : 13865,
            "out" : 13865
          },
          "name" : "mutate"
        }, {
          "id" : "354be9128e71ef57ba65bda163ec748428f6fe59cbe3ae7e400730d7f833d781",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "translate"
        }, {
          "id" : "98fde9847e0f70e665f221e910769fc59c6f2b48340bf86c23f9a5abfa3ecaaa",
          "events" : {
            "duration_in_millis" : 411,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "translate"
        }, {
          "id" : "7bd845293eb826b79e3555cd23854e19d9e77723f75b4f019fbf4b2ae0185eb8",
          "events" : {
            "duration_in_millis" : 0,
            "in" : 0,
            "out" : 0
          },
          "name" : "mutate"
        }, {
          "id" : "2a01f36ceaa63e607f6e787a433999aec8e2d2cb13e6edfa3a3c0c55f6d76880",
          "events" : {
            "duration_in_millis" : 308,
            "in" : 89297,
            "out" : 89297
          },
          "name" : "translate"
        } ],
        "outputs" : [ {
          "id" : "c896caf91f9cc208e25e3dbb9a6a31e9510df37bfb3f73c58bcd612e19a8a8f7",
          "events" : {
            "duration_in_millis" : 3984257,
            "in" : 375328,
            "out" : 375328
          },
          "name" : "stdout"
        } ]
      },
      "reloads" : {
        "successes" : 0,
        "failures" : 0,
        "last_failure_timestamp" : null,
        "last_error" : null,
        "last_success_timestamp" : null
      },
      "queue" : {
        "type" : "memory",
        "events_count" : 0,
        "queue_size_in_bytes" : 0,
        "max_queue_size_in_bytes" : 0
      },
      "hash" : "1781769fd2896d41c637d3ea4b2ca07409170e50ab6055f01ad4f21c4d23e0c3",
      "ephemeral_id" : "e51906cc-d2f9-4423-8840-673a6e07e6f0"
    }
  }
bash-4.2$ curl -XGET 'localhost:9600/_node/stats/events?pretty'
{
  "host" : "9da12cd83ec0",
  "version" : "7.16.3",
  "http_address" : "0.0.0.0:9600",
  "id" : "8271ea02-bf31-45df-96b8-7e182b60f2eb",
  "name" : "9da12cd83ec0",
  "ephemeral_id" : "a870f636-ad60-49e6-a99d-05417e49ebc3",
  "status" : "green",
  "snapshot" : false,
  "pipeline" : {
    "workers" : 16,
    "batch_size" : 2000,
    "batch_delay" : 50
  },
  "events" : {
    "in" : 375341,
    "filtered" : 375341,
    "out" : 375341,
    "duration_in_millis" : 4356256,
    "queue_push_duration_in_millis" : 214073
  }
}bash-4.2$ curl -XGET 'localhost:9600/_node/stats/reloads?pretty'
{
  "host" : "9da12cd83ec0",
  "version" : "7.16.3",
  "http_address" : "0.0.0.0:9600",
  "id" : "8271ea02-bf31-45df-96b8-7e182b60f2eb",
  "name" : "9da12cd83ec0",
  "ephemeral_id" : "a870f636-ad60-49e6-a99d-05417e49ebc3",
  "status" : "green",
  "snapshot" : false,
  "pipeline" : {
    "workers" : 16,
    "batch_size" : 2000,
    "batch_delay" : 50
  },
  "reloads" : {
    "successes" : 0,
    "failures" : 0
  }
2

Can you explain this? Does CSV have a column name header or you set manually? How many fields you have? Please provide more info about CSV structure.

3

yes it was put as permanent in the pipeline

csv {
		separator => ";"
	    
		columns => ["file_name",  "type", "event", ...]
		 skip_empty_columns => true
4

Have you try to set to false? You will have all columns.

I would recommend:

  • As I remember, when I had stress tests - XML processing, 6 or 8 GB of RAM was enough for 100 GB file. Don't expect 1-2 GB to process 50 - 100 GB. Also test will the persistent queue have some memory benefits.

  • Parameters:
    Workers - set as number of logical CPUs
    Pipeline Batch Size - try to incease
    Queue Max Bytes - increase to 2-4 GB

  • Disable all debug modes, do not use codec => rubydebug{} in output , log.level: info, config.debug: false, config.reload.automatic: false

Test everything then set as the final/production values.

5

I found the root cause
when while contains such many of data rows and file has size ~70MB logstash. File does not load the entire file only part of it, what restrictions are with defaults set what should expand. If it does not load the whole file exceptions appear and therefore the file is not matched with the pattern.
It's also worth to mention that such files contains over 96000 of events.

Should I increase number of workers?

CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 32000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 2000), or changing the number of pipeline workers (currently 16) {:pipeline_id=>"main", :thread=>"#<Thread:0x42750374 run>"}

http.host: "0.0.0.0"
pipeline.workers: 16
pipeline.batch.size: 2000
pipeline.batch.delay: 50
config.reload.automatic: true
xpack.monitoring.enabled: false
xpack.management.pipeline.id: ["main"]

How we can tune up above parameters due to server architecture

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2

what are the relationships and rules here?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.