Limited output from logstash

Hi
I'm wondering if it's possible to limited output event form logstash?

I've observed many of circuit_breaking_exception

:error=>{"type"=>"circuit_breaking_exception", "reason"=>"[parent] Data too large, data for [indices:data/write/bulk[s]] would be [8166539722/7.6gb], which is larger than the limit of [8160437862/7.5gb], real usage: [8166244216/7.6gb], new bytes reserved: [295506/288.5kb], usages [fielddata=1443244/1.3mb, request=5013504/4.7mb, inflight_requests=295506/288.5kb, model_inference=0/0b, eql_sequence=0/0b]", "bytes_wanted"=>8166539722, "bytes_limit"=>8160437862, "durability"=>"TRANSIENT"}}

:error=>{"type"=>"circuit_breaking_exception", "reason"=>"[parent] Data too large, data for [indices:data/write/bulk[s]] would be [8448556430/7.8gb], which is larger than the limit of [8160437862/7.5gb], real usage: [8448288200/7.8gb], new bytes reserved: [268230/261.9kb], usages [request=8028272/7.6mb, fielddata=238005396/226.9mb, eql_sequence=0/0b, model_inference=0/0b, inflight_requests=282664/276kb]", "bytes_wanted"=>8448556430, "bytes_limit"=>8160437862, "durability"=>"PERMANENT"}}

name                   heap.current heap.percent heap.max
es_data_hdd_5_3               3.2gb           81      4gb
es_data_hdd_7_3               3.1gb           78      4gb
es_data_ssd_1_1               6.5gb           82      8gb
es_data_hdd_1_3               3.1gb           79      4gb
es_master_2_1                 4.9gb           61      8gb
es_data_ssd_3_3               6.7gb           84      8gb
es_data_ssd_3_1               6.6gb           83      8gb
es_master_1_3                 3.9gb           49      8gb
es_data_hdd_6_1                 3gb           75      4gb
es_data_ssd_4_3               6.7gb           84      8gb
es_data_ssd_1_3                 7gb           87      8gb
es_data_hdd_9_1                 3gb           77      4gb
es_data_ssd_1_2                 7gb           87      8gb
es_data_hdd_1_1               3.3gb           83      4gb
es_data_hdd_7_1               3.2gb           81      4gb
es_data_ssd_5_1               7.3gb           92      8gb
es_data_ssd_2_2               7.5gb           93      8gb
es_data_hdd_4_1                 3gb           77      4gb
es_master_1_2                 2.1gb           26      8gb
es_data_hdd_9_3               2.6gb           66      4gb
es_data_hdd_3_3                 3gb           76      4gb
es_data_hdd_7_2                 3gb           77      4gb
es_data_hdd_3_2               3.2gb           82      4gb
es_data_hdd_6_2               3.1gb           78      4gb
es_data_hdd_6_3               3.3gb           82      4gb
es_data_hdd_3_1               3.2gb           81      4gb
es_data_ssd_5_3               6.6gb           83      8gb
es_data_hdd_4_2               3.4gb           85      4gb
es_data_hdd_2_3               3.3gb           83      4gb
es_data_ssd_3_3_ingest          1gb           25      4gb
es_data_hdd_4_3               2.9gb           74      4gb
es_data_hdd_2_1               3.2gb           81      4gb
es_master_1_1                 3.9gb           49      8gb
es_data_hdd_8_3               2.3gb           59      4gb
es_data_ssd_2_3               6.7gb           84      8gb
es_data_ssd_3_1_ingest        1.5gb           39      4gb
es_data_ssd_4_1               7.2gb           90      8gb
es_data_ssd_4_2               7.1gb           89      8gb
es_data_hdd_5_2               2.9gb           73      4gb
es_data_ssd_5_2               6.9gb           87      8gb
es_data_hdd_1_2               3.2gb           81      4gb
es_data_hdd_5_1               2.9gb           74      4gb
es_data_hdd_2_2               3.2gb           80      4gb
es_data_hdd_9_2                 3gb           75      4gb
es_data_ssd_3_2               6.9gb           86      8gb
es_master_2_2                 3.7gb           47      8gb
es_master_2_3                 5.3gb           67      8gb
es_data_ssd_3_2_ingest        2.7gb           68      4gb
es_data_ssd_2_1               6.8gb           85      8gb
es_data_hdd_8_1               2.5gb           64      4gb
es_data_hdd_8_2               2.7gb           68      4gb

Sometimes you want to filter or output an event only under certain conditions. For that, you can use a conditional.

if EXPRESSION {
  ...
} else if EXPRESSION {
  ...
} else {
  ...
}

I meant about it if logstash can slice data for peaces for example if now he can hit the data with over 105Mb in on bulk , so if it's possible to limit to 50Mb?

Why do you need to pick the data and again restrict it at logstash.

Rather than doing that, why can't you restrict the data you do not want at the source itself by filtering or excluding data.

all the data are desired I don't want to truncation content of data

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.