# Files too big for Sentinel plugin

**URL:** <https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530>\
**Category:** Logstash\
**Created:** [December 4, 2023, 8:37am UTC](https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530 "2023-12-04T08:37:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph\_Leiber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_leiber/32/128584_2.png) [@Joseph\_Leiber](https://discuss.elastic.co/u/Joseph_Leiber)\
**Post date:** [December 4, 2023, 8:37am UTC](https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530/1 "2023-12-04T08:37:04Z")

</div>

Hi Logstash Experts -

This is my first time dealing with Logstash, so I'm not quite sure why the logs are being formatted like this, whether this is expected/normal, or how to handle them.

I'm hitting an issue with logs from Artifactory -\> Fluent Bit -\> Logstash. The main issue is that the log files are too big to pass to the Azure Sentinel plugin - I'm getting the error

```auto
[ERROR][logstash.outputs.microsoftsentineloutput][artifactory][...] Received document above the max allowed size - dropping the document [document size: 3920975, max allowed size: 1036576

```

Is there some way to split the logs into smaller files? Is this possible with Logstash, or does it need to be done before the logs are sent from Fluent Bit?

My config is very simple:

```auto
input {
  tcp {
    port => 8084
  }
}

output {
    s3 {
        region => "ap-northeast-1"
        bucket => "MY_BUCKET"       
        prefix => "artifactory-logs/%{+YYYY}/%{+MM}/%{+dd}"
        time_file => 5
        additional_settings => {
            "force_path_style" => true
            "follow_redirects" => false
        }
        codec => json
    }

    file {
      path => "/var/log/artifactory_test.log"
      write_behavior => "overwrite"
    }

    microsoft-sentinel-logstash-output-plugin {
      client_app_Id => MY_ID
      client_app_secret => MY_SECRET
      tenant_id => MY_TENANT_ID
      data_collection_endpoint => MY_ENDPOINT
      dcr_immutable_id => MY_DCR
      dcr_stream_name => MY_STREAM
    }
}

```

Each log file contains tons of entries, and they're grouped together in an odd way, with square brackets and no delimiters. Each pair of square brackets contains an arbitrary number of log entries separated by curly braces and commas. All quotation marks are escaped.

```auto
[{"message":"dummy data","purpose":"testing","log_server":"splunk"},{"message":"dummy data","purpose":"testing","log_server":"splunk"},{"message":"dummy data","purpose":"testing","log_server":"splunk"},{"message":"dummy data","purpose":"testing","log_server":"splunk"}]
[{"message":"dummy data","purpose":"testing","log_server":"splunk"}]

```

---

<div class="post-metadata">

**Author:** ![Joseph\_Leiber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_leiber/32/128584_2.png) [@Joseph\_Leiber](https://discuss.elastic.co/u/Joseph_Leiber)\
**Post date:** [December 13, 2023, 11:41pm UTC](https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530/2 "2023-12-13T23:41:01Z")

</div>

This is resolved - turned out the issue was that Fluent Bit was sending us logs in `json` format. We fixed the issue by changing the format to `json_lines`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2024, 11:41pm UTC](https://discuss.elastic.co/t/files-too-big-for-sentinel-plugin/348530/3 "2024-01-10T23:41:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.