# Increase Size Limit of logs

**URL:** <https://discuss.elastic.co/t/increase-size-limit-of-logs/80496>\
**Category:** Logstash\
**Created:** [March 29, 2017, 2:34pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496 "2017-03-29T14:34:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![AlecBruns](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@AlecBruns](https://discuss.elastic.co/u/AlecBruns)\
**Post date:** [March 29, 2017, 2:34pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/1 "2017-03-29T14:34:31Z")

</div>

Hey there,

Is it possible to increase the size limit of incoming logs to logstash? I am using filebeat but logstash will break up large logs which then makes the grok filter unable to parse it. I am trying to log some big xml response bodies so being able to increase the limit for logstash would be great! Thanks!

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 31, 2017, 4:01pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/2 "2017-03-31T16:01:26Z")

</div>

Hello @AlecBruns,

By default Logstash should not break anything that was sent to him by Filebeat, but it will send 1 event per line in your log, is your XML document is made of multiple lines?

Can you add log sample to this thread and your filebeat configuration?

---

<div class="post-metadata">

**Author:** ![AlecBruns](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@AlecBruns](https://discuss.elastic.co/u/AlecBruns)\
**Post date:** [April 3, 2017, 3:11pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/3 "2017-04-03T15:11:45Z")

</div>

Hi @pierhugues , I cannot share log since it is sensitive info but the log is on one line. File beat is picking up the log that is being made by nginx. As far as I know nginx doesn't log to multiple lines. The log format for nginx is:

```
log_format main '$remote_addr $status $request_time $upstream_header_time $upstream_response_time'
    ' $request $body_bytes_sent '
     '$http_user_agent $soap_action $request_body';

```

This is my filebeat conf:

```
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/*.log
  document_type: nginx-access
output:
  logstash:
    enabled: true
    hosts:

```

Thank you

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [April 7, 2017, 1:29pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/4 "2017-04-07T13:29:26Z")

</div>

I cannot find anything that would make LS break large log, maybe you see newline in the soap xml object?

Instead of using grok to parse the content, did you try to use the [logstash-filter-xml](https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html) , You can use xpath to extract the data from the xml.

---

<div class="post-metadata">

**Author:** ![AlecBruns](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@AlecBruns](https://discuss.elastic.co/u/AlecBruns)\
**Post date:** [April 11, 2017, 7:14pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/5 "2017-04-11T19:14:16Z")

</div>

Hmm, I don't believe it is being split into multiple lines but I can investigate it further.

However, the logs have extra meta data from nginx so the xml filter can't be used to filter all the data.

Thank you for all the help thus far though.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [April 11, 2017, 7:17pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/6 "2017-04-11T19:17:08Z")

</div>

Maybe writing the events to a file using the file output will give us some idea what is going on here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 9, 2017, 7:21pm UTC](https://discuss.elastic.co/t/increase-size-limit-of-logs/80496/7 "2017-05-09T19:21:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.