# Splitting for Logstash working intermittently

**URL:** <https://discuss.elastic.co/t/splitting-for-logstash-working-intermittently/318280>\
**Category:** Logstash\
**Created:** [November 6, 2022, 11:51pm UTC](https://discuss.elastic.co/t/splitting-for-logstash-working-intermittently/318280 "2022-11-06T23:51:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![eleong](https://avatars.discourse-cdn.com/v4/letter/e/7cd45c/32.png) [@eleong](https://discuss.elastic.co/u/eleong)\
**Post date:** [November 6, 2022, 11:51pm UTC](https://discuss.elastic.co/t/splitting-for-logstash-working-intermittently/318280/1 "2022-11-06T23:51:15Z")

</div>

Hi,

Currently, I have Logstash configured for splitting. The issue is that, sometimes it works, sometimes it goes about 10-15 minutes without any output (it suppose to have an output every 5 minutes).

At the backend, I have scheduled a cronjob to call to an API and then write to a file (JSON format). Logstash will then in turn, read the file and split "result" field.

Here's my Logstash code:

```auto
input {
 file {
   start_position => "end"
   path => ["/var/log/logstash/abc_api-*.json"]
   sincedb_path => "/dev/null"
 }
#stdout { codec => rubydebug { metadata => true } }
}

filter {
 json {
   source => "message"
 }
 split {
   field => "result"
 }
#stdout { codec => rubydebug { metadata => true } }
}

output {
  elasticsearch {
  ssl => true
  ssl_certificate_verification => false
  cacert => "/etc/logstash/elasticsearch-ca.pem"
  hosts => "https://10.0.0.2:9200"
  user => "${LS_USER}"
  password => "${LS_PWD}"
  manage_template => true
  index => "abc-logs-%{+YYYY.MM.dd}"
  pipeline => "abc-api"
  }
#stdout { codec => rubydebug { metadata => true } }
 }

```

I've searched the forum and I **think** this [post](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/10) helps. But I don't fully understand the "splitData.rb" code, hence, it's not implemented. It seems to be removing the "message" field but I could be wrong.

Also, I have configured Logstash to have 4GB of heap memory. In the array file, it has about 500 items for now (will grow to ~5000). Usually I use Filebeat for all things Elastic (its less resource intensive), but Filebeat does not support splitting feature. So, this is an exception.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 7, 2022, 12:01am UTC](https://discuss.elastic.co/t/splitting-for-logstash-working-intermittently/318280/2 "2022-11-07T00:01:03Z")

</div>

> [@eleong](#):
>
> I **think** this [post](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/10) helps.

No. That code was only needed because there was an issue in the split filter that caused excessive memory use. That issue has since been fixed.

Have you confirmed that there are files that match the regexp that have not been read by logstash?

---

<div class="post-metadata">

**Author:** ![eleong](https://avatars.discourse-cdn.com/v4/letter/e/7cd45c/32.png) [@eleong](https://discuss.elastic.co/u/eleong)\
**Post date:** [November 7, 2022, 2:08am UTC](https://discuss.elastic.co/t/splitting-for-logstash-working-intermittently/318280/3 "2022-11-07T02:08:49Z")

</div>

Hi Badger,

The API calls always output the same JSON pattern. From the picture below, there is a gap, but I'm not sure what happened. I'll further monitor this. Just your confirmation on the memory usage fix is already good enough since I can rule that out from my troubleshooting.

Perhaps will work on the pipeline a little. Right now, in Elastic, each document @timestamp is based on the time it was ingested. Maybe I can change it to take the timestamp directly from API query instead since within the API result, there is a timestamp value.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fc3a8eb94409588c7b21981cc0dc797e3663c765.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2022, 2:09am UTC](https://discuss.elastic.co/t/splitting-for-logstash-working-intermittently/318280/4 "2022-12-05T02:09:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.