# Logstash with elasticsearch input and output keep looping results

**URL:** <https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031>\
**Category:** Logstash\
**Created:** [January 11, 2018, 7:43am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031 "2018-01-11T07:43:53Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [January 11, 2018, 7:43am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/1 "2018-01-11T07:43:54Z")

</div>

I would like to reindex and filter my log again. What I get the information from Internet is using the logstash to filter the data again. I tried and it can really split my data into different fields, however, the data keeps looping. That is, I have 100,000 log but after filter and output to elasticsearch, I found that more than 100,000 log output into elasticsearch and the log are deplicated. Does anyone have idea on that?

Moreover, I receive below log when running logstash, although it said that error phasing JSON, I found that the log can still be filtered. Why would be like that? Thank you!

**Here is my logstash config:**

```
input {
      elasticsearch {
      hosts => "10.0.132.56"
      index => "logstash-2018.01.04"
      }
}
filter{
     grok {
            match => {"message" => "%{TIMESTAMP_ISO8601:logdate} %{GREEDYDATA:vmname} %{GREEDYDATA:message}"}
            overwrite => ["message"]
     }
}
filter {
       json {
            source => "scrmsg"
       }
}
output {
       elasticsearch {
            hosts => ["10.0.132.64:9200"]
            manage_template => false
            index => "logstash-2018.01.04-1"
     }
}

```

**Here is the error log:**

```
[2018-01-11T15:15:32,010][WARN][logstash.filters.json] Error parsing json {:source=>"scrmsg", :raw=>"Trident/5.0)\",\"geoip_country\":\"US\",\"allowed\":\"1\",\"threat_score\":\"268435456\",\"legacy_unique_id\":\"\",\"cache_status\":\"-\",\"informed_id\":\"\",\"primitive_id\":\"2BC2D8AD-7AD0-3CAD-9453-B0335F409701\",\"valid_ajax\":\"0\",\"orgin_response_time\":\"0.081\",\"request_id\":\"cd2ae0a8-0921-48b6-b03f-15c71a55100b\",\"bytes_returned_origin\":\"83\",\"server_ip\":\"10.0.10.16\",\"origin_status_code\":\"\",\"calculated_pages_per_min\":\"1\",\"calculated_pages_per_session\":\"1\",\"calculated_session_length\":\"0\",\"k_s\":\"\",\"origin_address\":\"10.0.10.16:443\",\"request_protocol\":\"https\",\"server_serial\":\"5c3eb4ad-3799-4bd8-abb2-42edecd54b99\",\"nginx_worker_process\":\"19474\",\"origin_content_type\":\"application/json;charset=UTF-8\",\"lb_request_time\":\"\",\"SID\":\"\",\"geoip_org\":\"Drake Holdings LLC\",\"accept\":\"*/*\",\"accept_encoding\":\"gzip, deflate\",\"accept_language\":\"\",\"connection\":\"Keep-Alive\",\"http_request_length\":\"418\",\"real_ip_header_value\":\"204.79.180.18\",\"http_host\":\"www.honeyworkshop.com\",\"machine_learning_score\":\"\",\"HSIG\":\"ALE_UHCF\",\"ZID\":\"\",\"ZUID\":\"\",\"datacenter_id\":\"363\",\"new_platform_domain_id\":\"3063fc0b-5b48-4413-9bc7-600039caf64c\",\"whitelist_score\":\"0\",\"billable\":\"1\",\"distil_action\":\"@proxy\",\"js_additional_threats\":\"\",\"js_kv_additional_threats\":\"\",\"re_field_1\":\"\",\"re_field_2\":\"\",\"re_field_3\":\"\",\"http_accept_charset\":\"\",\"sdk_token_id\":\"\",\"sdk_application_instance_id\":\"\",\"per_path_calculated_pages_per_minute\":\"1\",\"per_path_calculated_pages_per_session\":\"1\",\"path_security_type\":\"api\",\"identification_provider\":\"web\",\"identifier_record_pointer\":\"\",\"identifier_record_value\":\"\",\"path_rule_scope_id\":\"\",\"experiment_id\":\"0\",\"experiment_score\":\"\",\"experiment_group_id\":\"\",\"experiment_auxiliary_string\":\"\",\"type\":\"distil\"}\n", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Trident': was expecting ('true', 'false' or 'null') at [Source: (byte[])"Trident/5.0)","geoip_country":"US","allowed":"1","threat_score":"268435456","legacy_unique_id":"","cache_status":"-","informed_id":"","primitive_id":"2BC2D8AD-7AD0-3CAD-9453-B0335F409701","valid_ajax":"0","orgin_response_time":"0.081","request_id":"cd2ae0a8-0921-48b6-b03f-15c71a55100b","bytes_returned_origin":"83","server_ip":"10.0.10.16","origin_status_code":"","calculated_pages_per_min":"1","calculated_pages_per_session":"1","calculated_session_length":"0","k_s":"","origin_address":"10.0.10"[truncated 1180 bytes]; line: 1, column: 9]>}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [January 11, 2018, 8:05am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/2 "2018-01-11T08:05:40Z")

</div>

Please format your code using `</>` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21). It will make your post more readable.

Or use markdown style like:

````
```
CODE
```

````

I moved your question to #logstash

---

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [January 11, 2018, 8:29am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/3 "2018-01-11T08:29:36Z")

</div>

Thanks for reminder!!

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 11, 2018, 9:52am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/4 "2018-01-11T09:52:33Z")

</div>

It looks like the value of the `scrmsg` is not valid JSON.  
It starts without a leading `{` bracket.

---

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [January 12, 2018, 4:41am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/5 "2018-01-12T04:41:16Z")

</div>

Thank you for your reminder. I have found that some log was error phasing by JSON but some are not. I correct the logstash indexer again. But will it affect the elasticsearch loops the index input?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 12, 2018, 3:04pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/6 "2018-01-12T15:04:14Z")

</div>

By looping, I think you mean that the ES input re-reads the docs that the ES output adds to ES. Yes?

If so, I thought that as the input and output are using different indexes it should not loop.

@Christian_Dahlqvist - please comment on this ^.

---

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [January 15, 2018, 2:28am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/7 "2018-01-15T02:28:48Z")

</div>

Yes, it re-reads the docs. I already use different index name, and the host is different too.

---

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [January 26, 2018, 1:14am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/8 "2018-01-26T01:14:40Z")

</div>

Anyone can help? 😫

---

<div class="post-metadata">

**Author:** ![OvBalaban](https://avatars.discourse-cdn.com/v4/letter/o/d2c977/32.png) [@OvBalaban](https://discuss.elastic.co/u/OvBalaban)\
**Post date:** [January 28, 2018, 1:01pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/9 "2018-01-28T13:01:29Z")

</div>

I have the same issue with Logstash 6.1.1. Regardless if I use it on the same host or different hosts, Logstash loops until stopped.

I didn't use Kibana, I used curl directly on the Elasticsearch indices.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 29, 2018, 3:23pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/10 "2018-01-29T15:23:25Z")

</div>

@layla

Please try with very different index names (no possible overlap) e.g. ES input `logstash-2018.01.04` and ES output `logstash-1-2018.01.04`. Does it still loop?

---

<div class="post-metadata">

**Author:** ![OvBalaban](https://avatars.discourse-cdn.com/v4/letter/o/d2c977/32.png) [@OvBalaban](https://discuss.elastic.co/u/OvBalaban)\
**Post date:** [January 29, 2018, 4:02pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/11 "2018-01-29T16:02:54Z")

</div>

@guyboertje my indexes are as different as notag\_anxl00-18 and anxl-0018 and I still have the looping issue.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 29, 2018, 5:26pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/12 "2018-01-29T17:26:15Z")

</div>

I asked one of our Consultants and they said...

> I used that approach a while ago and did not have issues. One thing that comes to mind is to increase the `scroll` time to a higher value. Perhaps logstash can't process the events fast enough and starts over?

`scroll` defaults to "1m".

---

<div class="post-metadata">

**Author:** ![OvBalaban](https://avatars.discourse-cdn.com/v4/letter/o/d2c977/32.png) [@OvBalaban](https://discuss.elastic.co/u/OvBalaban)\
**Post date:** [January 29, 2018, 9:09pm UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/13 "2018-01-29T21:09:01Z")

</div>

Thank you, @guyboertje.

I have increased the scroll value to 5m for an ongoing operation. I'll let people know if this changes anything.

---

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [January 30, 2018, 2:02am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/14 "2018-01-30T02:02:08Z")

</div>

May I know how to set the scroll value? Thanks @guyboertje!

---

<div class="post-metadata">

**Author:** ![layla](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@layla](https://discuss.elastic.co/u/layla)\
**Post date:** [February 22, 2018, 3:59am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/15 "2018-02-22T03:59:16Z")

</div>

How's you result after tuning the scroll value? 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2018, 3:59am UTC](https://discuss.elastic.co/t/logstash-with-elasticsearch-input-and-output-keep-looping-results/115031/16 "2018-03-22T03:59:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.