# How to consume a JSON stream and keep the original stream data unchanged

**URL:** <https://discuss.elastic.co/t/how-to-consume-a-json-stream-and-keep-the-original-stream-data-unchanged/127451>\
**Category:** Kibana\
**Created:** [April 10, 2018, 9:28am UTC](https://discuss.elastic.co/t/how-to-consume-a-json-stream-and-keep-the-original-stream-data-unchanged/127451 "2018-04-10T09:28:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahrtr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahrtr/32/29551_2.png) [@ahrtr](https://discuss.elastic.co/u/ahrtr)\
**Post date:** [April 10, 2018, 9:28am UTC](https://discuss.elastic.co/t/how-to-consume-a-json-stream-and-keep-the-original-stream-data-unchanged/127451/1 "2018-04-10T09:28:55Z")

</div>

I am implementing a kibana plugin to check/authorize the request payload. The format of some request payload is JSONStream, an example is as below,

> {"index":["logstash1\*"],"ignore\_unavailable":true,"preference":1523345500966}  
> {"size":0,"\_source":{"excludes":}}  
> {"index":["logstash2\*"],"ignore\_unavailable":true,"preference":1523345500966}  
> {"size":0,"\_source":{"excludes":}}

I need to extract all the index names (logstash1\* and logstash2\* in above example) firstly, then do some check/authorize on the index names, the code is something as below,

```
import JSONStream from 'JSONStream'; 
import eventStream from 'event-stream';

async function checkIndexNames(request) {
    const payload = request.payload;
    let indexArray = [];
    await eventStream.pipeline(
        payload, 
        JSONStream.parse(['index']),
        eventStream.mapSync(function(data){
            indexArray.push(data);
        })
    );

    // Do some checks on the index names 
}

```

Basically the above code works well, all the index names can be extracted correctly. After my plugin finishes checking the payload, then kibana/elasticsearch needs to continue to process the payload.

The issue is that kibana/elasticsearch fails to process the payload, and complains that " [parse\_exception] request body or source parameter is required". The reason is that the payload has already been consumed by my plugin, and the original request payload doesn't exist any more. If I comment out the "await eventStream.pipeline(...)", then everything is OK with kibana.

So the question is how can I consume the request payload, and keep the original payload unchanged afterwards? Thanks.

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [April 10, 2018, 6:48pm UTC](https://discuss.elastic.co/t/how-to-consume-a-json-stream-and-keep-the-original-stream-data-unchanged/127451/2 "2018-04-10T18:48:55Z")

</div>

Hey there, unfortunately we don't currently provide support for third-party plugin development because our APIs are still evolving and are liable to break things as they change.

Thanks,  
CJ

---

<div class="post-metadata">

**Author:** ![ahrtr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahrtr/32/29551_2.png) [@ahrtr](https://discuss.elastic.co/u/ahrtr)\
**Post date:** [April 10, 2018, 11:46pm UTC](https://discuss.elastic.co/t/how-to-consume-a-json-stream-and-keep-the-original-stream-data-unchanged/127451/3 "2018-04-10T23:46:19Z")

</div>

@cjcenizal

Thanks for the feedback. Actually this is a generic question about nodejs instead of kibana/elasticsearch. It's appreciated if whatever suggestions can be provided.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2018, 11:46pm UTC](https://discuss.elastic.co/t/how-to-consume-a-json-stream-and-keep-the-original-stream-data-unchanged/127451/4 "2018-05-08T23:46:47Z")

</div>

Closed on May 8, 2018, 11:46pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.