# Input file (json format) through logstash to elasticsearch

**URL:** <https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109>\
**Category:** Logstash\
**Created:** [November 16, 2018, 5:03pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109 "2018-11-16T17:03:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![shiv94](https://avatars.discourse-cdn.com/v4/letter/s/bc8723/32.png) [@shiv94](https://discuss.elastic.co/u/shiv94)\
**Post date:** [November 16, 2018, 5:03pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/1 "2018-11-16T17:03:33Z")

</div>

Hey,

logstash.conf

input {  
file {  
path =\> "/root/file.json"  
start\_position =\> "beginning"  
codec =\> json  
type =\> "state"  
}  
}

output {  
if [type] == "state" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "state-%{+YYYY.MM.dd}"  
}  
}  
}

I am saving the json file (input to logstash) by  
curl -k --request GET --url [https://xxx](https://xxx) --header "x-auth-token: xxx" | jq '.' \> file.json

json file looks like

{  
"server": {  
"status": "ok",  
"code": "",  
"message": "Operation done successfully."  
},  
"counts": {  
"data\_counts": 21,  
"total\_counts": 21  
},  
"data": {  
"pprc": [  
{  
"sourcevolume": {  
"id": "xxx",  
"link": {  
"rel": "self",  
"href": "xxx"  
}  
},  
"targetvolume": {  
"id": "xxx",  
"link": {}  
},  
"targetsystem": {  
"id": "xxxx",  
"link": {}  
},  
"type": "globalcopy",  
"state": "copy\_pending"  
},  
}  
]  
}  
}

In the kibana dashboard, it looks like reading each line as seperate index

{  
"\_index": "state-2018.11.16",  
"\_type": "doc",  
"\_id": "aZxaHWcB1vLuNnA9uMrP",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"@timestamp": "2018-11-16T16:29:03.955Z",  
"@version": "1",  
"path": "/root/file.json",  
"host": "xxxx",  
"message": " "type": "globalcopy",",  
"tags": [  
"\_jsonparsefailure"  
],  
"type": "state"  
},  
"fields": {  
"@timestamp": [  
"2018-11-16T16:29:03.955Z"  
]  
},  
"sort": [  
1542385743955  
]  
}

{  
"\_index": "state-2018.11.16",  
"\_type": "doc",  
"\_id": "epxaHWcB1vLuNnA9uMrP",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"@timestamp": "2018-11-16T16:29:03.955Z",  
"@version": "1",  
"path": "/root/file.json",  
"host": "xxx",  
"message": " "state": "copy\_pending"",  
"tags": [  
"\_jsonparsefailure"  
],  
"type": "state"  
},  
"fields": {  
"@timestamp": [  
"2018-11-16T16:29:03.955Z"  
]  
},  
"sort": [  
1542385743955  
]  
}

I added this to logstash filter, no data is sent to elasticsearch

```auto
 codec => multiline
        {
            pattern => '^\{'
            negate => true
            what => previous                
        }

```

I even tried by removing jq filter but did not send data to elasticsearch

curl -k --request GET --url [https://xxx](https://xxx) --header "x-auth-token: xxx" \> file.json

json file

{"server":{"status":"ok","code":"","message":"Operation done successfully."},"counts":{"data\_counts":21,"total\_counts":21},"data":{"pprc":[{"sourcevolume":{"id":"xxx","link":{"rel":"self","href":"https:xxxx"}},"targetvolume":{"id":"1403","link":{}},"targetsystem":{"id":"xxx","link":{}},"type":"globalcopy","state":"copy\_pending"}}]}}

Can anyone help me?

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [November 16, 2018, 7:25pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/2 "2018-11-16T19:25:40Z")

</div>

You are correct in adding the multi-line codec. Logstash uses new lines as a delimiter for events. For your multi-line pattern, are you using single or double quotes? If single, like your example, change that to double-quotes and if that still doesn't work, remove the backslash, I don't believe`{` is a special character that needs to be escaped.

---

<div class="post-metadata">

**Author:** ![shiv94](https://avatars.discourse-cdn.com/v4/letter/s/bc8723/32.png) [@shiv94](https://discuss.elastic.co/u/shiv94)\
**Post date:** [November 16, 2018, 7:52pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/3 "2018-11-16T19:52:56Z")

</div>

Thanks for your reply.  
I will try using multi-line codec.

Why this format is not working the whole file is in a single line

{"server":{"status":"ok","code":"","message":"Operation done successfully."},"counts":{"data\_counts":21,"total\_counts":21},"data":{"pprc":[{"sourcevolume":{"id":"xxx","link":{"rel":"self","href":"https:xxxx"}},"targetvolume":{"id":"1403","link":{}},"targetsystem":{"id":"xxx","link":{}},"type":"globalcopy","state":"copy\_pending"}}]}}

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [November 17, 2018, 6:12pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/4 "2018-11-17T18:12:40Z")

</div>

> [@shiv94](#):
>
> {"server":{"status":"ok","code":"","message":"Operation done successfully."},"counts":{"data\_counts":21,"total\_counts":21},"data":{"pprc":[{"sourcevolume":{"id":"xxx","link":{"rel":"self","href":"https:xxxx"}},"targetvolume":{"id":"1403","link":{}},"targetsystem":{"id":"xxx","link":{}},"type":"globalcopy","state":"copy\_pending"}}]}}

That's weird...I didn't realize it was coming in as a single line when I originally responded. I wonder if the curl request is returning data in chunks and logstash is ingesting it before it's completed. You could try changing the file input `stat_interval` to `5`(default is 1s) to see if that improves your situation. Its possible the file is being read before all the data is written to file.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 18, 2018, 11:32am UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/5 "2018-11-18T11:32:55Z")

</div>

It is not valid JSON as there seems to be a curly brace too many towards the end. Have a look at it using e.g. [jsonlint](https://jsonlint.com), and you will see.

---

<div class="post-metadata">

**Author:** ![shiv94](https://avatars.discourse-cdn.com/v4/letter/s/bc8723/32.png) [@shiv94](https://discuss.elastic.co/u/shiv94)\
**Post date:** [November 19, 2018, 7:34pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/6 "2018-11-19T19:34:04Z")

</div>

This worked if I break the line.  
Added \n to file.json and applied filters json and split then it worked

input {  
file {  
path =\> "file/path/file.json"  
type =\> "state"  
start\_position =\> "beginning"  
}

}

filter  
{  
if [type] == "state" {  
json {  
source =\> "message"  
remove\_field =\> ["message"]  
}  
split {  
field =\> ["[data][pprc]"]  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 17, 2018, 7:34pm UTC](https://discuss.elastic.co/t/input-file-json-format-through-logstash-to-elasticsearch/157109/7 "2018-12-17T19:34:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.