# How to parse/load JSON file or .TXT file containing JSON to logstash using GROK

**URL:** <https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246>\
**Category:** Logstash\
**Created:** [April 16, 2020, 5:33am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246 "2020-04-16T05:33:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dhruv\_Mevada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhruv_mevada/32/48003_2.png) [@Dhruv\_Mevada](https://discuss.elastic.co/u/Dhruv_Mevada)\
**Post date:** [April 16, 2020, 5:33am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246/1 "2020-04-16T05:33:43Z")

</div>

I'm new in ELK & I have logs in JSON format & I can save it to JSON file or txt file. Now,

1. Which will be a better approach should I save logs to JSON file or txt file?
2. How should I parse/load the logs to Logstash, what will be the configuration file of Logstash?

**Sample logs**

```auto
{"time":"2019-04-15 5:00:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}
{"time":"2019-04-15 5:01:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}

```

I tried with below configuration but not works  
**Logstash configuration file**

```auto
input {
 file {
   type => "json"
   path => "/home/dhruv/logs.txt"
   start_position => beginning
 }
}
filter {
 json {
   source => "message"
 }
}
output {
   elasticsearch {
       hosts => ["localhost:9220"]
       index => "call_manager"
   }
}

```

I want to parse every key present in the logs. Can anyone please help me out with the configuration file.  
Thanks in advance..

---

<div class="post-metadata">

**Author:** ![quentin.legraverend](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin.legraverend/32/66373_2.png) [@quentin.legraverend](https://discuss.elastic.co/u/quentin.legraverend)\
**Post date:** [April 16, 2020, 8:13am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246/2 "2020-04-16T08:13:29Z")

</div>

The `type` parameter of an input is just adding a field named "type" with value "json" (in your case). See [this](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-type) for more info.

What you are actually looking for is the `codec` parameter that you can fix to "json" in your Logstash input. See [this](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-codec) and find the codec list [here](https://www.elastic.co/guide/en/logstash/7.6/codec-plugins.html).

Answering your questions:

1. If the format is `json`, I think putting the `.json` extension is appropriate, but even if you can save those files as .txt you can read them as json using the `codec` parameter.
2. Your workflow should be quite simple here:
  1. Input your files using `json codec`
  2. _Optional: you can use `grok` in the `filter` section to make additional parsing on specific fields._
  3. Output your event to `elasticsearch`

---

<div class="post-metadata">

**Author:** ![Dhruv\_Mevada](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dhruv_mevada/32/48003_2.png) [@Dhruv\_Mevada](https://discuss.elastic.co/u/Dhruv_Mevada)\
**Post date:** [April 16, 2020, 10:02am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246/3 "2020-04-16T10:02:18Z")

</div>

Hi, Thanks for the response.  
I have finalized the file format as JSON for logs.

**log.json**

```auto
{"time":"2019-04-15 5:00:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}
{"time":"2019-04-15 5:01:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}

```

So, as you shared, I gone through the "type" parameter details and also I have updated the conf file.

**Logstash configuration file**

```auto
input {
 file {
   type => "json"
   codec => "json"
   path => "/home/dhruv/logs.json"
   start_position => beginning
 }
}
filter {
 json {
   source => "time"
   source => "log-level" 
   source => "operation" 
   source => "module"
   source => "event"
   source => "message"
 }
}
output {
   elasticsearch {
       hosts => ["localhost:9220"]
       index => "call_manager"
   }
}

```

I tried above conf. file but Logstash didn't return anything also I checked in Kibana to add a new index "call\_manager" but not shows any index.

Can you please verify the conf. file especially the filter part.  
Thanks a lot

---

<div class="post-metadata">

**Author:** ![stevesimpson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stevesimpson/32/41437_2.png) [@stevesimpson](https://discuss.elastic.co/u/stevesimpson)\
**Post date:** [April 16, 2020, 10:42am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246/4 "2020-04-16T10:42:50Z")

</div>

Hi @Dhruv_Mevada, the filter looks wrong to me. Having recently done some parsing of JSON logs I would try the following:

```
input {
 file {
   codec => "json"
   path => "/home/dhruv/logs.json"
   start_position => beginning
 }
}
filter {
 json {
   source => "message"
 }
}
output {
   stdout { codec => rubydebug { metadata => true } }
}

```

I've also set the output to be stdout so you can view the Logstash logs to verify everything is correct. Once you are happy this can be set back to the elasticsearch output.

I'd also potentially look at using [https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-target](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-target) as you have a message field within you JSON which may cause conflicts with the default "message".

---

<div class="post-metadata">

**Author:** ![quentin.legraverend](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin.legraverend/32/66373_2.png) [@quentin.legraverend](https://discuss.elastic.co/u/quentin.legraverend)\
**Post date:** [April 16, 2020, 10:47am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246/5 "2020-04-16T10:47:44Z")

</div>

First of all, consider that the `type` parameter you add in the input section is just a flag added to the event. It might be used for rooting or filtering purpose but is not mandatory except is you need it.

To me, you do not need anything in your `filter` section, as said before, in your case this is optional in case you need to parse the json fields (extracted from codec input) themselves. You might be misunderstanding the json filter. Check the [doc page](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-source). You should use it when you have raw JSON in one of your field, but in your case the input `codec` is already splitting your input following json format. Try deleting everything in the filter section: `filter {}` should be enough.

I did not try myself the exact same configuration but with my understanding of the docs that's how I would configure it in your case.

Last but not the least, since you are in dev mode and you are reading the same file many times, I recommend you to check the [sincedb\_path](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path) and [sincedb\_clean\_after](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_clean_after). Native behaviour of Logstash it to read each file only once which is (I suppose) not matching your development case. Moreover, I think this is the issue you are currently facing regarding the data not spawning on Elasticsearch/Kibana.

Those changes might push something to your Elasticsearch. If it is not the case, you might need to dig more in Logstash or Elasticsearch log files to know what's the exact error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2020, 10:47am UTC](https://discuss.elastic.co/t/how-to-parse-load-json-file-or-txt-file-containing-json-to-logstash-using-grok/228246/6 "2020-05-14T10:47:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.