How to parse/load JSON file or .TXT file containing JSON to logstash using GROK

[Dhruv_Mevada]

I'm new in ELK & I have logs in JSON format & I can save it to JSON file or txt file. Now,

1. Which will be a better approach should I save logs to JSON file or txt file?
2. How should I parse/load the logs to Logstash, what will be the configuration file of Logstash?

Sample logs

{"time":"2019-04-15 5:00:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}
{"time":"2019-04-15 5:01:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}

I tried with below configuration but not works
Logstash configuration file

input {
 file {
   type => "json"
   path => "/home/dhruv/logs.txt"
   start_position => beginning
 }
}
filter {
 json {
   source => "message"
 }
}
output {
   elasticsearch {
       hosts => ["localhost:9220"]
       index => "call_manager"
   }
}

I want to parse every key present in the logs. Can anyone please help me out with the configuration file.
Thanks in advance..

[quentin.legraverend]

The type parameter of an input is just adding a field named "type" with value "json" (in your case). See this for more info.

What you are actually looking for is the codec parameter that you can fix to "json" in your Logstash input. See this and find the codec list here.

Answering your questions:

1. If the format is json, I think putting the .json extension is appropriate, but even if you can save those files as .txt you can read them as json using the codec parameter.
2. Your workflow should be quite simple here:
   1. Input your files using json codec
   2. Optional: you can use grok in the filter section to make additional parsing on specific fields.
   3. Output your event to elasticsearch

[Dhruv_Mevada]

Hi, Thanks for the response.
I have finalized the file format as JSON for logs.

log.json

{"time":"2019-04-15 5:00:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}
{"time":"2019-04-15 5:01:00","log-level":"INFO","operation":"database","module":"call-manager","event":"call","message":"Call initiated"}

So, as you shared, I gone through the "type" parameter details and also I have updated the conf file.

Logstash configuration file

input {
 file {
   type => "json"
   codec => "json"
   path => "/home/dhruv/logs.json"
   start_position => beginning
 }
}
filter {
 json {
   source => "time"
   source => "log-level" 
   source => "operation" 
   source => "module"
   source => "event"
   source => "message"
 }
}
output {
   elasticsearch {
       hosts => ["localhost:9220"]
       index => "call_manager"
   }
}

I tried above conf. file but Logstash didn't return anything also I checked in Kibana to add a new index "call_manager" but not shows any index.

Can you please verify the conf. file especially the filter part.
Thanks a lot

[stevesimpson]

Hi @Dhruv_Mevada, the filter looks wrong to me. Having recently done some parsing of JSON logs I would try the following:

input {
 file {
   codec => "json"
   path => "/home/dhruv/logs.json"
   start_position => beginning
 }
}
filter {
 json {
   source => "message"
 }
}
output {
   stdout { codec => rubydebug { metadata => true } }
}

I've also set the output to be stdout so you can view the Logstash logs to verify everything is correct. Once you are happy this can be set back to the elasticsearch output.

I'd also potentially look at using https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-target as you have a message field within you JSON which may cause conflicts with the default "message".

[quentin.legraverend]

First of all, consider that the type parameter you add in the input section is just a flag added to the event. It might be used for rooting or filtering purpose but is not mandatory except is you need it.

To me, you do not need anything in your filter section, as said before, in your case this is optional in case you need to parse the json fields (extracted from codec input) themselves. You might be misunderstanding the json filter. Check the doc page. You should use it when you have raw JSON in one of your field, but in your case the input codec is already splitting your input following json format. Try deleting everything in the filter section: filter {} should be enough.

I did not try myself the exact same configuration but with my understanding of the docs that's how I would configure it in your case.

Last but not the least, since you are in dev mode and you are reading the same file many times, I recommend you to check the sincedb_path and sincedb_clean_after. Native behaviour of Logstash it to read each file only once which is (I suppose) not matching your development case. Moreover, I think this is the issue you are currently facing regarding the data not spawning on Elasticsearch/Kibana.

Those changes might push something to your Elasticsearch. If it is not the case, you might need to dig more in Logstash or Elasticsearch log files to know what's the exact error.