How to parse multiple json files in hierarchical directory structure

(Shrimant Taur) #1

Hi,

I am new in ELK Stack. I want to parse multiple json files from hierarchical directory structure.

Following is the directory structure.

data
|-- 2017
|   |-- 01
|   |   |-- 01
|   |   |   |-- a.json
|   |   |   `-- b.json
|   |   `-- 02
|   |       |-- c.json
|   |       `-- d.json
|   `-- 02
|       |-- 01
|       |   |-- f.json
|       |   `-- g.json
|       `-- 02
|-- 2018
|   |-- 01
|   |   |-- 01
|   |   |   |-- ab.json
|   |   |   `-- ac.json
|   |   |-- 02
|   |   |   |-- ba.json
|   |   |   `-- bc.json
|   |   |-- 03
|   |   `-- 04
|   `-- 02
`-- 2019
    |-- 01
    |   |-- 01
    |   |   `-- x.json
    |   |-- 02
    |   `-- 03
    |-- 02
    `-- 03

Following is the config file

input {
  file {
    path => "/home/stour/data/"
    start_position => "beginning"
    codec => "json"
    sincedb_path => "/dev/null"
    type => "json"
  }
}
filter {
  json {
    source => "message"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => ["logstash-%{+YYYY.DD.MM.hh.mm.ss}"]
  }
  stdout {
    codec => "rubydebug"
  }
}

How to parse json files like above directory structure?

#2

You would use 'path => /home/stour/data/**/*.json'. You may want to use read mode rather than tail mode, but make sure you understand the default value of file_completed_action.

(Shrimant Taur) #3

Hi Badger,

Thank you for the help. I tried with above solution but it is not working.

Please check below config file

input {
  file {
    path => "/home/stour/data/**/*.json"
    #start_position => "beginning"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/home/stour/Documents/logs"
    codec => "json_lines"
    sincedb_path => "/dev/null"
    type => "json"
    exclude => "*.tgz"
  }
}
filter {
  json {
    source => "message"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => ["logstash-%{+YYYY.DD.MM}"]
  }
  stdout {
    codec => "rubydebug"
  }
}

And the json file contains data like
{"version":1,"checksum":"68e9982a3c55657ef9429918590e629835ffcf24","data":{"time":"2017-05-30T16:01:02.5153013Z","os":{"product":"Wondows 10","build":"9200"},"product":{"hash":"xyz","version":"1.1.12","status":"active"}}}

There is no new line character at the end of json file.

Please help me to solve this issue.

#4

The file input will not read files that do not have a delimiter on the final line.

(Shrimant Taur) #5

What will be the delimiter at final line?

#6

Whatever is native for the platform you are on -- \n on UNIX, \r\n on Windows.