# How to parse multiple json files in hierarchical directory structure

**URL:** <https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147>\
**Category:** Logstash\
**Created:** [May 8, 2019, 9:56am UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147 "2019-05-08T09:56:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![stour](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@stour](https://discuss.elastic.co/u/stour)\
**Post date:** [May 8, 2019, 9:56am UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/1 "2019-05-08T09:56:36Z")

</div>

Hi,

I am new in ELK Stack. I want to parse multiple json files from hierarchical directory structure.

Following is the directory structure.

```
data
|-- 2017
| |-- 01
| | |-- 01
| | | |-- a.json
| | | `-- b.json
| | `-- 02
| | |-- c.json
| | `-- d.json
| `-- 02
| |-- 01
| | |-- f.json
| | `-- g.json
| `-- 02
|-- 2018
| |-- 01
| | |-- 01
| | | |-- ab.json
| | | `-- ac.json
| | |-- 02
| | | |-- ba.json
| | | `-- bc.json
| | |-- 03
| | `-- 04
| `-- 02
`-- 2019
    |-- 01
    | |-- 01
    | | `-- x.json
    | |-- 02
    | `-- 03
    |-- 02
    `-- 03

```

Following is the config file

```
input {
  file {
    path => "/home/stour/data/"
    start_position => "beginning"
    codec => "json"
    sincedb_path => "/dev/null"
    type => "json"
  }
}
filter {
  json {
    source => "message"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => ["logstash-%{+YYYY.DD.MM.hh.mm.ss}"]
  }
  stdout {
    codec => "rubydebug"
  }
}

```

How to parse json files like above directory structure?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 8, 2019, 12:59pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/2 "2019-05-08T12:59:13Z")

</div>

You would use 'path =\> /home/stour/data/\*\*/\*.json'. You may want to use [read](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_read_mode) mode rather than tail mode, but make sure you understand the default value of [file\_completed\_action](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-file_completed_action).

---

<div class="post-metadata">

**Author:** ![stour](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@stour](https://discuss.elastic.co/u/stour)\
**Post date:** [May 15, 2019, 6:59am UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/3 "2019-05-15T06:59:13Z")

</div>

Hi Badger,

Thank you for the help. I tried with above solution but it is not working.

Please check below config file

```
input {
  file {
    path => "/home/stour/data/**/*.json"
    #start_position => "beginning"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/home/stour/Documents/logs"
    codec => "json_lines"
    sincedb_path => "/dev/null"
    type => "json"
    exclude => "*.tgz"
  }
}
filter {
  json {
    source => "message"
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => ["logstash-%{+YYYY.DD.MM}"]
  }
  stdout {
    codec => "rubydebug"
  }
}

```

And the json file contains data like  
`{"version":1,"checksum":"68e9982a3c55657ef9429918590e629835ffcf24","data":{"time":"2017-05-30T16:01:02.5153013Z","os":{"product":"Wondows 10","build":"9200"},"product":{"hash":"xyz","version":"1.1.12","status":"active"}}}`

There is no new line character at the end of json file.

Please help me to solve this issue.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2019, 12:36pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/4 "2019-05-15T12:36:42Z")

</div>

> [@stour](#):
>
> There is no new line character at the end of json file.
> 
> Please help me to solve this issue.

The file input will not read files that do not have a delimiter on the final line.

---

<div class="post-metadata">

**Author:** ![stour](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@stour](https://discuss.elastic.co/u/stour)\
**Post date:** [May 15, 2019, 12:48pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/5 "2019-05-15T12:48:34Z")

</div>

What will be the delimiter at final line?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2019, 1:00pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/6 "2019-05-15T13:00:51Z")

</div>

Whatever is native for the platform you are on -- \n on UNIX, \r\n on Windows.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2019, 1:01pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-in-hierarchical-directory-structure/180147/7 "2019-06-12T13:01:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.