# Indexing JSON files from a local directory to elastic

**URL:** <https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667>\
**Category:** Logstash\
**Created:** [May 29, 2018, 11:10am UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667 "2018-05-29T11:10:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yaniv\_Boneh](https://avatars.discourse-cdn.com/v4/letter/y/eb9ed0/32.png) [@Yaniv\_Boneh](https://discuss.elastic.co/u/Yaniv_Boneh)\
**Post date:** [May 29, 2018, 11:10am UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667/1 "2018-05-29T11:10:07Z")

</div>

Hello,

I wish to configure my Logstash pipline so it will index JSON files from a local directory.  
Of course, I already went through relevant topics and found no satisfying solution for my case, for instance, this issue here:

> [@Input JSON file to elasticsearch via logstash](https://discuss.elastic.co/t/input-json-file-to-elasticsearch-via-logstash/72437):
>
> hi, im facing an issue with logstash while inserting json data to elasticsearch. here is my logstash config file input { file { codec =\> multiline { pattern =\> '^{' negate =\> true what =\> previous } path =\> ["/usr/local/Cellar/logstash/5.1.1/test\_payment.json"] start\_position =\> "beginning" sincedb\_path =\> "/dev/null" codec =\> "json" } } filter { mutate { replace =\> ["message", "%{message}"] gsub =\> ['message','\n',''] } if [message] =~ /^{.\*}$/ { j…

is very similar to mine.

My pipline config file looks as follows:

> input  
> {  
> file  
> {  
> codec =\> multiline  
> {  
> pattern =\> '^{'  
> negate =\> true  
> what =\> previous  
> }  
> path =\> ["c:/work/UXMresults/_.json"]  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> exclude =\> "_.gz"  
> }  
> }
> 
> filter  
> {
> 
> }  
> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> index =\> "yeti"  
> }
> 
> }

where a typical json file I want to index looks like this:

> {  
> "testParams":  
> {  
> "testingDevice":"UXM",  
> "visaAddress": "TCPIP0::172.25.150.216::5125::SOCKET",  
> "testCase":"",  
> "testDescription":"",  
> "stopCondition":  
> {  
> "numOfSubFrames":""  
> },  
> "rfBoxConfiguration":"Yeti\_2x8"
> 
> ```
> },
> "preScriptUxmParams":
> {
> "antConfig":"D2U1",
> "schedulerMode":"",
> "bw":"BW20",
> "duplex":"tdd",
> "band":"41",
> "periodicCsi":"",
> "aperiodicCsi":"",
> "mcs":"15",
> "tm":"",
> "tddConfig":"",
> "ssfConfig":"",
> "numOfCw":"",
> "awgn":"",
> "channel":"",
> "allocationType":"",
> "cfi":"",
> "numOfLayers":"",
> "rsPower":""
> },
> "runScriptUxmParams":
> {
> "macPadding":""
> },
> "results":
> {
> "dl":
> {
> "throughput":
> {
> "max":"164",
> "avarage":"50"
> },
> "BLER":"3.2"
> },
> "ul":
> {
> "throughput":
> {
> "max":"10",
> "avarage":"10"
> },
> "BLER":"0"
> }		
> }                        
> 
> ```
> 
> }

Logstash seems to accept this pipline config when I run it

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/8/38dd2b20b8ad284ea41cf3088899a99ce958db22.png)

nevertheless,  
**I cant find any trace to the index I gave in the config file("yeti") on Kibana\>\>Discover**

Help will be mush appreciated  
Yaniv

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 29, 2018, 11:35am UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667/2 "2018-05-29T11:35:32Z")

</div>

> sincedb\_path =\> "/dev/null"

On Windows use "nul", not "/dev/null".

---

<div class="post-metadata">

**Author:** ![Yaniv\_Boneh](https://avatars.discourse-cdn.com/v4/letter/y/eb9ed0/32.png) [@Yaniv\_Boneh](https://discuss.elastic.co/u/Yaniv_Boneh)\
**Post date:** [May 29, 2018, 11:58am UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667/3 "2018-05-29T11:58:51Z")

</div>

Still no luck

> input  
> {  
> file  
> {
> 
> ```
> path => ["C:\work\UXMresults\test_ex2.json"]
> start_position => "beginning"
> sincedb_path => "nul"
> exclude => "*.gz"
> codec => json
> }
> 
> ```
> 
> }
> 
> filter  
> {  
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> index =\> "yeti"  
> }
> 
> }

Logstash is ok with the config file

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/c/3c07857c7de8c7787b140767461d90c23cb0e4aa.png)

But Kibana doesn't seem to recognize the given index

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8fcb70c7ac85ec4e52daf41618daa6f7fe093d3f.png)

Besides using the console, is there another sanity check I can do to test the ability of kibana to identify new indexed documents?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 30, 2018, 6:04am UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667/4 "2018-05-30T06:04:00Z")

</div>

Temporarily replace the elasticsearch output with a `stdout { codec => rubydebug }` output to just dump all events being read. Are you getting anything then?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [May 30, 2018, 2:33pm UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667/5 "2018-05-30T14:33:10Z")

</div>

You can't use the json codec like this - it is expecting one flat JSON doc per line and multiline is a real pain

Pretty printed JSON files are a problem.

**if the files are content complete** , i.e. their size is fixed, then there is a side effect hack one can use but **you will need the latest logstash file input v4.1.2**.

This version (4.1.2) has a `read` mode. In this mode the end-of-file is significant and we use this to flush any content that accumulates in the delimiter search buffer.

How do we read the whole file into the buffer without breaking it up into lines? We set an impossible delimiter that can never be found in the content. For example, unicode `U+00B6` Pilcrow Sign '¶' or `U+00A7` Section Sign '§' or a combination of the two. This has the effect of creating a document with the whole file content in the message field, newlines and all - obviously, the file should not be too big (you don't want OOM). You can use the JSON filter to parse the message field into the document.  
Example:

```auto
input {
  file {
    path => "/Users/guy/tmp/testing/logs/sample.json"
    sincedb_path => "/dev/null"
    delimiter => "§¶¶§"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "/Users/guy/tmp/testing/logs/test-json-ml-hack-completed.txt"
  }
}

filter {
  json {
    source => "[message]"
    remove_field => ["[message]"]
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

```

The JSON file content looks like this:

```auto
{"widget": {
    "debug": "on",
    "window": {
        "title": "Sample Konfabulator Widget",
        "name": "main_window",
        "width": 500,
        "height": 500
    },
    "image": {
        "src": "Images/Sun.png",
        "name": "sun1",
        "hOffset": 250,
        "vOffset": 250,
        "alignment": "center"
    },
    "text": {
        "data": "Click Here",
        "size": 36,
        "style": "bold",
        "name": "text1",
        "hOffset": 250,
        "vOffset": 100,
        "alignment": "center",
        "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
    }
}}

```

And the resultant Logstash docs (event):

```auto
{
      "@version" => "1",
    "@timestamp" => 2018-05-30T14:26:43.308Z,
          "host" => "Elastics-MacBook-Pro.local",
          "path" => "/Users/guy/tmp/testing/logs/sample.json",
        "widget" => {
         "debug" => "on",
         "image" => {
                 "name" => "sun1",
              "vOffset" => 250,
                  "src" => "Images/Sun.png",
            "alignment" => "center",
              "hOffset" => 250
        },
        "window" => {
            "height" => 500,
              "name" => "main_window",
             "title" => "Sample Konfabulator Widget",
             "width" => 500
        },
          "text" => {
                 "name" => "text1",
            "alignment" => "center",
                 "size" => 36,
            "onMouseUp" => "sun1.opacity = (sun1.opacity / 100) * 90;",
                 "data" => "Click Here",
              "vOffset" => 100,
                "style" => "bold",
              "hOffset" => 250
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 27, 2018, 2:33pm UTC](https://discuss.elastic.co/t/indexing-json-files-from-a-local-directory-to-elastic/133667/6 "2018-06-27T14:33:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.