# Logstash configuration for multiple json (nested) files in a directory

**URL:** <https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333>\
**Category:** Logstash\
**Tags:** ingest-pipeline\
**Created:** [March 16, 2021, 8:46am UTC](https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333 "2021-03-16T08:46:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Razeen](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@Razeen](https://discuss.elastic.co/u/Razeen)\
**Post date:** [March 16, 2021, 8:46am UTC](https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333/1 "2021-03-16T08:46:03Z")

</div>

Hi,

My logstash configuration is as follows,

```auto
    input {
    	file {
    		start_position => "beginning"
    		path => "/path/to/json/files*.json"
    		sincedb_path => "/dev/null"
    		codec => json
    	
    	}
    }

    filter {
      split {     
        field => "[data]"
      }
      
      mutate {
        add_field => { 
    	    "url" => "%{[data][url]}"
            "vendorId" => "%{[data][vendorId]}"
            "vendor" => "%{[data][vendor]}"
            "productId" => "%{[data][productId]}"
            "product" => "%{[data][product]}"
            "vendor_details" => "%{[data][vendor_details]}"
        }
        
        remove_field => ["[data]" ]
      }
      
    }

    output {
    	elasticsearch {
    		hosts => ["elasticsearch/url"]
    		index => "test-index"
    		user => test
    		password => test333
    	}

    	stdout { codec => rubydebug }
    }

```

with the above configuration, the data is being indexed to elasticsearch, but as a message.

```auto
    "_index" : "test-index",
        "_type" : "_doc",
        "_id" : "NuQoOngBH1fxUIM9w4b7",
        "_score" : 1.0,
        "_source" : {
          "url" : "%{[data][url]}",
          "productId" : "%{[data][productId]}",
          "tags" : [
            "_jsonparsefailure",
            "_split_type_failure"
          ],
          "vendor" : "%{[data][vendor]}",
          "@version" : "1",
          "vendorId" : "%{[data][vendorId]}",
          "product" : "%{[data][product]}",
          "message" : "\"vendor_details\": [{\"installCountry\": \"Mexico\", \"installRegion\": \"LATAM\", \"vendorId\": \"476\", \"vendor\": \"General Technology\", \"productId\": \"823\", \"product\": \"Ajax\", \"

```

Please let me know what am i doing wrong here?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 16, 2021, 3:05pm UTC](https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333/2 "2021-03-16T15:05:25Z")

</div>

Your [message] field is not valid JSON, so a json filter cannot parse it. Perhaps the JSON in your file is pretty-printed, in which case you need to combine lines to form a complete JSON object. You would use a multiline codec to do that. If you want to read the entire file as one event then [this](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/2) describes how to do that.

---

<div class="post-metadata">

**Author:** ![Razeen](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@Razeen](https://discuss.elastic.co/u/Razeen)\
**Post date:** [March 16, 2021, 4:42pm UTC](https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333/3 "2021-03-16T16:42:07Z")

</div>

hi Badger,

Thanks for your reply.  
I tried the codec you suggested.

instead of

```auto
codec => json

```

I used

```auto
codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }

```

my json file looks like this,

```auto
{
    "vendor_details": [
        {
            "vendorId": "273",
            "vendor": "test vendor1",
            "productId": "830",
            "product": "test product1T"
        },
        {
            "vendorId": "273",
            "vendor": "test vendor2",
            "productId": "12174",
            "product": "test product2"
        }
    ],
    "url": "tester.com",
    "company": "test company",
    "hqPhone": "",
    "productCount": 2,
    "revenue": 20000
}

```

with above codec change my logstash api starts, but nothing gets ingested.

---

<div class="post-metadata">

**Author:** ![Razeen](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@Razeen](https://discuss.elastic.co/u/Razeen)\
**Post date:** [March 16, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333/4 "2021-03-16T16:54:11Z")

</div>

when i use the codec as

```auto
codec => json

```

the data gets ingested but not the way i expect.  
i get the following error too,

```auto
        [0] "_jsonparsefailure",
        [1] "_split_type_failure"

```

and the indexed data looks like this,

```auto
"vendor" => "%{[data][vendor]}",
"product" => "%{[data][product]}",
 "vendorCount" => "%{[data][vendorCount]}",

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-configuration-for-multiple-json-nested-files-in-a-directory/267333/5 "2021-04-13T16:54:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.