Logstash configuration for multiple json (nested) files in a directory

[Razeen]

Hi,

My logstash configuration is as follows,

    input {
    	file {
    		start_position => "beginning"
    		path => "/path/to/json/files*.json"
    		sincedb_path => "/dev/null"
    		codec => json
    	
    	}
    }

    filter {
      split {     
        field => "[data]"
      }
      
      mutate {
        add_field => { 
    	    "url" => "%{[data][url]}"
            "vendorId" => "%{[data][vendorId]}"
            "vendor" => "%{[data][vendor]}"
            "productId" => "%{[data][productId]}"
            "product" => "%{[data][product]}"
            "vendor_details" => "%{[data][vendor_details]}"
        }
        
        remove_field => [ "[data]" ]
      }
      
    }

    output {
    	elasticsearch {
    		hosts => ["elasticsearch/url"]
    		index => "test-index"
    		user => test
    		password => test333
    	}

    	stdout { codec => rubydebug }
    }

with the above configuration, the data is being indexed to elasticsearch, but as a message.

    "_index" : "test-index",
        "_type" : "_doc",
        "_id" : "NuQoOngBH1fxUIM9w4b7",
        "_score" : 1.0,
        "_source" : {
          "url" : "%{[data][url]}",
          "productId" : "%{[data][productId]}",
          "tags" : [
            "_jsonparsefailure",
            "_split_type_failure"
          ],
          "vendor" : "%{[data][vendor]}",
          "@version" : "1",
          "vendorId" : "%{[data][vendorId]}",
          "product" : "%{[data][product]}",
          "message" : "\"vendor_details\": [{\"installCountry\": \"Mexico\", \"installRegion\": \"LATAM\", \"vendorId\": \"476\", \"vendor\": \"General Technology\", \"productId\": \"823\", \"product\": \"Ajax\", \"

Please let me know what am i doing wrong here?

[Badger]

Your [message] field is not valid JSON, so a json filter cannot parse it. Perhaps the JSON in your file is pretty-printed, in which case you need to combine lines to form a complete JSON object. You would use a multiline codec to do that. If you want to read the entire file as one event then this describes how to do that.

[Razeen]

hi Badger,

Thanks for your reply.
I tried the codec you suggested.

instead of

codec => json

I used

codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }

my json file looks like this,

{
    "vendor_details": [
        {
            "vendorId": "273",
            "vendor": "test vendor1",
            "productId": "830",
            "product": "test product1T"
        },
        {
            "vendorId": "273",
            "vendor": "test vendor2",
            "productId": "12174",
            "product": "test product2"
        }
    ],
    "url": "tester.com",
    "company": "test company",
    "hqPhone": "",
    "productCount": 2,
    "revenue": 20000
}

with above codec change my logstash api starts, but nothing gets ingested.

[Razeen]

when i use the codec as

codec => json

the data gets ingested but not the way i expect.
i get the following error too,

        [0] "_jsonparsefailure",
        [1] "_split_type_failure"

and the indexed data looks like this,

"vendor" => "%{[data][vendor]}",
"product" => "%{[data][product]}",
 "vendorCount" => "%{[data][vendorCount]}",