I'm feeding JSON files to logstash from filebeat. Logstash is set to output JSON to ElasticSearch.
However, my JSON key-value pairs all show up as a giant clump within the "message" field. I believe ElasticSearch is designed to work with JSON files, so how can I have fields be automatically generated from the keys in my JSON?
For example this is what filebeat is outputting to logstash:
2016-08-02T14:53:58-05:00 DBG Publish: {
"@timestamp": "2016-08-02T19:53:51.247Z",
"beat": {
"hostname": "USAUSWNCN094TTP",
"name": "USAUSWNCN094TTP"
},
"count": 1,
"fields": null,
"input_type": "log",
"message": "{\"LeadId\":22472,\"LeadGuid\":\"E1EB5EA4-E57F-41F0-AD95-FC215E2D8399\",\"CustomerFirstName\":\"NYLA\",\"CustomerCity\":\"atlanta\",\"CustomerState\":\"GA\",\"CustomerPostalCode\":\"21532\",\"CustomerCountryCode\":\"US\",\"CustomerLanguageCode\":\"en-us\",\"HasFilledOutAccessoryCoupon\":false,\"RequestedDealerContactAboutTrade\":false,\"HasFilledOutTradeInIncentiveCoupon\":false,\"HasAcknowledgedPrivacyPolicy\":true,\"HasAQuestionForDealer\":false,\"HasSelectedFandI\":true,\"HasSelectedAccessories\":true,\"HasGmRewardsCard\":false,\"HasSubmittedToDealer\":true,\"HasSubmittedCreditApplication\":false ...
However, the JSON file I am feeding in starts with the LeadId key-value pair. I don't want it all clumped up inside "message" because then I can't filter adequately using kibana after it gets sent to elastic.
Any ideas?