# Json of varying length and multiline **URL:** \ **Category:** Logstash\ **Created:** [July 22, 2019, 5:09pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689 "2019-07-22T17:09:39Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 22, 2019, 5:09pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/1 "2019-07-22T17:09:39Z")
I'm very new to everything elastic, logstash and json. I was able to stand everything up and ingest one set of logs and query on them in kibana. My second data set is a json file whose fields vary from message to message. There are a lot of fields, about 176, but a single record which is multilined may only have 30 of the fields represented. My first question is will a records ingest into elastic stack if the field is in the index but others in the index are not present, or do I need to add the null fields? I haven't been able to index any docs for this json source. The second question is how do I represent the multiline pattern in my config file. I have read many of the multiline posts, but I just can't seem to generalize their solutions to meet my need. I put message in the filter thinking that is some generic value that denotes a message. Does this value have to be some field structure that is present in the json? Here is my current config file: ``` input { file { type => "json" path => "C:/ESTK/*.json" codec => multiline { pattern => "}}" negate => "false" what => "previous" } start_position => "beginning" sincedb_path => "NUL" } } filter { json {source => "message"} } output { elasticsearch { action => "index" hosts => "http://localhost:9200" index => "gt" } } ``` I appreciate any help with my issues. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 22, 2019, 5:28pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/2 "2019-07-22T17:28:19Z")
To answer your first question, no, you do not need to add null values for fields that are missing from some events. Please edit your post, select the configuration and click on \ in the toolbar above the edit panel. You should see the text appear in the preview panel on the right ``` like this with indentation preserved. ``` Thirdly, can you show us what the json looks like. It's impossible to suggest what multiline pattern you should use if we do not know what the data looks like. You can sanitize all the values if you want. ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 23, 2019, 9:08am UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/3 "2019-07-23T09:08:55Z")
Firstly, thank you for the insight regarding excluded fields in a record. I didn't mention before, but I am using es 7.2 and logstash 7.2 I have adjusted my post. Here is a sample json event: ``` { "THRUSHPLUS": { "BILL_OF_LADING": "GBHTY7755563", "CARGO_DESCRIPTION": "33 PALLET, CEREALS|", "CARRIER": "ACL", "CONTAINER_NUMBER": "TOLU9813600", "DESTINATION_LRT_CD": "UNECE", "DESTINATION_RAW_INFO": "GBLIV", "DISCHARGE_PORT_ETA": "2019-07-03 00:00:00", "DISCHARGE_PORT_LRT_CD": "UNECE", "DISCHARGE_PORT_RAW_INFO": "GBLIV", "EQUIPMENT_STATUS": "IMPORT", "FCL_LCL": "FCL", "FULL_EMPTY": "FULL", "ISO_CONTAINER_CODE": "20GP", "LLOYDS_NO": "9670573", "LOAD_PORT_LRT_CD": "UNECE", "LOAD_PORT_RAW_INFO": "NLRTM", "MANIFEST_QUANTITY": "1", "MANIFEST_QUANTITY_UOM": "LT", "MESSAGE_TYPE": "DISCHARGE", "RAW_FILE_NAME": "1790180309", "SEALING_PARTY": "CA", "SEAL_NUMBER": "43790217", "SHIPPER_CODE": "WT782", "DATA_REF": "OPEN", "TIME_FRAME": "20180903", "VESSEL_CALL_SIGN": "2ITA4", "VESSEL_NAME": "ATLANTIC STAR", "VOYAGE_REF": "762E" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 23, 2019, 1:03pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/4 "2019-07-23T13:03:08Z")
If you JSON is pretty-printed like that then you can make your pattern look for the non-indented } that ends the object ``` codec => multiline { pattern => "^}" negate => true what => next auto_flush_interval => 1 } ``` ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 23, 2019, 7:40pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/5 "2019-07-23T19:40:03Z")
I haven't been able to load any data. I thought that maybe my mapping was not correct. I did several things this afternoon: 1. I simplified the json event to not be pretty-printed and to no longer be nested under the THRUSHPLUS field. 2. I altered my es index map to conform to the new format. 3. I altered my config file to suit the non-nest pretty-print format. I am unsure of how to use the filter now. It was so easy with the CSV data that I have in es. I really don't know what to make of the json { source =\> XXXXX} filter. There is no single field under which the message data lies. I have have embedded the new message format and conf file. ``` input { file { type => "json" path => "C:/ESTK/*.json" start_position => "beginning" sincedb_path => "NUL" } } filter { json {source => "message"} } output { elasticsearch { action => "index" hosts => "http://localhost:9200" index => "gt" } } ``` {"BARGE\_CODE":"PAU2752","CARGO\_DESCRIPTION":"LIRIODENDRON TULIPIFERA KD LUMBER LIRIODENDRON TULIPIFERA KD LUMBER HS CODE: 4407.97|.","CARRIER":"QRT","CHECK\_DIGIT\_CALCULATED":"0","CHECK\_DIGIT\_VALID":"TRUE","VESSEL\_NAME":"MV BRIGADOON","VOYAGE\_REF":"92W","WEIGHT\_1":"21850"} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 23, 2019, 8:51pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/6 "2019-07-23T20:51:19Z")
> [@Shipper01](#): > > {"BARGE\_CODE":"PAU2752","CARGO\_DESCRIPTION":"LIRIODENDRON TULIPIFERA KD LUMBER LIRIODENDRON TULIPIFERA KD LUMBER HS CODE: 4407.97|.","CARRIER":"QRT","CHECK\_DIGIT\_CALCULATED":"0","CHECK\_DIGIT\_VALID":"TRUE","VESSEL\_NAME":"MV BRIGADOON","VOYAGE\_REF":"92W","WEIGHT\_1":"21850"} If that JSON is a single line from the file then you will get an event that looks like ``` "VESSEL_NAME" => "MV BRIGADOON", "CHECK_DIGIT_CALCULATED" => "0", "VOYAGE_REF" => "92W", "WEIGHT_1" => "21850", "CARGO_DESCRIPTION" => "LIRIODENDRON TULIPIFERA KD LUMBER LIRIODENDRON TULIPIFERA KD LUMBER HS CODE: 4407.97|.", "CHECK_DIGIT_VALID" => "TRUE", "CARRIER" => "QRT", "BARGE_CODE" => "PAU2752" ``` What do you not like about that? ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 23, 2019, 9:21pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/7 "2019-07-23T21:21:26Z")
I like it, but it doesn't seem to go into es. I do a doc count and it shows 0. ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 23, 2019, 9:22pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/8 "2019-07-23T21:22:45Z")
So my config is ok? Fyi, i don't have a field called message. Is this a problem? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 23, 2019, 9:54pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/9 "2019-07-23T21:54:17Z")
If you are using a file input like that then each event will have a field called message and it will contain a single line from the file. ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 24, 2019, 9:37am UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/10 "2019-07-24T09:37:45Z")
ok, so if I want to have all the data in the single line json in es as a single event message with all the fields, I should change the json format to an array and the es map as well? ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 24, 2019, 10:07am UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/11 "2019-07-24T10:07:55Z")
> [@Shipper01](#): > > BARGE\_CODE What I am trying to manipulate is to have that input file shown above to ingest as a single event or doc in es. That is if event and doc are equivalent. I control the structure of the input json, so I can adjust it if need be. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 24, 2019, 12:04pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/12 "2019-07-24T12:04:02Z")
With the configuration you posted, each line from the file will be a single event in logstash and a single document in elasticsearch. ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 24, 2019, 12:04pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/13 "2019-07-24T12:04:39Z")
I now don't believe that I want an array, but a grouping of objects (that single json record above as a single doc in es. ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 24, 2019, 3:39pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/14 "2019-07-24T15:39:04Z")
I still can't get data to go into es I have made some changes: 1. event file looks like this: {"THRUSHPLUS": {"AGENT\_ADDRESS\_1": "","BARGE\_CODE": "","BILL\_OF\_LADING": "","BL\_ISSUING\_AGENT": "","BOL\_TYPE": "","CARGO\_DESCRIPTION": "","CARGO\_VOLUME": "1","CARRIER": "","CONTAINER\_NUMBER": "","CONTAINER\_TYPE": "","DISCHARGE\_PORT\_ATA": "2019-07-22 23:23:23","DISCHARGE\_PORT\_CITY": "","DISCHARGE\_PORT\_COUNTRY": "","DISCHARGE\_PORT\_ETA": "2019-07-22 23:23:23","DISCHARGE\_PORT\_LRT\_CD": "","DISCHARGE\_PORT\_RAW\_INFO": "","EQUIPMENT\_STATUS": "","FCL\_LCL": "","HAZMAT\_CLASS": "","HAZMAT\_CODE": "","HAZ\_DESCRIPTION": "","HAZ\_FLAG": "","ISO\_CONTAINER\_CODE": "","LOAD\_PORT\_ATD": "2019-07-22 23:23:23","LOAD\_PORT\_CITY": "","LOAD\_PORT\_COUNTRY": "","LOAD\_PORT\_ETD": "2019-07-22 23:23:23","LOAD\_PORT\_LRT\_CD": "","LOAD\_PORT\_RAW\_INFO": "","TARE\_WEIGHT": "1","TEMP\_CONTROLLED": "","TO\_COUNTRY": "","TO\_UNECE": "","TRAIN\_CODE": "","TRUCK\_CODE": "","VESSEL\_NAME": "","VOYAGE\_REF": "","WEIGHT\_1": "1","WEIGHT\_2": "1"}} My logstash config looks like this: ``` input { file { type => "json" path => "C:/ESTK/*.json" start_position => "beginning" sincedb_path => "NUL" } } filter { json {source => "THRUSHPLUS"} } output { elasticsearch { action => "index" hosts => "http://localhost:9200" index => "gt" } } ``` Though I have also tried substituting "message" for THRUSHPLUS. This is my index map for es: ``` PUT gt { "settings" : { "number_of_shards" : 1 }, "mappings" : { "properties" : { "CARGOLINKPLUS" : { "properties" : { "BARGE_CODE": { "type" : "text"}, "BOL_TYPE": { "type" : "text"}, "CARGO_DESCRIPTION": { "type" : "text"}, "CARGO_VOLUME": { "type" : "float"}, "CARRIER": { "type" : "text"}, "CONTAINER_NUMBER": { "type" : "text"}, "CONTAINER_TYPE": { "type" : "text"}, "DISCHARGE_PORT_ATA": { "type" : "date","format": "yyyy-MM-dd HH:mm:ss"}, "DISCHARGE_PORT_CITY": { "type" : "text"}, "DISCHARGE_PORT_COUNTRY": { "type" : "text"}, "DISCHARGE_PORT_ETA": { "type" : "date","format": "yyyy-MM-dd HH:mm:ss"}, "DISCHARGE_PORT_LOC_ID": { "type" : "text"}, "DISCHARGE_PORT_LRT_CD": { "type" : "text"}, "DISCHARGE_PORT_RAW_INFO": { "type" : "text"}, "EQUIPMENT_STATUS": { "type" : "text"}, "FCL_LCL": { "type" : "text"}, "HAZMAT_CLASS": { "type" : "text"}, "HAZMAT_CODE": { "type" : "text"}, "HAZ_DESCRIPTION": { "type" : "text"}, "HAZ_FLAG": { "type" : "text"}, "ISO_CONTAINER_CODE": { "type" : "text"}, "LOAD_PORT_ATD": { "type" : "date","format": "yyyy-MM-dd HH:mm:ss"}, "LOAD_PORT_CITY": { "type" : "text"}, "LOAD_PORT_COUNTRY": { "type" : "text"}, "LOAD_PORT_ETD": { "type" : "date","format": "yyyy-MM-dd HH:mm:ss"}, "LOAD_PORT_LOC_ID": { "type" : "text"}, "LOAD_PORT_LRT_CD": { "type" : "text"}, "LOAD_PORT_RAW_INFO": { "type" : "text"}, "TARE_WEIGHT": { "type" : "integer"}, "TEMP_CONTROLLED": { "type" : "text"}, "TRAIN_CODE": { "type" : "text"}, "TRUCK_CODE": { "type" : "text"}, "VESSEL_NAME": { "type" : "text"}, "VOYAGE_REF": { "type" : "text"}, "WEIGHT_1": { "type" : "float"}, "WEIGHT_2": { "type" : "float"} } } } } } ``` I just get no data going in. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 24, 2019, 3:57pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/15 "2019-07-24T15:57:15Z")
What happens if you delete the mapping? ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 24, 2019, 9:35pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/16 "2019-07-24T21:35:41Z")
Sorry for the delay. The stack crashed. Rebuilt it and will respond in the AM. ---
**Author:** ![Shipper01](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@Shipper01](https://discuss.elastic.co/u/Shipper01)\ **Post date:** [July 29, 2019, 2:21pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/17 "2019-07-29T14:21:25Z")
Sorry for the delay. Deleting the mapping enabled es to index the data. That meant that the mapping and the data weren’t matched, so I reevaluated the mapping. Some extra spaces were found in the mapping. Have redone the mapping to ensure the data types desired are defined. Recreated the index and data is indexing. Thank you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 26, 2019, 2:21pm UTC](https://discuss.elastic.co/t/json-of-varying-length-and-multiline/191689/18 "2019-08-26T14:21:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.