Split data and convert to json

[rahulkothanath (Rahul)]

I want to split a text ,based on a pattern and parse each split separate and combine last to make final json. I am new to log-stash and wants to find the best solution for my problem.
Input: @CM0033520180530135914732477 0 IB HOST COMPONENTO20180530135917049898PQR000201805300959140003 5519035830232316 03EUF AUTO P2 PERS 3 06800000T5MTTIN00003 30000000000000000@TY001radha.moningi@abc.com 000003CAUEP8eQkx2B000000009.50000001000050469820110.241.168.5 C E 0 0 ALLOW_TRANSFER CA7Mdt7t @TY001MGZFY Z MILLER 0000004275577768226505 Y10.240.73.46

The input is having two components starting with @CM003,@TY001. the grok match pattern for each component is different.
I have to find the components of each input and parse it to create json.
do I need to use a for loop,to process each component separate or is there any plugin to solve this .
possible options:

1. split text on @CM003|@TYOO1 pattern and process each component inside a for loop.(but it will take more time due to for loop)

Sample output:

indent preformatted text by 4 spaces{
      "COMP_VER_NO": "@CM003",
      "MSG_SRC_TP": "24",
      "XCN_TMSTMP": "",
      "FILLER_1": "",
      "PEND_RELS_CD": "",
      "CRD_VRSN_NO": "",
      "MSG_SRC_STR": "IB HOSTMPONENT",
      "PROC_CTR_CD": "O",
      "XCN_TMSTMP1": "2018120105522652",
      "XCN_TYPE_CD": "PQR",
      "XCN_RESPONSE_CD": "000",
      "XCN_LOCAL_DATE": "20181201",
      "XCN_LOCAL_TIME": "005312",
      "CRD_FI_ID": "0003",
      "CRD_NO": "451903029784",
      "CRD_ISSUE_NO": "00",
      "CLIENT_NM": "JANET E3F5",
      "CRD_ENTLMNT_CD": 999,
      "RESPONSIBLTY_TRNST": 0,
      "TERMINAL_ID_CD": "T",
      "FI_ID": "0003",
      "TRM_TYPE_CD": "5",
      "CLNT_DOB": "00000000",
      "CLNT_DATE_ON_FILE": "00000000",
      "XCN_DATE_TIME": "2018-12-01T05:53:12.922",
      "TB_IB_signon_transaction_component_version_no": "@TY001",
      "SIGNON_TYPE": "4",
      "SIGNON_SUCCESS_IND": "C",
      "IP_ADDRESS": "",
      "TOUCH_ID_IND": "",
      "FILLER_2": "",
      "IP_ADDRESS_OCT": "",
      "BROWSER_IND": "0",
      
      
      
    }

Thanks for reading and helping.

[Badger]

Your output contains data that does not even appear in the input. My guess is you will have to use a bunch of custom patterns to consume fixed width fields, but you do not supply enough information for us to be sure.

[rahulkothanath (Rahul)]

Sorry ,my output is just a sample .
basically I want to split the message R CA7Mdt7t |@TY001MGZF on the pipe and process each portion separate based on component(@CM001,TY001).
example:
1.input: message(mesage1message2)
2.split message into message1 and message2.
3. use grok match to process message 1 and message 2 seperatley with custom pattern.

I am not sure ,how to accomplish this.

[Badger]

You could start with

grok { match => { "message" => "^(?<part1>[^\|]+)\|%{GREEDYDATA:part2}" } }

[rahulkothanath (Rahul)]

Thanks for the help. I think this helps me

what if I have more than two parts in the message. I want to split on | and put each part in a list and process using a loop.
eg:
input: message(msg1|msg2|msg3....|msgn)
list=[msg1,msg2,....,msgn]

now I want to loop through the list and process each msg.

[Badger]

    mutate { add_field => { "someField" => "msg1|msg2|msg3....|msgn" } }
    mutate { split => { "someField" => "|" } }

would give you

 "someField" => [
    [0] "msg1",
    [1] "msg2",
    [2] "msg3....",
    [3] "msgn"
],