Split an event help needed

Elastic Stack Logstash
1

Hi,

i want help in split the field below, I want to create array of all the occurrence in the field below which starts with "START - PHASE 1 09:02:16.31, time is the dynamic value generated.
Please find the input field and expected output updated below. Am looking of the solution online am not understanding it quite well, please help.

Field to be split:

"restData" => "START - PHASE 1 09:02:16.31\nSTART - PHASE 1 SUBPHASE 1 09:02:16.31\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_UPGRADEMGMT IL_RIESEN IL_RIAMAN IL_MFSPRB IL_NASEVA \n IL_EFOCEF \nPROCESSING FAMILIES \nREADY - PHASE 1 SUBPHASE 1 09:02:16.67\nSTART - PHASE 1 SUBPHASE 3 09:02:16.67\nPROCESSING FAMILIES \nFILE LOADING TSLCGRGX SWOCGRGX SCHORDGX SMPERMGX CMECGRGX \n CGNAMEGX UCQFILNG UCYFILNG UC2FILNG UC3FILNG \n NGDATANG ID9FILNG LEFILEGX M3PARAGX IP9FILNG \n IP8FILNG IP7FILNG ALQFILGX TDNFILGX NG2DATNG \n IW6FILNG VAITXTNG VAD01001 VAD01002 PHPARAGX \n IWPARAGX VAXL00NG VAXL01NG VAXL02NG VAXL03NG \n VAXL04NG VAXL05NG VAXL06NG VAXL07NG VAXL08NG \n VAXL09NG VAXL10NG VAXL11NG VAXL12NG VAXL13NG \n VAXL14NG VAXL15NG VAXE00NG VAXE01NG VAXE02NG \n VAXE03NG VAXE04NG VAXE05NG VAXE06NG VAXE07NG \n VAXE08NG VAXE09NG VAXE10NG VAXE11NG VAXE12NG \n VAXE13NG VAXE14NG VAXE15NG IWQFILNG TO3FILNG \n TN3FILNG P5PARAGX LEKFILNG LTGFILNG H24LOGNG \n AKFILEGX VKFILEGX RKFILEGX TKFILEGX IP2FILNG \n IP4FILNG EIKFILNG \nPROCESSING FAMILIES \nREADY - PHASE 1 SUBPHASE 3 09:02:17.22\nSTART - PHASE 1 SUBPHASE 2 09:02:17.22\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_WUPMAN IL_FMPPRB IL_AMNPRB IL_THERMO IL_LASTPROC12 \nPROCESSING FAMILIES IL_LASTPROC12 \nREADY - PHASE 1 SUBPHASE 2 09:02:17.23\nREADY - PHASE 1\nSTART - PHASE 2 09:02:17.23\nSTART - PHASE 2 SUBPHASE 1 09:02:17.23\nPROCESSING FAMILIES \nPROCESSING FAMILIES \nREADY - PHASE 2 SUBPHASE 1 09:02:17.23\nSTART - PHASE 2 SUBPHASE 3 09:02:17.23\nPROCESSING FAMILIES \nFILE LOADING \nPROCESSING FAMILIES \nREADY - PHASE 2 SUBPHASE 3 09:02:17.23\nSTART - PHASE 2 SUBPHASE 2 09:02:17.23\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_MM5PRB IL_GIVCLIENT IL_CPMONITOR IL_LASTPROC22 \nPROCESSING FAMILIES IL_LASTPROC22 \nREADY - PHASE 2 SUBPHASE 2 09:02:17.27\nREADY - PHASE 2\nSTART - PHASE 3 09:02:17.27\nSTART - PHASE 3 SUBPHASE 1 09:02:17.27\nPROCESSING FAMILIES IL_RIESEN \nPROCESSING FAMILIES \nREADY - PHASE 3 SUBPHASE 1 09:02:17.27\nSTART - PHASE 3 SUBPHASE 2 09:02:17.27\nPROCESSING FAMILIES IL_THERMO \nSTARTING FAMILIES MGW_PHLPRB MGW_VMGWHANDLER MGW_NSMANA MGW_REMOTETDMMGR MGW_TDMLOGICSTATES \n MGW_MLPMAS MGW_PSEMAN MGW_UMXPRB MGW_RTBPRB MGW_RFHPRB MGW_ATVPRB \n MGW_PMHPRB MGW_TG3CAL MGW_CM4PRB MGW_CMQPRB MGW_NEMED MGW_ZAUPRB \n MGW_UMZPRO MGW_TR6FRA MGW_TDMPROXY MGW_UPPHANDLER IL_LASTPROC32 \nPROCESSING FAMILIES IL_THERMO \nREADY - PHASE 3 SUBPHASE 2 09:02:17.52\nREADY - PHASE 3\nSTART - PHASE 4 09:02:17.52\nWARMING BEGIN 09:02:17.52\nWARMING FAILURE WITH ERROR 0xff\nWARMING END 09:06:12.72\nPROCESSING FAMILIES IL_UPGRADEMGMT IL_LASTPROC32 \nREADY - PHASE 4\nREADY - SE 09:06:12.72\n-----------Unit Starting up at 2017-10-01 09:16:14.48---------------"

Output Expected:

[
START - PHASE 1 09:02:16.31\nSTART - PHASE 1 SUBPHASE 1 09:02:16.31\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_UPGRADEMGMT IL_RIESEN IL_RIAMAN IL_MFSPRB IL_NASEVA \n IL_EFOCEF \nPROCESSING FAMILIES \nREADY - PHASE 1 SUBPHASE 1 09:02:16.67\nSTART - PHASE 1 SUBPHASE 3 09:02:16.67\nPROCESSING FAMILIES \nFILE LOADING TSLCGRGX SWOCGRGX SCHORDGX SMPERMGX CMECGRGX \n CGNAMEGX UCQFILNG UCYFILNG UC2FILNG UC3FILNG \n NGDATANG ID9FILNG LEFILEGX M3PARAGX IP9FILNG \n IP8FILNG IP7FILNG ALQFILGX TDNFILGX NG2DATNG \n IW6FILNG VAITXTNG VAD01001 VAD01002 PHPARAGX \n IWPARAGX VAXL00NG VAXL01NG VAXL02NG VAXL03NG \n VAXL04NG VAXL05NG VAXL06NG VAXL07NG VAXL08NG \n VAXL09NG VAXL10NG VAXL11NG VAXL12NG VAXL13NG \n VAXL14NG VAXL15NG VAXE00NG VAXE01NG VAXE02NG \n VAXE03NG VAXE04NG VAXE05NG VAXE06NG VAXE07NG \n VAXE08NG VAXE09NG VAXE10NG VAXE11NG VAXE12NG \n VAXE13NG VAXE14NG VAXE15NG IWQFILNG TO3FILNG \n TN3FILNG P5PARAGX LEKFILNG LTGFILNG H24LOGNG \n AKFILEGX VKFILEGX RKFILEGX TKFILEGX IP2FILNG \n IP4FILNG EIKFILNG \nPROCESSING FAMILIES \nREADY - PHASE 1 SUBPHASE 3 09:02:17.22\nSTART - PHASE 1 SUBPHASE 2 09:02:17.22\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_WUPMAN IL_FMPPRB IL_AMNPRB IL_THERMO IL_LASTPROC12 \nPROCESSING FAMILIES IL_LASTPROC12 \nREADY - PHASE 1 SUBPHASE 2 09:02:17.23\nREADY - PHASE 1\n,
START - PHASE 2 09:02:17.23\nSTART - PHASE 2 SUBPHASE 1 09:02:17.23\nPROCESSING FAMILIES \nPROCESSING FAMILIES \nREADY - PHASE 2 SUBPHASE 1 09:02:17.23\nSTART - PHASE 2 SUBPHASE 3 09:02:17.23\nPROCESSING FAMILIES \nFILE LOADING \nPROCESSING FAMILIES \nREADY - PHASE 2 SUBPHASE 3 09:02:17.23\nSTART - PHASE 2 SUBPHASE 2 09:02:17.23\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_MM5PRB IL_GIVCLIENT IL_CPMONITOR IL_LASTPROC22 \nPROCESSING FAMILIES IL_LASTPROC22 \nREADY - PHASE 2 SUBPHASE 2 09:02:17.27\nREADY - PHASE 2\n,
START - PHASE 3 09:02:17.27\nSTART - PHASE 3 SUBPHASE 1 09:02:17.27\nPROCESSING FAMILIES IL_RIESEN \nPROCESSING FAMILIES \nREADY - PHASE 3 SUBPHASE 1 09:02:17.27\nSTART - PHASE 3 SUBPHASE 2 09:02:17.27\nPROCESSING FAMILIES IL_THERMO \nSTARTING FAMILIES MGW_PHLPRB MGW_VMGWHANDLER MGW_NSMANA MGW_REMOTETDMMGR MGW_TDMLOGICSTATES \n MGW_MLPMAS MGW_PSEMAN MGW_UMXPRB MGW_RTBPRB MGW_RFHPRB MGW_ATVPRB \n MGW_PMHPRB MGW_TG3CAL MGW_CM4PRB MGW_CMQPRB MGW_NEMED MGW_ZAUPRB \n MGW_UMZPRO MGW_TR6FRA MGW_TDMPROXY MGW_UPPHANDLER IL_LASTPROC32 \nPROCESSING FAMILIES IL_THERMO \nREADY - PHASE 3 SUBPHASE 2 09:02:17.52\nREADY - PHASE 3\n,
START - PHASE 4 09:02:17.52\nWARMING BEGIN 09:02:17.52\nWARMING FAILURE WITH ERROR 0xff\nWARMING END 09:06:12.72\nPROCESSING FAMILIES IL_UPGRADEMGMT IL_LASTPROC32 \nREADY - PHASE 4\nREADY - SE 09:06:12.72\n-----------Unit Starting up at 2017-10-01 09:16:14.48---------------
]

2

You can use a ruby regular expression with a negative lookahead assertion.

ruby { code => 'event.set("matches", event.get("message").scan(/^START - PHASE [0-9] [0-9](?:(?!START - PHASE [0-9] [0-9]).)*/m))' }

Basically that says start capturing whereever you see "START - PHASE [0-9] [0-9]" but stop capturing if you see another occurrence of "START - PHASE [0-9] [0-9]" (or end of string). This results in

       "matches" => [
        [0] "START - PHASE 1 09:02:16.31\nSTART - PHASE 1 SUBPHASE 1 09:02:16.31\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_UPGRADEMGMT IL_RIESEN IL_RIAMAN IL_MFSPRB IL_NASEVA \n IL_EFOCEF \nPROCESSING FAMILIES \nREADY - PHASE 1 SUBPHASE 1 09:02:16.67\nSTART - PHASE 1 SUBPHASE 3 09:02:16.67\nPROCESSING FAMILIES \nFILE LOADING TSLCGRGX SWOCGRGX SCHORDGX SMPERMGX CMECGRGX \n CGNAMEGX UCQFILNG UCYFILNG UC2FILNG UC3FILNG \n NGDATANG ID9FILNG LEFILEGX M3PARAGX IP9FILNG \n IP8FILNG IP7FILNG ALQFILGX TDNFILGX NG2DATNG \n IW6FILNG VAITXTNG VAD01001 VAD01002 PHPARAGX \n IWPARAGX VAXL00NG VAXL01NG VAXL02NG VAXL03NG \n VAXL04NG VAXL05NG VAXL06NG VAXL07NG VAXL08NG \n VAXL09NG VAXL10NG VAXL11NG VAXL12NG VAXL13NG \n VAXL14NG VAXL15NG VAXE00NG VAXE01NG VAXE02NG \n VAXE03NG VAXE04NG VAXE05NG VAXE06NG VAXE07NG \n VAXE08NG VAXE09NG VAXE10NG VAXE11NG VAXE12NG \n VAXE13NG VAXE14NG VAXE15NG IWQFILNG TO3FILNG \n TN3FILNG P5PARAGX LEKFILNG LTGFILNG H24LOGNG \n AKFILEGX VKFILEGX RKFILEGX TKFILEGX IP2FILNG \n IP4FILNG EIKFILNG \nPROCESSING FAMILIES \nREADY - PHASE 1 SUBPHASE 3 09:02:17.22\nSTART - PHASE 1 SUBPHASE 2 09:02:17.22\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_WUPMAN IL_FMPPRB IL_AMNPRB IL_THERMO IL_LASTPROC12 \nPROCESSING FAMILIES IL_LASTPROC12 \nREADY - PHASE 1 SUBPHASE 2 09:02:17.23\nREADY - PHASE 1\n",
        [1] "START - PHASE 2 09:02:17.23\nSTART - PHASE 2 SUBPHASE 1 09:02:17.23\nPROCESSING FAMILIES \nPROCESSING FAMILIES \nREADY - PHASE 2 SUBPHASE 1 09:02:17.23\nSTART - PHASE 2 SUBPHASE 3 09:02:17.23\nPROCESSING FAMILIES \nFILE LOADING \nPROCESSING FAMILIES \nREADY - PHASE 2 SUBPHASE 3 09:02:17.23\nSTART - PHASE 2 SUBPHASE 2 09:02:17.23\nPROCESSING FAMILIES \nSTARTING FAMILIES IL_MM5PRB IL_GIVCLIENT IL_CPMONITOR IL_LASTPROC22 \nPROCESSING FAMILIES IL_LASTPROC22 \nREADY - PHASE 2 SUBPHASE 2 09:02:17.27\nREADY - PHASE 2\n",
        [2] "START - PHASE 3 09:02:17.27\nSTART - PHASE 3 SUBPHASE 1 09:02:17.27\nPROCESSING FAMILIES IL_RIESEN \nPROCESSING FAMILIES \nREADY - PHASE 3 SUBPHASE 1 09:02:17.27\nSTART - PHASE 3 SUBPHASE 2 09:02:17.27\nPROCESSING FAMILIES IL_THERMO \nSTARTING FAMILIES MGW_PHLPRB MGW_VMGWHANDLER MGW_NSMANA MGW_REMOTETDMMGR MGW_TDMLOGICSTATES \n MGW_MLPMAS MGW_PSEMAN MGW_UMXPRB MGW_RTBPRB MGW_RFHPRB MGW_ATVPRB \n MGW_PMHPRB MGW_TG3CAL MGW_CM4PRB MGW_CMQPRB MGW_NEMED MGW_ZAUPRB \n MGW_UMZPRO MGW_TR6FRA MGW_TDMPROXY MGW_UPPHANDLER IL_LASTPROC32 \nPROCESSING FAMILIES IL_THERMO \nREADY - PHASE 3 SUBPHASE 2 09:02:17.52\nREADY - PHASE 3\n",
        [3] "START - PHASE 4 09:02:17.52\nWARMING BEGIN 09:02:17.52\nWARMING FAILURE WITH ERROR 0xff\nWARMING END 09:06:12.72\nPROCESSING FAMILIES IL_UPGRADEMGMT IL_LASTPROC32 \nREADY - PHASE 4\nREADY - SE 09:06:12.72\n-----------Unit Starting up at 2017-10-01 09:16:14.48---------------"
    ],
1 Like
3 
ruby { code => 'event.set("matches", event.get("message").scan(/^START - PHASE [0-9](?:(?!START - PHASE [0-9]).)*/m))' }

Kept one number range [0-9] instead of two as the phases never going to exceed 9 and it worked like awesome Badger.
Thank you so much for your timely help :slight_smile:,
I really appreciate your swift help :slight_smile:

4

There is a very big difference between

/^START - PHASE [0-9] [0-9](?:(?!START - PHASE [0-9] [0-9]).)*/m

which divides your example into 4 strings (where "START - PHASE n" is followed by the first digit of the time) and

/^START - PHASE [0-9](?:(?!START - PHASE [0-9]).)*/m

which divides it into 12, because that also splits out all the strings like "START - PHASE 2 SUBPHASE 1".

I checked and you only need the additional " [0-9]" in the lookahead, so

scan(/^START - PHASE [0-9](?:(?!START - PHASE [0-9] [0-9]).)*/m

also produces 4 matches and not 12.

1 Like
5

I wanted to get 4 matches Badger. i handled the duplicates with two white spaces, as a result only the correct intended line in the log will match.

ruby {
code => 'event.set("matches", event.get("restData").scan(/^START - PHASE [0-9]\s\s(?:(?!START - PHASE [0-9]\s\s).)*/m))'
}

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.