Parsing in logstash using (grok,prune,multiline filter}

Hello everyone,
i have an export of a script below that i need to parse through logstash.

Job Name       : save_start_config-sftp            Job Status: Success (0)
Schedule Name  : daily_save_start_config           User Name : service-mainframe
Completion time: Mon Jan  8 06:00:21 2018
--------------------------------- Job Output ---------------------------------
`run-script bootflash:/save_start_config_sftp.vsh`
`copy running-config startup-config
`

[#                                       ]   1%
[#                                       ]   2%
[##                                      ]   3%
[##                                      ]   4%
[###                                     ]   5%
[###                                     ]   6%
[###                                     ]   7%
[####                                    ]   8%
[#####                                   ]  10%
[#####                                   ]  11%
[#####                                   ]  12%
[######                                  ]  13%
[######                                  ]  14%
[#######                                 ]  15%
[#######                                 ]  16%
[#######                                 ]  17%



[########################################]  98%
[########################################] 100%
Copy complete.
`copy startup-config sftp://service-mainframe@10.155.158.186./home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt
`
Connected to 10.155.158.186.
sftp> 
sftp> put  /var/tmp/vsh/SANMF-DAL-A-startup-config  /home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt 
Uploading /var/tmp/vsh/SANMF-DAL-A-startup-config to /home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt
sftp> exit 

Copy complete.

what i need is to drop everything till the (first) copy complete so this the part that i need from the log

copy startup-config sftp://service-mainframe@10.155.158.186/home/service-mainframe/dir_CCH-CHA-B/conf_WWEF-PAL-A_2018-01-10-06.00.16.txt
Connected to 10.155.158.186
sftp>
sftp> put  /var/tmp/vsh/CCH-CHA-B-startup-config  /home/service-mainframe/dir_CCH-CHA-B/conf_WWEF-PAL-A_2018-01-10-06.00.16.txt
Uploading /var/tmp/vsh/CCH-CHA-B-startup-config to /home/service-maiframe/dir_CCH-CHA-B/conf_WWEF-PAL-A_2018-01-10-06.00.16.txt
sftp> exit
Copy complete.

My grok patterns are:

%{WORD:action1} %{WORD}-%{WORD} %{WORD:protocol}://%{USER:utilisateur}@%{IP:clientip}/%{GREEDYDATA:repertoire}
%{GREEDYDATA} %{GREEDYDATA} %{IP}
%{GREEDYDATA}
%{WORD}> %{WORD:action2} %{GREEDYDATA:source} %{GREEDYDATA:destination}
%{WORD:action3} %{GREEDYDATA}
%{WORD}> %{WORD:action4}
%{GREEDYDATA:Status}

and the results are:

{
  "action1": [
    "copy"
  ],
  "protocol": [
    "sftp"
  ],
  "utilisateur": [
    "service-mainframe"
  ],
  "clientip": [
    "10.155.158.186"
  ],
  "repertoire": [
    "home/service-mainframe/dir_CCH-CHA-B/conf_WWEF-PAL-A_2018-01-10-06.00.16.txt"
  ],
  "action2": [
    "put"
  ],
  "source": [
    " /var/tmp/vsh/CCH-CHA-B-startup-config "
  ],
  "destination": [
    "/home/service-mainframe/dir_CCH-CHA-B/conf_WWEF-PAL-A_2018-01-10-06.00.16.txt"
  ],
  "action3": [
    "Uploading"
  ],
  "action4": [
    "exit"
  ],
  "Status": [
    "Copy complete."
  ]
}

how can i do this in logstash 2.4 what filter should i use other than Grok (prune or multiline could help?}
i am just a beginner in logstash what is the easiest way to parse this file through logstash, what i need do is:

1- Get rid of the first part of the log file (prune)
2-change the multine into a line (multiline filter)
3-write a grok patterns (grok filter)
4-assign each pattern as a field to make a dashboard on kibana

thank you !

Did you try with the grok patterns you have written above?

the grok patterns worked on the grok debugger site but at first i need to skip all the lines and keep only this part below to apply my grok patterns

`copy startup-config sftp://service-mainframe@10.155.158.186./home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt
`
Connected to 10.155.158.186.
sftp> 
sftp> put  /var/tmp/vsh/SANMF-DAL-A-startup-config  /home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt 
Uploading /var/tmp/vsh/SANMF-DAL-A-startup-config to /home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt
sftp> exit 

Copy complete.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.