Grok duplicate lines problem


(Sylvio de cezare) #1

Hi i have a switch log file log file which shows 3 parts. i am only interested in the third part
but the problem the lines are duplicated i am interested to show only the server name XXXX-XX-X
and the Status : Copy complete. here's the log:

`copy startup-config sftp://mfservice@10.240.151.155/home/mfservice/conf_XXXX-XXX-X-2018-01-08-06.00.16.txt
`
Connected to 10.240.151.155
sftp> 
sftp> put  /var/tmp/vsh/XXXX-XXX-X-startup-config  /home/mfservice/conf_XXXX-XXX-X-2018-01-08-06.00.16.txt 
Uploading /var/tmp/vsh/XXXX-XXX-X-startup-config to /home/mfservice/conf_XXXX-XXX-X_2018-01-08-06.00.16.txt
sftp> exit 

Copy complete.

`copy startup-config sftp://mfservice@10.240.151.155/home/mfservice/conf_XXXX-XXX-X_2018-01-09-06.00.15.txt
`
Connected to 10.240.151.155.
sftp> 
sftp> put  /var/tmp/vsh/XXXX-XXX-X-startup-config  /home/mfservice/conf_XXXX-XXX-X-2018-01-09-06.00.15.txt 
Uploading /var/tmp/vsh/XXXX-XXX-X-startup-config to /home/mfservice/conf_XXXX-XXX-X-2018-01-09-06.00.15.txt
sftp> exit 

Copy complete.

`copy startup-config sftp://mfservice@10.240.151.155/home/mfservice/dir_XXXX-XXX-X/conf_XXXX-XXX-X-2018-01-10-06.00.16.txt
`
Connected to 10.240.151.155.
sftp> 
sftp> put  /var/tmp/vsh/XXXX_XXX_X-startup-config  /home/mfservice/dir_XXX-XXX-X/conf_XXXX_XXX_X_2018-01-10-06.00.16.txt 
Uploading /var/tmp/vsh/XXXX_XXX_X-startup-config to /home/mfservice/dir_XXXX-XXX-X/conf_XXXX_XXX_X_2018-01-10-06.00.16.txt
sftp> exit 

Copy complete.

when i apply my grok i get 3 results of the pattern i only need one result, i am only interested in the part where the date in the txt file is 2018-01-10. any help please.

Thanks!


(Jymit Singh Khondhu) #2

The above looks like command line access. Can you share the make and model of the switch, logging coming out of the switch and your logstash config file? (input, filter, output).


(Sylvio de cezare) #3

well i didn't want to complicate thing i just wanted to know how to detect the last occurance of a word in the log file so i can apply my grok filter because i am only interested in the third part of the file.
here's my log file that i need to parse:

Job Name       : save_start_config-sftp            Job Status: Success (0)
    Schedule Name  : daily_save_start_config           User Name : mfservice
    Completion time: Mon Jan  8 06:00:21 2018
    --------------------------------- Job Output ---------------------------------
    `run-script bootflash:/save_start_config_sftp.vsh`
    `copy running-config startup-config
    `

    [#                                       ]   1%
    [#                                       ]   2%
    [##                                      ]   3%
    [##                                      ]   4%
    [###                                     ]   5%
    [###                                     ]   6%
    [###                                     ]   7%
    [####                                    ]   8%
    [#####                                   ]  10%
    [#####                                   ]  11%
 -------------------------------------------------
    [########################################] 100%
    Copy complete.
    `copy startup-config sftp://service-mainframe@10.155.158.186./home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-08-06.00.16.txt
    `
    Connected to 10.155.158.186.
    sftp> 
    sftp> put  /var/tmp/vsh/XXXX-XXX-X-startup-config  /home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-08-06.00.16.txt
    Uploading /var/tmp/vsh/XXXX-XXX-X-startup-config to /home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-08-06.00.16.txt
    sftp> exit 

    Copy complete.
    ==============================================================================
    Job Name       : save_start_config-sftp            Job Status: Success (0)
    Schedule Name  : daily_save_start_config           User Name : mfservice
    Completion time: Tue Jan  9 06:00:20 2018
    --------------------------------- Job Output ---------------------------------
    `run-script bootflash:/save_start_config_sftp.vsh`
    `copy running-config startup-config
    `

    [#                                       ]   1%
     [#                                       ]   2%
    [##                                      ]   3%
    [##                                      ]   4%
    [###                                     ]   5%
    [###                                     ]   6%
    [###                                     ]   7%
    [####                                    ]   8%
    [#####                                   ]  10%
    [#####                                   ]  11%
   -------------------------------------------------
    [########################################] 100%
    Copy complete.
    `copy startup-config sftp://service-mainframe@10.155.158.186./home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-10-06.00.16.txt
    `
    Connected to 10.155.158.186.
    sftp> 
    sftp> put  /var/tmp/vsh/XXXX-XXX-X-startup-config  /home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-09-06.00.16.txt
    Uploading /var/tmp/vsh/XXXX-XXX-X-startup-config to /home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-09-06.00.16.txt
    sftp> exit 

     Copy complete.
    ==============================================================================
    Job Name       : save_start_config-sftp            Job Status: Success (0)
    Schedule Name  : daily_save_start_config           User Name : mfservice
    Completion time: Tue Jan  10 06:00:20 2018
    --------------------------------- Job Output ---------------------------------
    `run-script bootflash:/save_start_config_sftp.vsh`
    `copy running-config startup-config
    `

    [#                                       ]   1%
     [#                                       ]   2%
    [##                                      ]   3%
    [##                                      ]   4%
    [###                                     ]   5%
    [###                                     ]   6%
    [###                                     ]   7%
    [####                                    ]   8%
    [#####                                   ]  10%
    [#####                                   ]  11%
   --------------------------------------------
    [#####                                   ]  100%
    Copy complete.
    `copy startup-config sftp://service-mainframe@10.155.158.186./home/service-mainframe/conf_WWF_CH_C_2018-01-08-06.00.16.txt
    `
    Connected to 10.155.158.186.
    sftp> 
    sftp> put  /var/tmp/vsh/XXXX-XXX-X-startup-config  /home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-10-06.00.16.txt
    Uploading /var/tmp/vsh/XXXX-XXX-X-startup-config to /home/service-mainframe/dir_XXXX-XXX-X/conf_XXXX-XXX_X_2018-01-10-06.00.16.txt
    sftp> exit.

i need to extract only the XXXX-XXX-XX and the second Occurence of Copy complete (in the third part).

so this is my filter :

filter {
if [message] !~ /^(Copy complete|`copy startup-config sftp:|Connected to|sftp> put|Uploading|sftp> exit)/{
drop {}

}

if  "`copy startup-config sftp:" in [message] {

       grok {

             match => { "message" => "%{WORD:action1} %{WORD}-%{WORD} %{WORD:protocol}://%{USER:utilisateur}@%{IP:clientip}/%{GREEDYDATA:repertoire}" }

              }}
 if "Copy" in [message] {
grok {
        match => {"message" => "%{GREEDYDATA:Status}" }
}

}

when this filter applied i get the same field of each part of the log file.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.