How can sort assorted logs to steam line logs

Elastic Stack Logstash
1

I have below assorted logs,

how can i sort to

Time
Device Source
only log message

and remove rest all unwanted log message

Sep 1 14:40:33 VPX-02 09/01/2018:13:40:33 GMT VPX-02 0-PPE-0 : default API CMD_EXECUTED 523995 0 : User nsroot - Remote_ip 192.168.255.47 - Command "stat ns" - Status "Success"
Sep 1 14:40:33 VPX-01 09/01/2018:13:40:33 GMT VPX-01 0-PPE-0 : default API CMD_EXECUTED 525164 0 : User nsroot - Remote_ip 192.168.255.47 - Command "stat ns" - Status "Success"
Sep 1 14:40:34 SWIAS02 7187590: Sep 1 14:40:33.081 BST: %ILPOWER-5-IEEE_DISCONNECT: Interface Fa1/0/39: PD removed
Sep 1 14:43:41 SITEIAS01 599982: Sep 1 14:43:40.604 BST: %ADJ-5-RESOLVE_REQ_FAIL: Adj resolve request failed for 10.188.98.231 on Vlan498
Sep 1 14:43:42 BRANCH01 BST: %PLATFORM_STACKPOWER-4-REDUNDANCY_LOSS: Switch 1's power stack lost redundancy and is now operating in power sharing mode
Sep 1 14:43:42 BRANCH2 1025166: Sep 1 14:43:41.893 BST: %SEC-6-IPACCESSLOGP: list medical-in denied tcp 192.168.216.4(60715) -> 10.11.12.100(6812), 5 packets

2

It's not clear exactly what you're asking, but it sounds like you want to keep only some of the messages. Which messages are wanted and which are unwanted?

3

Sorry i may be mentioned wrong and confused here.

Lets explain detailed here

I have raw log storing in from all the devices

each vendor send in different format

Like

VPX

Sep 1 14:40:33 VPX-02 09/01/2018:13:40:33 GMT VPX-02 0-PPE-0 : default API CMD_EXECUTED 523995 0 : User nsroot - Remote_ip 192.168.255.47 - Command "stat ns" - Status "Success"
Sep 1 14:40:33 VPX-01 09/01/2018:13:40:33 GMT VPX-01 0-PPE-0 : default API CMD_EXECUTED 525164 0 : User nsroot - Remote_ip 192.168.255.47 - Command "stat ns" - Status "Success"

IAS Switch

Sep 1 14:40:34 SWIAS02 7187590: Sep 1 14:40:33.081 BST: %ILPOWER-5-IEEE_DISCONNECT: Interface Fa1/0/39: PD removed
Sep 1 14:43:41 SITEIAS01 599982: Sep 1 14:43:40.604 BST: %ADJ-5-RESOLVE_REQ_FAIL: Adj resolve request failed for 10.188.98.231 on Vlan498

Branch Switch

Sep 1 14:43:42 BRANCH01 BST: %PLATFORM_STACKPOWER-4-REDUNDANCY_LOSS: Switch 1's power stack lost redundancy and is now operating in power sharing mode
Sep 1 14:43:42 BRANCH2 1025166: Sep 1 14:43:41.893 BST: %SEC-6-IPACCESSLOGP: list medical-in denied tcp 192.168.216.4(60715) -> 10.11.12.100(6812), 5 packets

I would like to have output

Date, Device_src, message

as example below -

Sep 1 14:40:33 VPX-02 default API CMD_EXECUTED 523995 0 : User nsroot - Remote_ip 192.168.255.47 - Command "stat ns" - Status "Success"

Sep 1 14:40:34 SWIAS02 %ILPOWER-5-IEEE_DISCONNECT: Interface Fa1/0/39: PD removed

Sep 1 14:43:42 BRANCH01 %SEC-6-IPACCESSLOGP: list medical-in denied tcp 192.168.216.4(60715) -> 10.11.12.100(6812), 5 packets

hope this make sense i guess ?

appreciate your help.

4

Use a grok filter to parse your logs and extract the fields you want. The filter can list multiple grok expressions (each one matching the logs from a particular kind of device) that'll be tried in order until there's a match.

5

Appreciate if you can point me some example, i was try to find in forum and docs, may be missing some direction here..

6

See https://www.elastic.co/guide/en/logstash/current/config-examples.html. The grok filter documentation contains an example of how to list multiple grok expressions: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match

7

Thank you let me look and get back here any issue.

Appreciate your inputs