Different multilines in one input

Elastic Stack Logstash
1

hi everyone,I have a problem here about multiline plugin,I have different log data in one file,and need use multiline to collapse multiline messages and merge them into a single event.
my code is:
input {
file {
path => "/home/logs/log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^=.*|^\sSlot|^\s+Power|^\s+State|^\s+Type|^\s+FAN"
what => "previous"
}
codec => multiline {
pattern => "^.SlotNo\s+Temperature."
what => "next"
}
}
}
But just the first multiline is success, the 2rd multiline is failed,the event just show the first line why this happened and how to achieve 2 or more different multilines ?
thanks

2

Hi,

you can use only one codec per file. Just set input per file, something like :

input {
    file {
      path => "/home/logs/log-one"
      start_position => "beginning"
      sincedb_path => "/mydir/sincedb-log-two"
      codec => multiline {
      pattern => "^=.*|^\sSlot|^\s+Power|^\s+State|^\s+Type|^\s+FAN"
      what => "previous"
   }
   file {
       path => "/home/logs/log-two"
       start_position => "beginning"
       sincedb_path => "/mydir/sincedb-log-two"
       codec => multiline {
       pattern => "^.SlotNo\s+Temperature."
       what => "next"
   }
}

Are you sure that sincedb_path to /dev/null ist a good idea, you will get file readed again if logstash is restarted.

pts0

1 Like
3

Thanks a lot ,that is work.But in my file ,lots of invalid data and duplicated data (log line) in one file, and a lot of data i need just one line,i know drop ,but still can not make my data clean , how to clean data use logstash, do you have some experice?

4

You can do really a lot of processing with logstash.
Can you please let us know what exactly you like to do ... is difficult to help you if you just tell duplicate data and we don't know ho to filter and "clean" you stream.

pts0

5

different formats in a log file:

`==================================================
===============display device===============

                  ^

% Unrecognized command found at '^' position.

===============display current-configuration===============

                  ^

% Unrecognized command found at '^' position.

===============display saved-configuration===============

                  ^

% Unrecognized command found at '^' position.

===============display interface===============

Bridge-Aggregation1 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 000f-e207-f2e0
Description: Bridge-Aggregation1 Interface
2Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 10
Port link-type: trunk
VLAN passing : 10
VLAN permitted: 10
Trunk port encapsulation: IEEE 802.1q

============================================================
name Tid Vid TSize Mod priority Status Total/Max/Last(Millsecs)

VFS 566a000 1 40 N 100 queblock 0/ 0/ 0
CBU 4b2c00 2 40 N 100 queblock 0/ 0/ 0
VIDL 4b2800 3 16 P 1 preemptready 2313520790/ 10/ 0
TICK 4b2400 4 16 P 250 preemptready 396116122/ 2/ 0
STMR 4b2000 5 40 N 150 eventblock 41399858/ 1/ 0
dGDB 4b3c00 6 40 N 180 eventblock 0/ 0/ 0
SYSM 4b3800 7 40 N 100 eventblock 36870803/ 3/ 0
LED 4b3400 8 40 N 180 eventblock 0/ 0/ 0
PORT 4b3000 9 40 N 180 eventblock 11880332/ 1/ 0
bDPC 4b4c00 10 16 N 95 semblock 0/ 0/ 0
L2X0 4b4800 11 16 N 55 semblock 434658212/ 11/ 0
bC.0 4b4400 12 16 N 55 semblock 54477031/ 3/ 2
bTX 4b4000 13 16 N 140 semblock 0/ 0/ 0
bATX 4b5c00 14 16 N 140 semblock 0/ 0/ 0
bLK0 4b5800 15 16 N 150 semblock 229731003/ 8/ 0
DQFD 4b5400 16 40 N 100 sleep 39925187/ 2/ 0
DQIT 4b5000 17 40 N 100 eventblock 85279714/ 2/ 0
LPDT 4b6c00 18 40 N 100 sleep 163768/ 1/ 0

%May 23 15:42:22:778 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System synchronization source lost!
%May 23 15:47:46:920 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System leap changes from 3 to 0 after clock update.
%May 23 15:47:46:920 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System stratum changes from 16 to 12 after clock update.
%May 23 15:57:34:824 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System stratum changes from 12 to 13 after clock update.
%May 23 16:05:13:443 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System stratum changes from 13 to 12 after clock update.
%May 23 16:09:32:86 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System stratum changes from 12 to 2 after clock update.
%May 23 16:11:41:385 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System synchronization source lost!
%May 23 16:16:02:916 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System leap changes from 3 to 0 after clock update.
%May 23 16:16:02:916 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System stratum changes from 16 to 12 after clock update.
%May 23 16:26:52:451 2017 GM-KP-5500-1 NTP/4/NTP_LOG:
System stratum changes from 12 to 13 after clock update.
......

I need some information from those file,so I need extract data from it. Because logstash read every line to a event, so lots of data is useless,and same device name could appear multiline so it is duplicated.That is why I need drop some data to make it "clean".And when i use this data, i need merge multiline data in a event,but some data may not be adjacent,i don't know how to merge this kind of data.
I am sorry if i made you confused.
Thanks a lot

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.