Tek_Chand
August 3, 2018, 10:12am
1
Hello All,
I have single file of logs in which some logs are single line and some logs are multiline logs. I have wrote the logstash filter for single line log and its working fine.
But i am facing the problem while try to parse the multiline logs. So first of all i want to handle the multiline logs at filebeat level.
Below are the sample log of my multiline logs. Please help me how i can manage it at filebeat level.
I, [2018-08-02T06:45:57.333437 #2352] INFO -- : [8d1bee5c-3240-4916-944f-b6473b29936e] request body: <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CommandUUID</key>
<string>63987</string>
<key>QueryResponses</key>
<dict>
<key>AvailableDeviceCapacity</key>
<real>24.996974945068359</real>
<key>BatteryLevel</key>
<real>1</real>
</dict>
<key>Status</key>
<string>Acknowledged</string>
<key>UDID</key>
<string>c2c0fc6ecbd021ad4a9d1faae070d61e73b94236</string>
</dict>
</plist>
Thanks in advance.
kvch
August 3, 2018, 3:12pm
2
Try the following config:
multiline.pattern: </plist>
multiline.negate: true
multiline.match: before
Tek_Chand
August 6, 2018, 4:57am
3
Hello Noemi,
Thank you for your response.
I have tried this pattern, but its not working for me.
When i am using this pattern its combining many single line logs with multiline logs which are appear on dashboard.
Please help me.
kvch
August 6, 2018, 5:33am
4
Could you please share more example logs both logs which needs to be multiline and single line?
Tek_Chand
August 6, 2018, 5:46am
5
Hello Noemi,
As requested by you please find the sample logs for single line and multilines:
For Single Line:
I, [2018-08-04T07:10:46.039686 #15053] INFO -- : [Sidekiq::Extensions::DelayedClass 5d5c7adbf4d8b01bee221108] End at: 2018-08-04 07:10:46 -0400 :: Time taken: 0
I, [2018-08-04T07:10:46.103794 #6637] INFO -- : send_device_inactivity_emails :: finding inactive devices for duration: twenty_four_hours
E, [2018-08-04T07:10:46.107704 #6633] ERROR -- : api failed :: bad request: get-/account :: body: nil :: error: 401 Unauthorized: Unauthorized
I, [2018-08-04T07:10:46.110869 #6637] INFO -- : send_device_inactivity_emails :: inactive device count for duration: twenty_four_hours :: 0
I, [2018-08-04T07:10:46.118054 #6637] INFO -- : send_device_inactivity_emails :: finding inactive devices for duration: seventy_two_hours
I, [2018-08-04T07:10:46.125281 #6637] INFO -- : send_device_inactivity_emails :: finding inactive devices for duration: one_week
I, [2018-08-04T07:10:46.126548 #6637] INFO -- : send_device_inactivity_emails :: finding inactive devices for duration: five_mins
I, [2018-08-04T07:10:46.130964 #6637] INFO -- : send_device_inactivity_emails :: inactive device count for duration: five_mins :: 3
I, [2018-08-04T07:10:46.151704 #6637] INFO -- : with_device_properties_lock :: acquired lock in: 0.0032452046871185303 seconds
I, [2018-08-04T07:10:46.154688 #6637] INFO -- : send_device_inactivity_emails :: creating device activity logs for device ids: [] - START
I, [2018-08-04T07:10:46.154751 #6637] INFO -- : send_device_inactivity_emails :: device activity logs created for device ids: [] - END
I, [2018-08-04T07:10:46.157323 #15053] INFO -- : [Sidekiq::Extensions::DelayedClass b0bbdbc03dee4586ab581ac3] Start at: 2018-08-04 07:10:46 -0400: {"class"=>"Sidekiq::Extensions::DelayedClass", "args"=>["---\n- !ruby/class 'Device'\n- :notify_devices_inactive\n- - []\n - 1533381046000\n"], "retry"=>true, "queue"=>"default", "jid"=>"d1fa5cab5535a7fcc7124f34", "created_at"=>1533381046.155595, "enqueued_at"=>1533381046.1556513}
For Multiline:
I, [2018-08-04T07:05:44.183266 #14492] INFO -- : [279fdc58-7d63-49c7-b0d0-a41a9bb411a8] sending command to device: 13565 :: <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict><key>Command</key><dict><key>RequestType</key><string>DeviceInformation</string><key>Queries</key><array><string>DeviceName</string><string>OSVersion</string><string>BuildVersion</string><string>ModelName</string><string>Model</string><string>SerialNumber</string><string>DeviceCapacity</string><string>AvailableDeviceCapacity</string><string>BatteryLevel</string><string>CellularTechnology</string><string>ModemFirmwareVersion</string><string>IsSupervised</string><string>IsDeviceLocatorServiceEnabled</string><string>IsActivationLockEnabled</string><string>IsDoNotDisturbInEffect</string><string>DeviceID</string><string>EASDeviceIdentifier</string><string>IsCloudBackupEnabled</string><string>OSUpdateSettings</string><string>LocalHostName</string><string>HostName</string><string>SystemIntegrityProtectionEnabled</string><string>ActiveManagedUsers</string><string>IsMDMLostModeEnabled</string><string>MaximumResidentUsers</string><string>iTunesStoreAccountIsActive</string><string>iTunesStoreAccountHash</string><string>ICCID</string><string>BluetoothMAC</string><string>WiFiMAC</string><string>CurrentCarrierNetwork</string><string>SIMCarrierNetwork</string><string>SubscriberCarrierNetwork</string><string>CarrierSettingsVersion</string><string>PhoneNumber</string><string>VoiceRoamingEnabled</string><string>DataRoamingEnabled</string><string>IsRoaming</string><string>PersonalHotspotEnabled</string><string>SubscriberMCC</string><string>SubscriberMNC</string><string>CurrentMCC</string><string>CurrentMNC</string></array></dict><key>CommandUUID</key><string>64491</string></dict></plist>
I, [2018-08-04T07:05:46.488135 #14492] INFO -- : [0d433cfc-c5d4-48ad-8b58-d1f9c2da6514] request body: <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CommandUUID</key>
<string>64491</string>
<key>QueryResponses</key>
<dict>
<key>AvailableDeviceCapacity</key>
<real>24.441822052001953</real>
<key>BatteryLevel</key>
<real>0.60000002384185791</real>
<key>BluetoothMAC</key>
<string>a8:5c:2c:e6:26:23</string>
<key>BuildVersion</key>
<string>15F79</string>
<key>CarrierSettingsVersion</key>
<string>32.1</string>
<false/>
<key>IsMDMLostModeEnabled</key>
<false/>
<key>DeviceCapacity</key>
<real>26.619827270507812</real>
<key>DeviceName</key>
<string>iPhone</string>
<key>EASDeviceIdentifier</key>
<string>IRK2QD1FDL27T7RQ5NMF21SUMK</string>
<key>IsActivationLockEnabled</key>
<false/>
<key>IsCloudBackupEnabled</key>
<false/>
<key>IsDeviceLocatorServiceEnabled</key>
<false/>
<key>IsDoNotDisturbInEffect</key>
<false/>
<key>IsMDMLostModeEnabled</key>
<false/>
<key>IsRoaming</key>
<false/>
<key>IsSupervised</key>
<true/>
<key>Model</key>
<string>MQ3D2HN</string>
<key>ModelName</key>
<string>iPhone</string>
<key>ModemFirmwareVersion</key>
<string>6.60.00</string>
<key>OSVersion</key>
<string>11.4</string>
<key>PersonalHotspotEnabled</key>
<false/>
<key>SIMCarrierNetwork</key>
<string>AirTel</string>
<key>SerialNumber</key>
<string>FFPW3ANEHXR5</string>
<key>SubscriberCarrierNetwork</key>
<string>AirTel</string>
<key>SubscriberMCC</key>
<string></string>
<key>SubscriberMNC</key>
<string></string>
<key>VoiceRoamingEnabled</key>
<false/>
<key>WiFiMAC</key>
<string>a8:5c:2c:e6:26:22</string>
<key>iTunesStoreAccountIsActive</key>
<false/>
</dict>
<key>Status</key>
<string>Acknowledged</string>
<key>UDID</key>
<string>c2c0fc6ecbd021ad4a9d1faae070d61e73b94236</string>
</dict>
</plist>
kvch
August 6, 2018, 8:35am
6
This should separate your messages correctly:
multiline.pattern: '[A-Z]{1}, \[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
Let me know if it works.
1 Like
Tek_Chand
August 6, 2018, 9:32am
7
Hello Noemi,
Thanks a lot. It working fine for my messages.
system
September 3, 2018, 9:32am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.