Hi,
I had run a software which generates logs using log4j and saves them in a file ept.log the format is as follows
launch TIMESTAMP START: 2016.11.15 05:30:23
Level: Message
INFO: Batch lock retry wait = 30, max = 720
INFO: Launching Batch Validation
INFO: Timestamp: Nov 15,2016 05:30:24 AM
INFO: rules from properties null
INFO: ... Checking rules requested against rule info from the database
INFO: Starting run on oracle.jdbc.driver.OracleDriver jdbc:oracle:thin:@tch:1521:tch WEU_MAP_IW *******...
AUDIT: opening connection to user:CLONE_WSM url:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=tch)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=tch)))
INFO: com.xyz.dataserver.jdbcserver.JdbcDatasetReader
INFO: create desc com.xyz.workspace.DBDescriptor: name=app, dataset=Url: jdbc:oracle:thin:@tch:1521:tch, driver: oracle.jdbc.driver.OracleDriver, User: WEU_MAP_IW, connection tag: dbConnectionString, converage=<CoverageDescriptor: extent=<com.xyz.dataserver.extent.RegionExtent region code:F1 region-to-admin query?:true region-to-carto query?:false loadBufferData?:true>, partialpolys=[], allCartoIds=[], onlyAdmins=false, includeChildrenInHierarchy=true, dbversion=0, countrystate=false, admincartoonly=false, longhaul=false, xzipcity=false, xpostal=false, crfs=[], queryMetadataOnly=false, (obs) loadAllMetadata=true, loadLocalMetadata=true, loadGlobalMetadata=true, loadTrafficData=true, loadTpegData=true, loadPhoneticData=false, loadGeoOverrideData=false, loadHistoryData=false, loadTopologySegmentData=false, queryAllLinks=true, queryLinkLimit=25000, deltas=false, unintDeltasOnly=false, unintDelta=false, unintDeltaPrev=false, synch=false, synchPrev=false, synchPrevPS=false, taskCode=0, scopeRectangles=null, syncType=null, versionToIgnoreForSync=0, allRoadNameIds=[], unrestrictedReadAllQuery=false, loadStatisticalAreas = false, loadBasicHeightData=false, filterLevel=NONE>
INFO: Creating Loader & Reader (bypassing app-server)
INFO: Loading an RMOB
INFO: creating the Reader
INFO: get info for Reader from properties
INFO: create a local Reader
INFO: set up the Reader by calling setFromReaderDescriptor()
INFO: Reader created
INFO: QUERY COUNT CartoFaces : 224815
INFO: 16950000 packets. Current packet is class com.xyz.kernel.flat.FlatPostalAreaAdminPlace [Loader] [tstamp: 3968 time: 3952.5 (1:05:52) split: 01.7] [Memory: 16.06G total: 22.20G free: 6.139G]
INFO: com.xyz.val.validation.BatchRoadPointValidation Validation statistics: 2 rule(s) applied, 0 exception(s) occurred (0 out of scope)
INFO: [BatchValidationCommand] [tstamp: 4992 time: 39.1 split: 00.0] [Memory: 18.95G total: 37.49G free: 18.54G]
INFO: Done executing all batch validations. [ValidationService] [tstamp: 4992 time: 38.7 split: 38.7] [Memory: 18.95G total: 37.49G free: 18.54G]
[save rule violations] [tstamp: 4998 time: 4997.6 (1:23:17) split: 4997.6 (1:23:17)] [Memory: 23.25G total: 37.49G free: 14.24G]
INFO:
INFO: Rule Code Rule Status
INFO: ---------- --------------------------------------------------------------------------- ------
INFO: PAD032 Point Address/Micro Point Address Consists of Illegal Characters OK
INFO: PRB046 Point Address Crossing Product Boundary OK
INFO: RDS069 Location Table Number Not Valid for Country or Admin OK
INFO:
INFO: SUMMARY REPORT
INFO:
INFO: BATCH VALIDATION CLASS NUM OF RULES NUM OF EXCEPTIONS
INFO: ------------------------------------------------------ ------------ ------------------
INFO: BatchCartoLinkValidation 1 0
INFO: BatchRoadPointValidation 2 0
INFO: TOTAL: 3 0
INFO: Batch Validation final status: SUCCESS
launch TIMESTAMP END: 2016.11.15 06:53:45
now how shall I extract data from it to visualize and update my config files and filters of logstash
so that if able to extract important metrics like Memory consumption at a particular timestamp or CPU usage or maybe you guys can help me figure out other aspects of it?