HI all,
I have deleted my default .yml file and updated with this configuration and it worked.
thanks for your help all of you.
filebeat.prospectors:
-
paths:
- E:\ELK-STACK\logstash-tutorial-dataset.log
input_type: log
document_type: apachelogs
-
paths:
- C:\Logs\GatewayService\GatewayService-Processor.Transactions-20170810.slog
input_type: log
document_type: gatewaylogs
-
paths:
- E:\automation frameowrk\AUTO_JUL27_Debit_Purchase_Ig3_EditError\FirstData-BuyPass-HC.log
input_type: log
document_type: authenticlogs
output.logstash:
hosts: ["localhost:5043"]
now i have one more question is that i have a unstructured authentic(our project) logs which doesn't follow same pattern so how may i apply the pattern on these type of logs?
sample of the logs is:
07:35:19.119 TRACE [SocketConnectionHandler:ListProc-Q0:I5] [UID:SER22VVM211:CPBridge:201707270735180832:000000000000000001, Message Timestamp:2017-07-27 07:35:18.832] Writing raw message to socket
07:35:19.119 DEBUG [ListProcessor:ListProc-Q0:I5] Action [AcquirerRequest.AuxRequest] executed with result [TERMINATE_ALL(2)]
07:35:19.119 DEBUG [ListMgr:SocketConnection-Executor] Scheduling task .... priority=false
07:35:19.120 TRACE [ListMgr:SocketConnection-Executor] Message queued into List Processor queue
07:35:19.120 DEBUG [SocketConnectionHandler:ListProc-Q0:I0] connection:/SER22VVM211/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC%231: WriteAcknowledgedEvent received on FirstData-BuyPass-HC#1
07:35:19.120 TRACE [ConnectionStatisticalInfo:ListProc-Q0:I0] BucketPeriod : 10 Seconds, 1 ms starting at Thu Jul 27 07:35:10 EDT 2017 will be retained until Thu Jul 27 07:36:00 EDT 2017
07:35:19.265 TRACE [SocketConnection:SelectorHandler-0] Message received with MLI : 119
07:35:19.268 DEBUG [ListMgr:SocketConnection-Executor] Scheduling task .... priority=false
07:35:19.268 TRACE [ListMgr:SocketConnection-Executor] Message queued into List Processor queue
07:35:19.268 WARN [SocketConnectionHandler:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] *****SENSITIVE DATA DISPLAYED CLEAR AT THIS LEVEL - THIS MAY INVALIDATE PCI DSS COMPLIANCE, RUNNING AT THIS LEVEL MAY SLOW DOWN THE SYSTEM*******
07:35:19.268 INFO [SocketConnectionHandler:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] connection:/SER22VVM211/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC%231: Message received from FirstData-BuyPass-HC#1
07:35:19.268 DEBUG [SocketConnectionHandler:ListProc-Q0:I4] Acquirer data cannot be retrieved with Acquirer ID[-1]
07:35:19.269 TRACE [SocketConnectionHandler:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] connection:/SER22VVM211/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC%231
Receiving incoming message from java.nio.channels.SocketChannel[connected local=/153.71.66.148:22734 remote=/153.61.241.191:7736]
0000:000000: 41 52 50 33 30 30 39 30 31 30 32 30 30 20 39 30 | ARP3009010200 90 |
0010:000016: 30 36 30 30 33 30 30 30 30 30 30 30 31 30 20 20 | 06003000000010 |
0020:000032: 20 20 20 20 30 37 33 35 31 33 30 34 30 31 31 31 | 073513040111 |
0030:000048: 39 30 30 37 30 30 37 30 30 37 31 20 20 20 20 20 | 90070070071 |
0040:000064: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
0050:000080: 20 20 20 45 32 47 45 44 49 54 20 45 52 3A 4F 50 | E2GEDIT ER:OP |
0060:000096: 54 20 44 41 54 41 20 20 20 20 20 20 20 31 34 20 | T DATA 14 |
0070:000112: 20 39 30 30 30 30 30 | 900000 |
07:35:19.272 TRACE [LengthVar:ListProc-Q0:I4] Optional Area : DataLength = 0 Max Data Length = 999
07:35:19.273 DEBUG [MsgParser:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773]
Parsed Incoming Body Fields :
EXT-ID[---] FLD[Device ID] FRMT[Fixed] LL[0] LEN[11] DATA[3009010200 ]
EXT-ID[---] FLD[Sequence Number] FRMT[Fixed] LL[0] LEN[6] DATA[900600]
EXT-ID[---] FLD[Transaction Code] FRMT[Fixed] LL[0] LEN[2] DATA[30]
EXT-ID[---] FLD[Auth Amount] FRMT[Fixed] LL[0] LEN[8] DATA[00000010]
EXT-ID[---] FLD[Posting Date] FRMT[Fixed] LL[0] LEN[6] DATA[ ]
EXT-ID[---] FLD[Transaction Time] FRMT[Fixed] LL[0] LEN[6] DATA[073513]
EXT-ID[---] FLD[Manual Entry Flag] FRMT[Fixed] LL[0] LEN[1] DATA[0]
EXT-ID[---] FLD[Card Data] FRMT[Fixed] LL[0] LEN[40] DATA[4011190070070071 ]
EXT-ID[---] FLD[Action Code] FRMT[Fixed] LL[0] LEN[1] DATA[E]
EXT-ID[---] FLD[Response Code] FRMT[Fixed] LL[0] LEN[2] DATA[2G]
EXT-ID[---] FLD[Terminal Display] FRMT[Fixed] LL[0] LEN[16] DATA[EDIT ER:OPT DATA]
EXT-ID[---] FLD[User Data] FRMT[Fixed] LL[0] LEN[7] DATA[ ]
EXT-ID[---] FLD[Authorizer Network ID] FRMT[Fixed] LL[0] LEN[2] DATA[14]
EXT-ID[---] FLD[Authorizer Network Res..] FRMT[Fixed] LL[0] LEN[2] DATA[ ]
EXT-ID[---] FLD[Version Number] FRMT[Fixed] LL[0] LEN[3] DATA[900]
EXT-ID[---] FLD[Optional Area] FRMT[LLLVAR Group] LL[3] LEN[0] DATA[]
07:35:19.273 DEBUG [Validator:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773]
VALIDATION:
07:35:19.274 DEBUG [MsgMapper:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773]
MAPPING:
SOURCE INT[FirstData-BuyPass-HC] MSG[Authorization Response] SEG[Authorization Response]
DESTINATION INT[Authentic] MSG[Generic Message] SEG[Generic Body]
07:35:19.275 DEBUG [MsgMapper:ListProc-Q0:I4] [UID:N/A, Message Timestamp:2017-07-27 07:35:15.773]
please reply