Persistence Queue Implimentation configuration

Elastic Stack Logstash
1

Hi Team,

I am trying to implement the PQ in the Logstash.
Step 1: syslog-ng is installed in multiple servers and logs are forwarded to centralized Logstash.
Step 2: Logstash is installed in centralized server and it will receive the log messages from the syslog-ng.
Step 3: Logstash filter the logs and stored in the disc.

Flow is : Syslog-ng --> Logstash --> Disc

For the above system I am implementing the Logstash Persistence Queue .
Then Flow will be as below:
Syslog-ng --> Logstash Persistence Queue-> Disc -->Logstash --> Disc

Sample LS PQ configuration:
path.data: /var/lib/logstash
queue.type: persisted
queue.max_bytes: 1024mb
queue.page_capacity: 250mb
queue.max_events: 0
To implement the LS PQ, configured the "logstash.yml" file as per the document and it is created the "queue" folder as "/var/lib/logstash/queue". Inside this folder I can see the below files.
a. checkpoint.head
b. page.0 (and its size is 250MB)
page.0 contains the below kind of log data for ex:
dtypefsyslogÿÿ¬NÇÆ A¿dMETA¿ÿdDATA¿j@timestampx2017-07-07T04:28:47.844Zh@versiona1dhostl10.13.106.67gmessagexÙSent:2017-07-07T00:30:57.000-04:00 Recv:2017-07-07T00:30:57.124-04:00 [local7] [info] Z1-ACCM.ipc.com lib_cli_nss[11318]: src/nss_ipc_usr_info.c getCliUserInfoByUid 937 getCliUserInfoByUid Curl get info failed : 404

Questions:

  1. How to confirm my persistence queue implementation is proper?
  2. It is creating the queue but why not it is not deleting the page.0 and checkpoint.head after reading it from the queue?
  3. Queue should contains some called page.N, one called checkpoint.head and some called checkpoint.N.
    But in my case I can't see a file like checkpoint.N, why it is not generating the checkpoint.N files?
    N is a number
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.