Hi there,
I am trying to visualize logs on Kibana from a log file which is updating in real time.
The log file is splitting in each minuite generating a new log file with the name of the
particular minute. I have written a bash script for this splitting of logs after each minute as follows.
fileCreator.sh file
#!/bin/bash
#execute at the time of creating the request-response-logger.log file
file="request-response-logger";
current_date_time="`date "+%Y-%m-%d %H:%M"`";
sleep 5
#this part is to extract and move the logs recorded in time between the start of generation og the main log file and the time of executing the script.
sed -i -e '/' "$current_date_time"'/{w '"$file-$current_date_time before'' ''-e 'd}' $file
echo "splitted logs on" $current_date_time "before"
sleep 1m
while [ -s $file ]
do
before_time="`date "+%Y-%m-%d %H:%M" -d "1 min ago"`"
sed -i -e '/'"$before_time"'/{w '"$file-$before_time"'' -e 'd}' $file
echo "splitted logs on" $before_time
sleep 1m
done
Executing of this script will generate log files in each minute as follows.
The main log file (request-response-logger.log) has been pointed as the input file to filebeat in filebeat.yml.
When loaded in to kibana, at every time logs recorded in the last minute are duplicating. That means two logs with the same id are recorded in Kibana for the logs in the log file generated in the last minute only.I tried several times with altering the functions in the script file but none of them worked out. The resulting issue is as follows.
Please help me to resolve this.
Filebeat, Logstash, Elasticsearch, Kibana versions : 6.4.0
