So my task was to install the metricbeat and enable the redis module to pick the INFO and Keyspace values. By providing that I was facing lot of issues mostly on TCP related. I have tried multiple ways to mitigate this but failed to resolve the issue. The error I was facing like below: [Masking the IP]
"message":"Error retrieving INFO stats: read tcp :35536->:6379: read: connection reset by peer","service.name":"metricbeat","ecs.version":"1.6.0"}
56},"message":"Error fetching data for metricset redis.keyspace: Failed to fetch redis info for keyspaces: read tcp :35536->:6379: read: connection reset by peer","se>
"message":"Error retrieving INFO stats: read tcp :35536->:6379: read: connection reset by peer","service.name":"metricbeat","ecs.version":"1.6.0"}
56},"message":"Error fetching data for metricset redis.info: failed to fetch redis info: read tcp :35536->:6379: read: connection reset by peer","service.name":
Now I have tried the another way to fetch the redis info details.
- Installed the filebat.
- wrote a shell script to fetch the redis details and store it to one output file and scheduled to crontab
- In filebeat I have given the path to fetch the output file.
So the idea is fetch the output file and send it to logstahs for parsing the logs. Now I have written the grok pattern and not able to parse properly.
Here is the format of the logs and filebat multiline pattern conf and logstash conf.
Filebeat:
multiline.pattern: '[1]'
multiline.negate: false
multiline.match: after
Lostash:
input {
beats {
port => "8"
}
}
filter {
if "redis_server_sit_1" in [tags] {
grok {
match => {
"message" => "# Server\sredis_version:%{DATA:redis_version}\sredis_git_sha1:%{NUMBER:redis_git_sha1}\sredis_git_dirty:%{NUMBER:redis_git_dirty}\sredis_build_id:%{WORD:redis_build_id}\sredis_mode:%{WORD:redis_mode}\sos:%{DATA:os}\sarch_bits:%{NUMBER:arch_bits}\smonotonic_clock:%{DATA:monotonic_clock}\smultiplexing_api:%{WORD:multiplexing_api}\satomicvar_api:%{DATA:atomicvar_api}\sgcc_version:%{DATA:gcc_version}\sprocess_id:%{NUMBER:process_id}\sprocess_supervised:%{WORD:process_supervised}\srun_id:%{DATA:run_id}\stcp_port:%{NUMBER:tcp_port}\sserver_time_usec:%{NUMBER:server_time_usec}\suptime_in_seconds:%{NUMBER:uptime_in_seconds}\suptime_in_days:%{NUMBER:uptime_in_days}\shz:%{NUMBER:hz}\sconfigured_hz:%{NUMBER:configured_hz}\slru_clock:%{NUMBER:lru_clock}\sexecutable:%{DATA:executable}\sconfig_file:%{DATA:config_file}\sio_threads_active:%{NUMBER:io_threads_active}\s*# Clients\sconnected_clients:%{NUMBER:connected_clients}\scluster_connections:%{NUMBER:cluster_connections}\smaxclients:%{NUMBER:maxclients}\sclient_recent_max_input_buffer:%{NUMBER:client_recent_max_input_buffer}\sclient_recent_max_output_buffer:%{NUMBER:client_recent_max_output_buffer}\sblocked_clients:%{NUMBER:blocked_clients}\stracking_clients:%{NUMBER:tracking_clients}\sclients_in_timeout_table:%{NUMBER:clients_in_timeout_table}\s*# Memory\sused_memory:%{NUMBER:used_memory}\sused_memory_human:%{DATA:used_memory_human}\sused_memory_rss:%{NUMBER:used_memory_rss}\sused_memory_rss_human:%{DATA:used_memory_rss_human}\sused_memory_peak:%{DATA:used_memory_peak}\sused_memory_peak_human:%{DATA:used_memory_peak_human}\sused_memory_peak_perc:%{DATA:used_memory_peak_perc}\sused_memory_overhead:%{NUMBER:used_memory_overhead}\sused_memory_startup:%{NUMBER:used_memory_startup}\sused_memory_dataset:%{NUMBER:used_memory_dataset}\sused_memory_dataset_perc:%{DATA:used_memory_dataset_perc}\sallocator_allocated:%{NUMBER:allocator_allocated}\sallocator_active:%{NUMBER:allocator_active}\sallocator_resident:%{NUMBER:allocator_resident}\stotal_system_memory:%{NUMBER:total_system_memory}\stotal_system_memory_human:%{DATA:total_system_memory_human}\sused_memory_lua:%{NUMBER:used_memory_lua}\sused_memory_lua_human:%{DATA:used_memory_lua_human}\sused_memory_scripts:%{NUMBER:used_memory_scripts}\sused_memory_scripts_human:%{WORD:used_memory_scripts_human}\snumber_of_cached_scripts:%{NUMBER:number_of_cached_scripts}\smaxmemory:%{NUMBER:maxmemory}\smaxmemory_human:%{DATA:maxmemory_human}\smaxmemory_policy:%{WORD:maxmemory_policy}\sallocator_frag_ratio:%{NUMBER:allocator_frag_ratio}\sallocator_frag_bytes:%{NUMBER:allocator_frag_bytes}\sallocator_rss_ratio:%{NUMBER:allocator_rss_ratio}\sallocator_rss_bytes:%{NUMBER:allocator_rss_bytes}\srss_overhead_ratio:%{NUMBER:rss_overhead_ratio}\srss_overhead_bytes:%{NUMBER:rss_overhead_bytes}\smem_fragmentation_ratio:%{NUMBER:mem_fragmentation_ratio}\smem_fragmentation_bytes:%{NUMBER:mem_fragmentation_bytes}\smem_not_counted_for_evict:%{NUMBER:mem_not_counted_for_evict}\smem_replication_backlog:%{NUMBER:mem_replication_backlog}\smem_clients_slaves:%{NUMBER:mem_clients_slaves}\smem_clients_normal:%{NUMBER:mem_clients_normal}\smem_aof_buffer:%{NUMBER:mem_aof_buffer}\smem_allocator:%{DATA:mem_allocator}\sactive_defrag_running:%{NUMBER:active_defrag_running}\slazyfree_pending_objects:%{NUMBER:lazyfree_pending_objects}\slazyfreed_objects:%{NUMBER:lazyfreed_objects}\s# Persistence\sloading:%{NUMBER:loading}\scurrent_cow_size:%{NUMBER:current_cow_size}\scurrent_cow_size_age:%{NUMBER:current_cow_size_age}\scurrent_fork_perc:%{NUMBER:current_fork_perc}\scurrent_save_keys_processed:%{NUMBER:current_save_keys_processed}\scurrent_save_keys_total:%{NUMBER:current_save_keys_total}\srdb_changes_since_last_save:%{NUMBER:rdb_changes_since_last_save}\srdb_bgsave_in_progress:%{NUMBER:rdb_bgsave_in_progress}\srdb_last_save_time:%{NUMBER:rdb_last_save_time}\srdb_last_bgsave_status:%{WORD:rdb_last_bgsave_status}\srdb_last_bgsave_time_sec:%{NUMBER:rdb_last_bgsave_time_sec}\srdb_current_bgsave_time_sec:%{NUMBER:rdb_current_bgsave_time_sec}\srdb_last_cow_size:%{NUMBER:rdb_last_cow_size}\saof_enabled:%{NUMBER:aof_enabled}\saof_rewrite_in_progress:%{NUMBER:aof_rewrite_in_progress}\saof_rewrite_scheduled:%{NUMBER:aof_rewrite_scheduled}\saof_last_rewrite_time_sec:%{NUMBER:aof_last_rewrite_time_sec}\saof_current_rewrite_time_sec:%{NUMBER:aof_current_rewrite_time_sec}\saof_last_bgrewrite_status:%{WORD:aof_last_bgrewrite_status}\saof_last_write_status:%{WORD:aof_last_write_status}\saof_last_cow_size:%{NUMBER:aof_last_cow_size}\smodule_fork_in_progress:%{NUMBER:module_fork_in_progress}\smodule_fork_last_cow_size:%{NUMBER:module_fork_last_cow_size}\s# Stats\stotal_connections_received:%{NUMBER:total_connections_received}\stotal_commands_processed:%{NUMBER:total_commands_processed}\sinstantaneous_ops_per_sec:%{NUMBER:instantaneous_ops_per_sec}\stotal_net_input_bytes:%{NUMBER:total_net_input_bytes}\stotal_net_output_bytes:%{NUMBER:total_net_output_bytes}\sinstantaneous_input_kbps:%{NUMBER:instantaneous_input_kbps}\sinstantaneous_output_kbps:%{NUMBER:instantaneous_output_kbps}\srejected_connections:%{NUMBER:rejected_connections}\ssync_full:%{NUMBER:sync_full}\ssync_partial_ok:%{NUMBER:sync_partial_ok}\ssync_partial_err:%{NUMBER:sync_partial_err}\sexpired_keys:%{NUMBER:expired_keys}\sexpired_stale_perc:%{NUMBER:expired_stale_perc}\sexpired_time_cap_reached_count:%{NUMBER:expired_time_cap_reached_count}\sexpire_cycle_cpu_milliseconds:%{NUMBER:expire_cycle_cpu_milliseconds}\sevicted_keys:%{NUMBER:evicted_keys}\skeyspace_hits:%{NUMBER:keyspace_hits}\skeyspace_misses:%{NUMBER:keyspace_misses}\spubsub_channels:%{NUMBER:pubsub_channels}\spubsub_patterns:%{NUMBER:pubsub_patterns}\slatest_fork_usec:%{NUMBER:latest_fork_usec}\stotal_forks:%{NUMBER:total_forks}\smigrate_cached_sockets:%{NUMBER:migrate_cached_sockets}\sslave_expires_tracked_keys:%{NUMBER:slave_expires_tracked_keys}\sactive_defrag_hits:%{NUMBER:active_defrag_hits}\sactive_defrag_misses:%{NUMBER:active_defrag_misses}\sactive_defrag_key_hits:%{NUMBER:active_defrag_key_hits}\sactive_defrag_key_misses:%{NUMBER:active_defrag_key_misses}\stracking_total_keys:%{NUMBER:tracking_total_keys}\stracking_total_items:%{NUMBER:tracking_total_items}\stracking_total_prefixes:%{NUMBER:tracking_total_prefixes}\sunexpected_error_replies:%{NUMBER:unexpected_error_replies}\stotal_error_replies:%{NUMBER:total_error_replies}\sdump_payload_sanitizations:%{NUMBER:dump_payload_sanitizations}\stotal_reads_processed:%{NUMBER:total_reads_processed}\stotal_writes_processed:%{NUMBER:total_writes_processed}\sio_threaded_reads_processed:%{NUMBER:io_threaded_reads_processed}\sio_threaded_writes_processed:%{NUMBER:io_threaded_writes_processed}\s*# Replication\srole:%{WORD:role}\smaster_host:%{IP:master_host}\smaster_port:%{NUMBER:master_port}\smaster_link_status:%{WORD:master_link_status}\smaster_last_io_seconds_ago:%{NUMBER:master_last_io_seconds_ago}\smaster_sync_in_progress:%{NUMBER:master_sync_in_progress}\sslave_read_repl_offset:%{NUMBER:slave_read_repl_offset}\sslave_repl_offset:%{NUMBER:slave_repl_offset}\sslave_priority:%{NUMBER:slave_priority}\sslave_read_only:%{NUMBER:slave_read_only}\sreplica_announced:%{NUMBER:replica_announced}\sconnected_slaves:%{NUMBER:connected_slaves}\smaster_failover_state:%{DATA:master_failover_state}\smaster_replid:%{WORD:master_replid}\smaster_replid2:%{WORD:master_replid2}\smaster_repl_offset:%{NUMBER:master_repl_offset}\ssecond_repl_offset:%{NUMBER:second_repl_offset}\srepl_backlog_active:%{NUMBER:repl_backlog_active}\srepl_backlog_size:%{NUMBER:repl_backlog_size}\srepl_backlog_first_byte_offset:%{NUMBER:repl_backlog_first_byte_offset}\srepl_backlog_histlen:%{NUMBER:repl_backlog_histlen}\s# CPU\sused_cpu_sys:%{NUMBER:used_cpu_sys}\sused_cpu_user:%{NUMBER:used_cpu_user}\sused_cpu_sys_children:%{NUMBER:used_cpu_sys_children}\sused_cpu_user_children:%{NUMBER:used_cpu_user_children}\sused_cpu_sys_main_thread:%{NUMBER:used_cpu_sys_main_thread}\sused_cpu_user_main_thread:%{NUMBER:used_cpu_user_main_thread}\s*# Modules\s*# Errorstats\serrorstat_ERR:count=%{NUMBER:errorstat_ERR_count}\s# Cluster\scluster_enabled:%{NUMBER:cluster_enabled}\s# Keyspace"
}
}
mutate { add_tag => "redis_logs" }
}
}
output {
if "redis_logs" in [tags] {
elasticsearch
{
hosts => ["host1:9200","host2:9200","host3:9200"]
ssl => true
user => "user"
password => "pass"
truststore => "/etc/ela12.ts"
truststore_password => "pass"
ssl_certificate_verification => true
ilm_rollover_alias => "redis-logs"
ilm_pattern => "{now/d}-000001"
ilm_policy => "redis-logs"
}
}
}
the above one is configuration. but got the grok parsing error but in grok debugger it is coming perfectly fine.
the log pattern is:
Server
redis_version:6.2.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:5d88ce217879027a
redis_mode:standalone
os:Linux 4.18.0-477.15.1.el8_8.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.5.0
process_id:3297159
process_supervised:systemd
run_id:47c587c5f86a9d37b780c4de6bbcf4656a8d4cab
tcp_port:6379
server_time_usec:1703757767880238
uptime_in_seconds:1466763
uptime_in_days:16
hz:10
configured_hz:10
lru_clock:9258951
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
Clients
connected_clients:16
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
Memory
used_memory:2325192
used_memory_human:2.22M
used_memory_rss:17420288
used_memory_rss_human:16.61M
used_memory_peak:2734208
used_memory_peak_human:2.61M
used_memory_peak_perc:85.04%
used_memory_overhead:2169768
used_memory_startup:813120
used_memory_dataset:155424
used_memory_dataset_perc:10.28%
allocator_allocated:2389120
allocator_active:2850816
allocator_resident:5304320
total_system_memory:6192857088
total_system_memory_human:5.77G
used_memory_lua:35840
used_memory_lua_human:35.00K
used_memory_scripts:528
used_memory_scripts_human:528B
number_of_cached_scripts:2
maxmemory:1073741824
maxmemory_human:1.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.19
allocator_frag_bytes:461696
allocator_rss_ratio:1.86
allocator_rss_bytes:2453504
rss_overhead_ratio:3.28
rss_overhead_bytes:12115968
mem_fragmentation_ratio:7.57
mem_fragmentation_bytes:15117888
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:307544
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:38
rdb_bgsave_in_progress:0
rdb_last_save_time:1703756924
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:536576
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
Stats
total_connections_received:122
total_commands_processed:9309604
instantaneous_ops_per_sec:6
total_net_input_bytes:704769007
total_net_output_bytes:3921457572
instantaneous_input_kbps:0.55
instantaneous_output_kbps:4.32
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:1
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:683
total_forks:553
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:199
dump_payload_sanitizations:0
total_reads_processed:9080383
total_writes_processed:21023215
io_threaded_reads_processed:0
io_threaded_writes_processed:0
Replication
role:slave
master_host:IP
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:2676731963
slave_repl_offset:2676731963
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:75c2674d500152ef1e92f72065940c8c9292e2f5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2676731963
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2675683388
repl_backlog_histlen:1048576
CPU
used_cpu_sys:1057.229563
used_cpu_user:1217.442233
used_cpu_sys_children:0.921917
used_cpu_user_children:0.246200
used_cpu_sys_main_thread:1049.807073
used_cpu_user_main_thread:1211.922890
Modules
Errorstats
errorstat_ERR:count=199
Cluster
cluster_enabled:0
Keyspace
Now what is happening by doing so...
In grok debugger the above one is parsing correctly.
But the problem is happening every line is treated as a new event.
Filebat does not able to send the whole logs at a time to logstash for parsing.
It is sending like
Server
as a different event
redis_version:6.2.7
as a different event
redis_git_sha1:00000000
as a different event
redis_git_dirty:0
as a different event
redis_build_id:5d88ce217879027a
as a different event
redis_mode:standalone
as a different event
And in kibana I can see every line in a every document.
like # Server is a one document
like redis_version:6.2.7 is a one document
like redis_git_sha1:00000000 is a one document
This is how 174 documents are generating instead having one document
# ↩︎