Failed to Fetching the Redis Info and keyspace logs to Kibana

So my task was to install the metricbeat and enable the redis module to pick the INFO and Keyspace values. By providing that I was facing lot of issues mostly on TCP related. I have tried multiple ways to mitigate this but failed to resolve the issue. The error I was facing like below: [Masking the IP]

"message":"Error retrieving INFO stats: read tcp :35536->:6379: read: connection reset by peer","service.name":"metricbeat","ecs.version":"1.6.0"}
56},"message":"Error fetching data for metricset redis.keyspace: Failed to fetch redis info for keyspaces: read tcp :35536->:6379: read: connection reset by peer","se>
"message":"Error retrieving INFO stats: read tcp :35536->:6379: read: connection reset by peer","service.name":"metricbeat","ecs.version":"1.6.0"}
56},"message":"Error fetching data for metricset redis.info: failed to fetch redis info: read tcp :35536->:6379: read: connection reset by peer","service.name":

Now I have tried the another way to fetch the redis info details.

  1. Installed the filebat.
  2. wrote a shell script to fetch the redis details and store it to one output file and scheduled to crontab
  3. In filebeat I have given the path to fetch the output file.

So the idea is fetch the output file and send it to logstahs for parsing the logs. Now I have written the grok pattern and not able to parse properly.

Here is the format of the logs and filebat multiline pattern conf and logstash conf.

Filebeat:
multiline.pattern: '[1]'
multiline.negate: false
multiline.match: after

Lostash:

input {
beats {
port => "8"
}
}

filter {
if "redis_server_sit_1" in [tags] {
grok {
match => {
"message" => "# Server\sredis_version:%{DATA:redis_version}\sredis_git_sha1:%{NUMBER:redis_git_sha1}\sredis_git_dirty:%{NUMBER:redis_git_dirty}\sredis_build_id:%{WORD:redis_build_id}\sredis_mode:%{WORD:redis_mode}\sos:%{DATA:os}\sarch_bits:%{NUMBER:arch_bits}\smonotonic_clock:%{DATA:monotonic_clock}\smultiplexing_api:%{WORD:multiplexing_api}\satomicvar_api:%{DATA:atomicvar_api}\sgcc_version:%{DATA:gcc_version}\sprocess_id:%{NUMBER:process_id}\sprocess_supervised:%{WORD:process_supervised}\srun_id:%{DATA:run_id}\stcp_port:%{NUMBER:tcp_port}\sserver_time_usec:%{NUMBER:server_time_usec}\suptime_in_seconds:%{NUMBER:uptime_in_seconds}\suptime_in_days:%{NUMBER:uptime_in_days}\shz:%{NUMBER:hz}\sconfigured_hz:%{NUMBER:configured_hz}\slru_clock:%{NUMBER:lru_clock}\sexecutable:%{DATA:executable}\sconfig_file:%{DATA:config_file}\sio_threads_active:%{NUMBER:io_threads_active}\s*# Clients\sconnected_clients:%{NUMBER:connected_clients}\scluster_connections:%{NUMBER:cluster_connections}\smaxclients:%{NUMBER:maxclients}\sclient_recent_max_input_buffer:%{NUMBER:client_recent_max_input_buffer}\sclient_recent_max_output_buffer:%{NUMBER:client_recent_max_output_buffer}\sblocked_clients:%{NUMBER:blocked_clients}\stracking_clients:%{NUMBER:tracking_clients}\sclients_in_timeout_table:%{NUMBER:clients_in_timeout_table}\s*# Memory\sused_memory:%{NUMBER:used_memory}\sused_memory_human:%{DATA:used_memory_human}\sused_memory_rss:%{NUMBER:used_memory_rss}\sused_memory_rss_human:%{DATA:used_memory_rss_human}\sused_memory_peak:%{DATA:used_memory_peak}\sused_memory_peak_human:%{DATA:used_memory_peak_human}\sused_memory_peak_perc:%{DATA:used_memory_peak_perc}\sused_memory_overhead:%{NUMBER:used_memory_overhead}\sused_memory_startup:%{NUMBER:used_memory_startup}\sused_memory_dataset:%{NUMBER:used_memory_dataset}\sused_memory_dataset_perc:%{DATA:used_memory_dataset_perc}\sallocator_allocated:%{NUMBER:allocator_allocated}\sallocator_active:%{NUMBER:allocator_active}\sallocator_resident:%{NUMBER:allocator_resident}\stotal_system_memory:%{NUMBER:total_system_memory}\stotal_system_memory_human:%{DATA:total_system_memory_human}\sused_memory_lua:%{NUMBER:used_memory_lua}\sused_memory_lua_human:%{DATA:used_memory_lua_human}\sused_memory_scripts:%{NUMBER:used_memory_scripts}\sused_memory_scripts_human:%{WORD:used_memory_scripts_human}\snumber_of_cached_scripts:%{NUMBER:number_of_cached_scripts}\smaxmemory:%{NUMBER:maxmemory}\smaxmemory_human:%{DATA:maxmemory_human}\smaxmemory_policy:%{WORD:maxmemory_policy}\sallocator_frag_ratio:%{NUMBER:allocator_frag_ratio}\sallocator_frag_bytes:%{NUMBER:allocator_frag_bytes}\sallocator_rss_ratio:%{NUMBER:allocator_rss_ratio}\sallocator_rss_bytes:%{NUMBER:allocator_rss_bytes}\srss_overhead_ratio:%{NUMBER:rss_overhead_ratio}\srss_overhead_bytes:%{NUMBER:rss_overhead_bytes}\smem_fragmentation_ratio:%{NUMBER:mem_fragmentation_ratio}\smem_fragmentation_bytes:%{NUMBER:mem_fragmentation_bytes}\smem_not_counted_for_evict:%{NUMBER:mem_not_counted_for_evict}\smem_replication_backlog:%{NUMBER:mem_replication_backlog}\smem_clients_slaves:%{NUMBER:mem_clients_slaves}\smem_clients_normal:%{NUMBER:mem_clients_normal}\smem_aof_buffer:%{NUMBER:mem_aof_buffer}\smem_allocator:%{DATA:mem_allocator}\sactive_defrag_running:%{NUMBER:active_defrag_running}\slazyfree_pending_objects:%{NUMBER:lazyfree_pending_objects}\slazyfreed_objects:%{NUMBER:lazyfreed_objects}\s# Persistence\sloading:%{NUMBER:loading}\scurrent_cow_size:%{NUMBER:current_cow_size}\scurrent_cow_size_age:%{NUMBER:current_cow_size_age}\scurrent_fork_perc:%{NUMBER:current_fork_perc}\scurrent_save_keys_processed:%{NUMBER:current_save_keys_processed}\scurrent_save_keys_total:%{NUMBER:current_save_keys_total}\srdb_changes_since_last_save:%{NUMBER:rdb_changes_since_last_save}\srdb_bgsave_in_progress:%{NUMBER:rdb_bgsave_in_progress}\srdb_last_save_time:%{NUMBER:rdb_last_save_time}\srdb_last_bgsave_status:%{WORD:rdb_last_bgsave_status}\srdb_last_bgsave_time_sec:%{NUMBER:rdb_last_bgsave_time_sec}\srdb_current_bgsave_time_sec:%{NUMBER:rdb_current_bgsave_time_sec}\srdb_last_cow_size:%{NUMBER:rdb_last_cow_size}\saof_enabled:%{NUMBER:aof_enabled}\saof_rewrite_in_progress:%{NUMBER:aof_rewrite_in_progress}\saof_rewrite_scheduled:%{NUMBER:aof_rewrite_scheduled}\saof_last_rewrite_time_sec:%{NUMBER:aof_last_rewrite_time_sec}\saof_current_rewrite_time_sec:%{NUMBER:aof_current_rewrite_time_sec}\saof_last_bgrewrite_status:%{WORD:aof_last_bgrewrite_status}\saof_last_write_status:%{WORD:aof_last_write_status}\saof_last_cow_size:%{NUMBER:aof_last_cow_size}\smodule_fork_in_progress:%{NUMBER:module_fork_in_progress}\smodule_fork_last_cow_size:%{NUMBER:module_fork_last_cow_size}\s# Stats\stotal_connections_received:%{NUMBER:total_connections_received}\stotal_commands_processed:%{NUMBER:total_commands_processed}\sinstantaneous_ops_per_sec:%{NUMBER:instantaneous_ops_per_sec}\stotal_net_input_bytes:%{NUMBER:total_net_input_bytes}\stotal_net_output_bytes:%{NUMBER:total_net_output_bytes}\sinstantaneous_input_kbps:%{NUMBER:instantaneous_input_kbps}\sinstantaneous_output_kbps:%{NUMBER:instantaneous_output_kbps}\srejected_connections:%{NUMBER:rejected_connections}\ssync_full:%{NUMBER:sync_full}\ssync_partial_ok:%{NUMBER:sync_partial_ok}\ssync_partial_err:%{NUMBER:sync_partial_err}\sexpired_keys:%{NUMBER:expired_keys}\sexpired_stale_perc:%{NUMBER:expired_stale_perc}\sexpired_time_cap_reached_count:%{NUMBER:expired_time_cap_reached_count}\sexpire_cycle_cpu_milliseconds:%{NUMBER:expire_cycle_cpu_milliseconds}\sevicted_keys:%{NUMBER:evicted_keys}\skeyspace_hits:%{NUMBER:keyspace_hits}\skeyspace_misses:%{NUMBER:keyspace_misses}\spubsub_channels:%{NUMBER:pubsub_channels}\spubsub_patterns:%{NUMBER:pubsub_patterns}\slatest_fork_usec:%{NUMBER:latest_fork_usec}\stotal_forks:%{NUMBER:total_forks}\smigrate_cached_sockets:%{NUMBER:migrate_cached_sockets}\sslave_expires_tracked_keys:%{NUMBER:slave_expires_tracked_keys}\sactive_defrag_hits:%{NUMBER:active_defrag_hits}\sactive_defrag_misses:%{NUMBER:active_defrag_misses}\sactive_defrag_key_hits:%{NUMBER:active_defrag_key_hits}\sactive_defrag_key_misses:%{NUMBER:active_defrag_key_misses}\stracking_total_keys:%{NUMBER:tracking_total_keys}\stracking_total_items:%{NUMBER:tracking_total_items}\stracking_total_prefixes:%{NUMBER:tracking_total_prefixes}\sunexpected_error_replies:%{NUMBER:unexpected_error_replies}\stotal_error_replies:%{NUMBER:total_error_replies}\sdump_payload_sanitizations:%{NUMBER:dump_payload_sanitizations}\stotal_reads_processed:%{NUMBER:total_reads_processed}\stotal_writes_processed:%{NUMBER:total_writes_processed}\sio_threaded_reads_processed:%{NUMBER:io_threaded_reads_processed}\sio_threaded_writes_processed:%{NUMBER:io_threaded_writes_processed}\s*# Replication\srole:%{WORD:role}\smaster_host:%{IP:master_host}\smaster_port:%{NUMBER:master_port}\smaster_link_status:%{WORD:master_link_status}\smaster_last_io_seconds_ago:%{NUMBER:master_last_io_seconds_ago}\smaster_sync_in_progress:%{NUMBER:master_sync_in_progress}\sslave_read_repl_offset:%{NUMBER:slave_read_repl_offset}\sslave_repl_offset:%{NUMBER:slave_repl_offset}\sslave_priority:%{NUMBER:slave_priority}\sslave_read_only:%{NUMBER:slave_read_only}\sreplica_announced:%{NUMBER:replica_announced}\sconnected_slaves:%{NUMBER:connected_slaves}\smaster_failover_state:%{DATA:master_failover_state}\smaster_replid:%{WORD:master_replid}\smaster_replid2:%{WORD:master_replid2}\smaster_repl_offset:%{NUMBER:master_repl_offset}\ssecond_repl_offset:%{NUMBER:second_repl_offset}\srepl_backlog_active:%{NUMBER:repl_backlog_active}\srepl_backlog_size:%{NUMBER:repl_backlog_size}\srepl_backlog_first_byte_offset:%{NUMBER:repl_backlog_first_byte_offset}\srepl_backlog_histlen:%{NUMBER:repl_backlog_histlen}\s# CPU\sused_cpu_sys:%{NUMBER:used_cpu_sys}\sused_cpu_user:%{NUMBER:used_cpu_user}\sused_cpu_sys_children:%{NUMBER:used_cpu_sys_children}\sused_cpu_user_children:%{NUMBER:used_cpu_user_children}\sused_cpu_sys_main_thread:%{NUMBER:used_cpu_sys_main_thread}\sused_cpu_user_main_thread:%{NUMBER:used_cpu_user_main_thread}\s*# Modules\s*# Errorstats\serrorstat_ERR:count=%{NUMBER:errorstat_ERR_count}\s# Cluster\scluster_enabled:%{NUMBER:cluster_enabled}\s# Keyspace"
}
}
mutate { add_tag => "redis_logs" }
}
}

output {
if "redis_logs" in [tags] {
elasticsearch
{
hosts => ["host1:9200","host2:9200","host3:9200"]
ssl => true
user => "user"
password => "pass"
truststore => "/etc/ela12.ts"
truststore_password => "pass"
ssl_certificate_verification => true
ilm_rollover_alias => "redis-logs"
ilm_pattern => "{now/d}-000001"
ilm_policy => "redis-logs"

  }

}
}

the above one is configuration. but got the grok parsing error but in grok debugger it is coming perfectly fine.

the log pattern is:

Server

redis_version:6.2.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:5d88ce217879027a
redis_mode:standalone
os:Linux 4.18.0-477.15.1.el8_8.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.5.0
process_id:3297159
process_supervised:systemd
run_id:47c587c5f86a9d37b780c4de6bbcf4656a8d4cab
tcp_port:6379
server_time_usec:1703757767880238
uptime_in_seconds:1466763
uptime_in_days:16
hz:10
configured_hz:10
lru_clock:9258951
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0

Clients

connected_clients:16
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

Memory

used_memory:2325192
used_memory_human:2.22M
used_memory_rss:17420288
used_memory_rss_human:16.61M
used_memory_peak:2734208
used_memory_peak_human:2.61M
used_memory_peak_perc:85.04%
used_memory_overhead:2169768
used_memory_startup:813120
used_memory_dataset:155424
used_memory_dataset_perc:10.28%
allocator_allocated:2389120
allocator_active:2850816
allocator_resident:5304320
total_system_memory:6192857088
total_system_memory_human:5.77G
used_memory_lua:35840
used_memory_lua_human:35.00K
used_memory_scripts:528
used_memory_scripts_human:528B
number_of_cached_scripts:2
maxmemory:1073741824
maxmemory_human:1.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.19
allocator_frag_bytes:461696
allocator_rss_ratio:1.86
allocator_rss_bytes:2453504
rss_overhead_ratio:3.28
rss_overhead_bytes:12115968
mem_fragmentation_ratio:7.57
mem_fragmentation_bytes:15117888
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:307544
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

Persistence

loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:38
rdb_bgsave_in_progress:0
rdb_last_save_time:1703756924
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:536576
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:122
total_commands_processed:9309604
instantaneous_ops_per_sec:6
total_net_input_bytes:704769007
total_net_output_bytes:3921457572
instantaneous_input_kbps:0.55
instantaneous_output_kbps:4.32
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:1
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:683
total_forks:553
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:199
dump_payload_sanitizations:0
total_reads_processed:9080383
total_writes_processed:21023215
io_threaded_reads_processed:0
io_threaded_writes_processed:0

Replication

role:slave
master_host:IP
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:2676731963
slave_repl_offset:2676731963
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:75c2674d500152ef1e92f72065940c8c9292e2f5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2676731963
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2675683388
repl_backlog_histlen:1048576

CPU

used_cpu_sys:1057.229563
used_cpu_user:1217.442233
used_cpu_sys_children:0.921917
used_cpu_user_children:0.246200
used_cpu_sys_main_thread:1049.807073
used_cpu_user_main_thread:1211.922890

Modules

Errorstats

errorstat_ERR:count=199

Cluster

cluster_enabled:0

Keyspace

Now what is happening by doing so...
In grok debugger the above one is parsing correctly.

But the problem is happening every line is treated as a new event.
Filebat does not able to send the whole logs at a time to logstash for parsing.

It is sending like

Server

as a different event

redis_version:6.2.7
as a different event

redis_git_sha1:00000000
as a different event

redis_git_dirty:0
as a different event

redis_build_id:5d88ce217879027a
as a different event

redis_mode:standalone
as a different event

And in kibana I can see every line in a every document.

like # Server is a one document
like redis_version:6.2.7 is a one document
like redis_git_sha1:00000000 is a one document

This is how 174 documents are generating instead having one document


  1. # ↩︎

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.