Sending logs from filebeat to redis and then logstash


(Reshma Chowdary Morampudi) #1

I am trying to send my logs from filebeat into redis and then from redis into logstash and elasticsearch.
My log file is in json format. What configuration is to be made in my filebeat.yml so that the output is redis. I am getting the error as
( Failed to RPUSH to redis list with ERR wrong number of arguments for 'rpush' command
2018-05-22T11:15:43.063+0530 ERROR pipeline/output.go:92
Failed to publish events: ERR wrong number of arguments for 'rpush' command).
Also what configuartion is to be made in my logstash.yml when my input is redis.


(Harsh Bajaj) #2

Hi,

Could you please brief your use case that why are you implementing this mean what is the requirement and paste your pipeline here so that we can check the issue.

Regards,


(Reshma Chowdary Morampudi) #3

i want to make visualizations on kibana of my logs. the pipeline is as follows :
Filebeat - Redis - Logstash - Elasticsearch - Kibana.

The reason i have included redis in my pipeline is beacause it acts as a buffer and hence in case of data loss we can approach redis for that. But i am unable to send data from filebeat to redis itself.

Filebeat - Logstash - Elasticsearch - Kibana this pipeline is working fine. But when I am using redis, I am unable to load data from filebeat to redis.

I have attached the log file which m trying to send in the pipeline. Please provide the solution.

2018-05-18 09:29:24,387 [main] INFO org.mule.module.launcher.application.DefaultMuleApplication -
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

  • Initializing app 'RealtimeFlow' +
    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    2018-05-18 09:29:24,543 [main] INFO org.mule.lifecycle.AbstractLifecycleManager - Initialising RegistryBroker
    2018-05-18 09:29:24,636 [main] INFO org.mule.module.extension.internal.manager.DefaultExtensionManager - Starting discovery of extensions
    2018-05-18 09:29:24,745 [main] INFO org.mule.module.extension.internal.manager.DefaultExtensionManager - Discovered 1 extensions
    2018-05-18 09:29:24,745 [main] INFO org.mule.module.extension.internal.manager.DefaultExtensionManager - Registering extension validation (version 3.8)
    2018-05-18 09:29:24,809 [main] INFO org.mule.config.spring.MuleArtifactContext - Refreshing org.mule.config.spring.MuleArtifactContext@9573b3b: startup date [Fri May 18 09:29:24 IST 2018]; root of context hierarchy
    2018-05-18 09:29:26,353 [main] INFO org.mule.config.spring.processors.NoDevkitInjectorProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
    2018-05-18 09:29:31,942 [main] INFO org.mule.module.management.agent.WrapperManagerAgent - This JVM hasn't been launched by the wrapper, the agent will not run.
    2018-05-18 09:29:31,973 [main] INFO org.mule.DefaultMuleContext -

  • Application: RealtimeFlow *
  • OS encoding: , Mule encoding: UTF-8 *
  •                                                                *
    
  • Agents Running: *
  • JMX Agent *
  • DevKit Extension Information *
  • Batch module default engine *
  • Wrapper Manager *

2018-05-18 09:29:41,140 [[RealtimeFlow].HTTP_Listener_Configuration.worker.01] INFO com.mulesoft.weave.mule.utils.MuleWeaveFactory$ - MimeType was not resolved '/' delegating to Java.
2018-05-18 09:29:41,141 [[RealtimeFlow].HTTP_Listener_Configuration.worker.01] INFO org.mule.api.processor.LoggerMessageProcessor -
{"versionId":"1.0","environment":{"organizationId":"ejgje","businessUnitId":"dfsf","projectId":"dvbfsd","environmentId":"DEV","serverId":"10.45.43.223"},"interface":{"interfaceId":"REALTIME_SAP_interface","sourceId":"SFDC","targetId":"SAP"},"transactionDetail":{"correlationId":"e425d140-5a4f-11e8-81ec-320920524153","businessId":null,"executionTime":"2018-05-18 09:29:41:140","executionPoint":"Execution staring point of REALTIME_SAP_interface ","executionMessage":"RealTime Main Flow has been Started and setting the flow vars for logging purpose","executionStatus":"INPROCESS","payload":{}},"error":{"errorType":null,"errorCode":null,"errorText":null}}
2018-05-18 09:29:41,157 [[RealtimeFlow].HTTP_Listener_Configuration.worker.01] INFO com.mulesoft.weave.mule.utils.MuleWeaveFactory$ - MimeType was not resolved '/' delegating to Java.


(Harsh Bajaj) #4

Hi,

Could you please paste your filebeat.yml and redis config file here for further invesigation.

Thanks,


(Reshma Chowdary Morampudi) #5

Hi,

Below is the configuration of filebeat.yml

filebeat.prospectors:

  • type: log
    enabled: true
    paths:
    • C:\Users\adfgdhchg\Documents\My Received Files\json.log

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

filebeat.modules:

  • module: redis
    log:
    enabled: true
    slowlog:
    enabled: true

#------------------------------- Redis output ----------------------------------
output.redis:
hosts: ["localhost:6379"]
key: "logs"
db: 0
timeout: 5

And in redis I didnt make any configuration.

In logstash I have given input as

input {
redis {
hosts => "localhost:6379"
data_type => "list"
key => "logs"
}
}

and output as

output{
elasticsearch {
hosts => "localhost:9200"
index => "finalmule_logs"
user=> elastic
password=>elastic
}
stdout { }
}

Thank you


(Harsh Bajaj) #6

Hi,

Hope you have checked the version compatibility if not please check with below link.

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-redis.html#_compatibility_14

And please paste your filebeat logs here while starting the service.

Regards,


(Reshma Chowdary Morampudi) #7

Hi,
My redis version is 2.4.5 and F E L K version is 6.2.4 I followed the steps on the provided link but still i am getting the same error( Failed to rpush to redis list). Please provide me the solution.


(Harsh Bajaj) #8

Hi,

Please let me know below details.

Are you able to connect the redis server from filebeat (please check with telnet) ?

could you please check with increase the below timeout parameter.

if you still getting the error please paste your debug logs here for more investigation.

Regards,


(Reshma Chowdary Morampudi) #9

Hi,
I increased the timeout paramater value also still i am getting the same error.

the error is :
Failed to rpush to redis list.
Failed to publish events.
Please help me out with this.
Thanks.


(Harsh Bajaj) #10

Hi,

Have you checked the below details. Please check if not.

Regards,


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.