Filebeat IBMMQ module logs

Hi Team,

We are trying to integrate IBMMQ logs to elasticsearch using filebeat and its default ibmmq module. We are able to see the logs getting send to the elasticsearch from the filebeat debug logs. But we are unable to see this in elasticsearch discovery. When we checked the index we can see the size of index and document count is increasing, but nothing is showed in kibana discovery. Can you please help me how to troubleshoot this. are there any logs which we can see in elasticsearch search for this pipeline

Pipeline name is

[filebeat-7.9.0-ibmmq-errorlog-pipeline]

Thanks,
Ajesh

Hi Team,

Can we enable any logs specifically for this ibmmq module why its not showing in the discovery . Please see the attached index and discovery screenshot.

What's the rest of the timepicker, is it until Now? What if you change that to something in the future?
What is the output from GET INDEXNAME/_search, replacing INDEXNAME, in Dev Tools?

Hello Mark,

Thanks for looking into this. This was working fine in our old version of elasticsearch 7.6.2. So i tried installed 7.6.2 filebeat and point it to the new elasticsearch cluster , but still no logs. Please find the answers below

If i change the time future also nothing is shown.

What is the output from GET INDEXNAME/_search , replacing INDEXNAME , in Dev Tools?

Blockquote
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "ibmmq-dc-2020.12.03-000001",
"_type" : "_doc",
"_id" : "GJYpKnYBT0aBwEH4afce",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "ash3-hostb",
"id" : "b58ebff3-3876-45ae-9230-d23e3017eee3",
"type" : "filebeat",
"ephemeral_id" : "e8b76c93-48ef-4f32-9d17-1c8ee9da5201",
"version" : "7.6.2"
},
"process" : {
"pid" : "10881.4",
"title" : "amqzmur0"
},
"log" : {
"file" : {
"path" : "/mq/iflyres/RESBV_MQP1/data/RESBV_MQP1/errors/AMQERR02.LOG"
},
"offset" : 0,
"flags" : [
"multiline"
]
},
"fileset" : {
"name" : "errorlog"
},
"message" : "WebSphere MQ V8.0.0.4 (p800-004-151017).",
"ibmmq" : {
"errorlog" : {
"qmgr" : "RESBV_MQP1",
"code" : "AMQ6287",
"commentinsert" : [
"",
"",
""
],
"installation" : "Installation1",
"action" : "Host Info :- Linux 4.1.12-124.42.3.el7uek.x86_64 (MQ Linux (x86-64 platform) 64-bit) Installation :- /projects/mqm/applns/mq_v8 (Installation1) Version :- 8.0.0.4 (p800-004-151017) ACTION: None.",
"arithinsert" : [
"",
""
],
"explanation" : "WebSphere MQ system"
}
},
"input" : {
"type" : "log"
},
"ecs" : {
"version" : "1.4.0"
},
"service" : {
"type" : "ibmmq",
"version" : "8.0.0.4"
},
"host" : {
"hostname" : "ash3-hostb",
"os" : {
"kernel" : "4.1.12-124.42.3.el7uek.x86_64",
"name" : "Oracle Linux Server",
"family" : "",
"version" : "7.8",
"platform" : "ol"
},
"containerized" : false,
"name" : "ash3-hostb",
"id" : "7712548109ff4992a8fd44b79c66d65f",
"architecture" : "x86_64"
},
"event" : {
"created" : "2020-12-03T19:52:59.606Z",
"module" : "ibmmq",
"dataset" : "ibmmq.errorlog"
},
"user" : {
"name" : "mqm"
}
}
},
{
"_index" : "ibmmq-dc-2020.12.03-000001",
"_type" : "_doc",
"_id" : "GZYpKnYBT0aBwEH4afce",
"_score" : 1.0,
"_source" : {
"agent" : {
"hostname" : "ash3-hostb",
"id" : "b58ebff3-3876-45ae-9230-d23e3017eee3",
"type" : "filebeat",
"ephemeral_id" : "e8b76c93-48ef-4f32-9d17-1c8ee9da5201",
"version" : "7.6.2"
},
"process" : {
"pid" : "10957.8413",
"title" : "amqrmppa"
},
"log" : {
"file" : {
"path" : "/mq/iflyres/RESBV_MQP1/data/RESBV_MQP1/errors/AMQERR02.LOG"
},
"offset" : 528,
"flags" : [
"multiline"
]
},
"fileset" : {
"name" : "errorlog"
},
"message" : "Queue Manager User ID initialization failed for 'mqm'.",
"ibmmq" : {
"errorlog" : {
"qmgr" : "RESBV_MQP1",
"code" : "AMQ9557",
"commentinsert" : [
"",
"",
""
],
"installation" : "Installation1",
"action" : "Correct the error and try again.",
"arithinsert" : [
"",
""
],
"explanation" : "The call to initialize the User ID 'mqm' failed with CompCode 2 and Reason 2035."
}
},

Thanks,
Ajesh

That look current.

What is the index pattern set as in Index Management?

Hi Mark,

The index pattern is ibmmq-. I have also tried sending to default index pattern which is filebeat- and its is the same.

Meanwhile i tested this on our POC cluster where there is no basic xpack security enabled and its started working.

Then again for confirmation i built another test cluster with xpack and basic security enabled and there it is not working.

Then again i tried disabling the xpack security and still it didn't work.

Its is also not an issues with user permission since i used superuser elastic in filebeat config after enabling xpack basic security.

From debug logs i can see events are getting published successfully and index doc count is also increasing.

so not sure if its related to xpack and not able to find what is wrong is it a bug.

Thanks,
Ajesh

Hi Team,

Any suggestion.

Regards,
Ajesh

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.