CustomLogs integration turns unhealthy with simple wildcard change

[CiscoMT (Cisco)]

Hello y'all,

I am trying to break down the different paths I have for app logs in my server into different customLogs integrations.

Originally I was using a single integration of the CustomLog type with path :

C:\inetpub\Hamilton\*\Logs\*.log

This worked fine

Now I am using multiple CustomLogs integrations, i.e:

customlogs called AccountManagement with path:

C:\inetpub\Hamilton\AccountManagement\Logs

customlogs called BookingManagement with path:

C:\inetpub\Hamilton\BookingManagement\Logs

I also tried setting the paths as below, but it didn't work either

C:\inetpub\Hamilton\BookingManagement\Logs\*.log

But as soon as I make this change, both integrations show as Unhealthy, I read that it was possible to use multiple Integrations of this type.

What may I be missing here?

[stephenb (Stephen Brown)]

Hi @CiscoMT Welcome to the community!

Multiple integrations should work..

@CiscoMT

C:\inetpub\Hamilton\AccountManagement\Logs

@CiscoMT

C:\inetpub\Hamilton\BookingManagement\Logs

Are you missing the \*.log at the end of both the new integrations

[CiscoMT (Cisco)]

Hi Stephen, Thanks for the welcome!

I think I read in the docs that you cold just leave the folder like that to consume the whole folder. But I did add the *.logs and we are still getting no logs and the agent still shows as Unhealthy.

C:\inetpub\Hamilton\AccountManagement\Logs\*.log

Looking at the agent logs, I noticed the issue below but I am not sure what it means:

  "components": [
    {
      "id": "log-default",
      "type": "log",
      "status": "HEALTHY",
      "message": "Healthy: communicating with pid '8520'",
      "units": [
        {
          "id": "log-default-logfile-logs-691a5a32-64ce-4c02-9f5d-d479cd5bb856",
          "type": "input",
          "status": "FAILED",
          "message": "[failed to reloading inputs: 2 errors: Error creating runner from config: Can only start an input when all related states are finished: {Id: native::1441792-2369445-1620868268, Finished: false, Fileinfo: &{19102023.log 32 {2945512199 31064694} {3402349450 31064729} {3402349450 31064729} 0 618885 0 0 {0 0}  1620868268 1441792 2369445 false}, Source: C:\\inetpub\\Hamilton\\AccountManagement\\Logs\\19102023.log, Offset: 1380633, Timestamp: 2023-10-19 16:41:18.6887885 -0400 EDT m=+21657.986057801, TTL: -1ns, Type: log, Meta: map[], FileStateOS: 1441792-2369445-1620868268}; Error creating runner from config: Can only start an input when all related states are finished: {Id: native::1703936-2369502-1620868268, Finished: false, Fileinfo: &{19102023.log 32 {2996491781 31064694} {3402619461 31064729} {3402619461 31064729} 0 908088 0 0 {0 0}  1620868268 1703936 2369502 false}, Source: C:\\inetpub\\Hamilton\\BookingManagement\\Logs\\19102023.log, Offset: 1934646, Timestamp: 2023-10-19 16:41:11.9932887 -0400 EDT m=+21651.290527201, TTL: -1ns, Type: log, Meta: map[], FileStateOS: 1703936-2369502-1620868268}]"
        },

[CiscoMT (Cisco)]

It seems like we managed to fix the issue by updating the agent version from 8.7.0 to 8.10.4. Once this was done the agent came back as Healthy, but we were not getting logs.

We also noticed the custom integration was using "-" as part of the dataset name, once this part was fixed we started receiving logs.

Now we are trying to filter in the stream by the dataset name provided in the custom integration but this one is not showing.
Appreciate any suggestions you could make with that part

Thanks!

[stephenb (Stephen Brown)]

@CiscoMT

Now we are trying to filter in the stream by the dataset name provided in the custom integration but this one is not showing.

Not sure what that means I would suggest opening a new topic with a complete description.

I am curious ...

@CiscoMT

I think I read in the docs that you cold just leave the folder like that to consume the whole folder.

Where you saw that in the docs, that did not work for me, I always have to provide the file *.* or *.log or * etc I tried just putting the folder / directory and it did not work

[CiscoMT (Cisco)]

Will create a new threat, Thanks following up @stephenb, appreciate the help!

Regarding the *.log part, funny, I didn't read that in the docs actually. It was a comment you left in some other topic a while ago hehe. Maybe I missed something and this doesn't apply to the custom logs integration.

Link here

[stephenb (Stephen Brown)]

Hi @CiscoMT, Awesome and Funny! I will need to look at that closer..

That was filebeat, but I would expect agent to work the same ... Hmmmmm

[stephenb (Stephen Brown)]

@CiscoMT THANKS for pointing that out ... no you need the trailing * I fixed that post. That was my misunderstanding not sure why I put that.

So yes now filbeat and agent work the same ... you need the actual file selector

..../* 
..../*.log 

etc