Path config for custom log integration?


I'm trying out the elastic agent feature in a test cluster but I haven't been able to figure out how to ship custom logs to it.

In the integration, I've specified the log path to be path/to/my/logs:

I was able to install and start an agent (with the correct policy) on my mac. I then created some path/to/my/log folders relative to where I installed the agent and put a few ECS formatted log files into them:

However, after a few minutes I did not find a data stream that contained the log files (only metric stuff, which I turned off later).

I believe my error was incorrectly specifying the path argument. How does the agent know what directory to scan once it's installed? And how do I specify the right path?

For reference, here's the policy I've created

id: 4d36f8c0-5d05-11eb-84d5-c3f10edecab1
revision: 3
    type: elasticsearch
      - 'XXXXX'
    enabled: false
    logs: false
    metrics: false
  - id: f222cd70-5f33-11eb-84d5-c3f10edecab1
    name: python-logs-integration
    revision: 1
    type: logfile
    use_output: default
        name: log
        version: 0.4.6
      namespace: default
      - id: logfile-log.log
          dataset: tbd
          - path/to/my/logs
    protocol: https
      - XXXX

pretty sure all paths to logs need to be absolute... not relative

That would make sense. So an example absolute path for Mac would look like:

I'll try that and see if that works. Also, do the log file names need to be formatted? I've named my log files log5.log and log6.json but I don't know if there's a "proper" name that the agent is looking for.


Paths respect globs see here

A list of glob-based paths that will be crawled and fetched. All patterns supported by Go Glob are also supported here.

if you put a directory path with * all files will be harvested...

if you do something like this only *.log files will be harvested

Thanks! Once I made those changes, I could see log files appearing in my data-stream (named tbd)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.