# Path config for custom log integration?

**URL:** <https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158>\
**Category:** Beats\
**Tags:** elastic-agent\
**Created:** [January 25, 2021, 6:38pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158 "2021-01-25T18:38:40Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tim\_Estes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_estes/32/64928_2.png) [@Tim\_Estes](https://discuss.elastic.co/u/Tim_Estes)\
**Post date:** [January 25, 2021, 6:38pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158/1 "2021-01-25T18:38:40Z")

</div>

Hello,

I'm trying out the elastic agent feature in a test cluster but I haven't been able to figure out how to ship custom logs to it.

In the integration, I've specified the log path to be `path/to/my/logs`:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/3/231afb5b50f9021e251ae2cd670fbe6f539a2122.png)

I was able to install and start an agent (with the correct policy) on my mac. I then created some `path/to/my/log` folders relative to where I installed the agent and put a few ECS formatted log files into them:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d5dc5291c47d75b79532e123b647b70e85252cb.png)

However, after a few minutes I did not find a data stream that contained the log files (only metric stuff, which I turned off later).

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/1/c1ebaa78892e7eace35a17ffe24bb51a37ce6d19.png)

I believe my error was incorrectly specifying the `path` argument. How does the agent know what directory to scan once it's installed? And how do I specify the right path?

For reference, here's the policy I've created

```auto
id: 4d36f8c0-5d05-11eb-84d5-c3f10edecab1
revision: 3
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'XXXXX'
agent:
  monitoring:
    enabled: false
    logs: false
    metrics: false
inputs:
  - id: f222cd70-5f33-11eb-84d5-c3f10edecab1
    name: python-logs-integration
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: log
        version: 0.4.6
    data_stream:
      namespace: default
    streams:
      - id: logfile-log.log
        data_stream:
          dataset: tbd
        paths:
          - path/to/my/logs
fleet:
  kibana:
    protocol: https
    hosts:
      - XXXX

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 25, 2021, 7:49pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158/2 "2021-01-25T19:49:33Z")

</div>

pretty sure all paths to logs need to be absolute... not relative

---

<div class="post-metadata">

**Author:** ![Tim\_Estes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_estes/32/64928_2.png) [@Tim\_Estes](https://discuss.elastic.co/u/Tim_Estes)\
**Post date:** [January 25, 2021, 8:01pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158/3 "2021-01-25T20:01:15Z")

</div>

That would make sense. So an example absolute path for Mac would look like:  
`/Users/<username>/Projects/GitLab/hello-world/path/to/my/logs`

I'll try that and see if that works. Also, do the log file names need to be formatted? I've named my log files `log5.log` and `log6.json` but I don't know if there's a "proper" name that the agent is looking for.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 25, 2021, 8:09pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158/4 "2021-01-25T20:09:11Z")

</div>

Yes

Paths respect globs see [here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#input-paths)

A list of glob-based paths that will be crawled and fetched. All patterns supported by [Go Glob](https://golang.org/pkg/path/filepath/#Glob) are also supported here.

if you put a directory path with \* all files will be harvested...  
`/Users/<username>/Projects/GitLab/hello-world/path/to/my/logs/*`

if you do something like this only \*.log files will be harvested  
`/Users/<username>/Projects/GitLab/hello-world/path/to/my/logs/*.log`

---

<div class="post-metadata">

**Author:** ![Tim\_Estes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_estes/32/64928_2.png) [@Tim\_Estes](https://discuss.elastic.co/u/Tim_Estes)\
**Post date:** [January 25, 2021, 8:16pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158/5 "2021-01-25T20:16:53Z")

</div>

Thanks! Once I made those changes, I could see log files appearing in my data-stream (named tbd)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1c8b3987cc583b0d057514a2b18e66de90fd400c.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 22, 2021, 10:16pm UTC](https://discuss.elastic.co/t/path-config-for-custom-log-integration/262158/6 "2021-02-22T22:16:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.