# Shipping gitlab job logs

**URL:** <https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 25, 2019, 8:59am UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005 "2019-12-25T08:59:46Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![meir](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@meir](https://discuss.elastic.co/u/meir)\
**Post date:** [December 25, 2019, 8:59am UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005/1 "2019-12-25T08:59:46Z")

</div>

HI,

I have started using filebeat for log shipping for my gitlab logs, and I have a directory structure that is changed every few minutes, each directory contains a log file which contains data that I want to ship to my elasticsearch and create dashboard from it.

I've configured my "filebeat" yaml file to scan such type of directory

```auto
- type: log
  enabled: true
  paths:
    - /artifacts/*/*/*/*/*/*/*.log
  fields:
    log: jobs-log

```

it starts scanning the system and when it's finished it does not continue scanning new files creation.

any idea how can I ship all the gitlab logs to elk?

Thankms

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [December 26, 2019, 2:03pm UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005/2 "2019-12-26T14:03:01Z")

</div>

Hi @meir, welcome to the Elastic community forums!

A few questions:

1. What version of Filebeat are you using?

2. Have you tried defining your path as a `recursive_glob`, so something like:

Thanks,

Shaunak

---

<div class="post-metadata">

**Author:** ![meir](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@meir](https://discuss.elastic.co/u/meir)\
**Post date:** [December 30, 2019, 8:26am UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005/3 "2019-12-30T08:26:25Z")

</div>

Hi @shaunak and thanks for replying.

- I'm using the latest version of filebeat

- I have already useing this configurtation on the filebeat.yaml file.

I have managed to collect those logs, now I facing new issues that filebeat is not fast enough to collect logs that are finished fast.

e.g:  
I execute the pipeline, XXX.log files were created and finished in 3-5 sec then the files are moved to "artifact directory" which causes filebeat to read X lines from the log file and not all the log content,

any idea how can I grub all the log output before it's moved to "artifact directory"?

Thanks,  
Meir.

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [December 30, 2019, 8:07pm UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005/4 "2019-12-30T20:07:30Z")

</div>

Forgive me if I'm misunderstanding your setup but why not just have Filebeat collect the logs from the artifact directory? It sounds like all logs eventually get moved there and would stay there long enough for Filebeat to collect them?

Shaunak

---

<div class="post-metadata">

**Author:** ![meir](https://avatars.discourse-cdn.com/v4/letter/m/2acd7d/32.png) [@meir](https://discuss.elastic.co/u/meir)\
**Post date:** [December 31, 2019, 7:54am UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005/5 "2019-12-31T07:54:23Z")

</div>

I did it at the beginning, and those issues that I have encountered

- The directory structure is huge and it will take some time to scan and push the structure and logs data to the elk.
- Open File limitation, I had to add "LimitNOFILE=100000" to filebeat.service to stop yelling about it
- Most of the log files in the "artifact directory" are old and no more relevant for us now.
- Filebeat will enter the old logs with the current timestamp which is wrong.

those are the reasons why I don't scan the "artifact directory"

Thanks,  
Meir

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 28, 2020, 7:54am UTC](https://discuss.elastic.co/t/shipping-gitlab-job-logs/213005/6 "2020-01-28T07:54:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.