# Elastic-Agent - Collect Custom \[Linux\] text file logs **URL:** \ **Category:** Elastic Agent\ **Tags:** filebeat, integrations\ **Created:** [July 29, 2023, 10:19am UTC](https://discuss.elastic.co/t/elastic-agent-collect-custom-linux-text-file-logs/339593 "2023-07-29T10:19:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Bryan\_Hamilton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_hamilton/32/82111_2.png) [@Bryan\_Hamilton](https://discuss.elastic.co/u/Bryan_Hamilton)\ **Post date:** [July 29, 2023, 10:19am UTC](https://discuss.elastic.co/t/elastic-agent-collect-custom-linux-text-file-logs/339593/1 "2023-07-29T10:19:11Z")
Hi, I have created a github issue for this question ([Support for Custom [Linux] text file logs · Issue #7186 · elastic/integrations · GitHub](https://github.com/elastic/integrations/issues/7186)), but I am also adding it here for greater visibility. We have custom applications in our environments that store logs under `/data/*` directories. We currently use standalone filebeat to collect and ship those logs to elasticsearch. We have been trying migrate from standalone filebeat to elastic-agent and collect those logs through the agent. However, our attempts by using the existing integrations so far have been unsuccessful. Most of those logs are in json format but some of them do not follow any standard formats. We have created custom datastreams and ingest pipelines for those logs. At the moment, to feed logs into those datastreams from filebeat, we add the fields data\_stream.\* through processors in the filebeat config file. Trying to do the same with the existing elastic-agent integrations is a bit more challenging. Event though a custom windows event log intergration exists for windows, where you can specify your own dataset name when setting up the integration (see picture below), similar intergration doesn't seem to exist for linux or windows **text file** logs (at least none that we could find so far). ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/1/a1726dddac459c8b7be67e2eb0ebd76ddb3f9edc.png) We have tried using the system/syslog integration, add the path to the custom logs and use the `reroute` processor in the `logs-system.syslog@custom` to reroute the logs to the appropriate data\_stream but this doesn't work because the managed `logs-system.syslog-` pipeline has a grok processor with specific patterns and when those patterns fail against our logs, an error is produced and no other processors below it are processed which means the log never gets to our reroute processor. Any suggestions on how to handle this case with the elastic-agent? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [July 29, 2023, 3:45pm UTC](https://discuss.elastic.co/t/elastic-agent-collect-custom-linux-text-file-logs/339593/2 "2023-07-29T15:45:10Z")
Hi @Bryan_Hamilton I am not sure I am completely following but have you looked at just the plain ole Custom Logs integration... There is no OOTB procession / pipeline. You can add your own top-level pipeline and then do as you please I did thin to pull in various type of logs... identify them .. then process and route where I pleased. Perhaps I am missing something. ![Screenshot 2023-07-29 at 8.41.20 AM](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac954b6888b5847bfc59943934b1fd8cc61a2e90.png) Youi add your pipeline in the custom config ![Screenshot 2023-07-29 at 8.44.40 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/4/14490cc80fcabb9bdb070d2ae160bc1365e3cdda.png) ---
**Author:** ![Bryan\_Hamilton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_hamilton/32/82111_2.png) [@Bryan\_Hamilton](https://discuss.elastic.co/u/Bryan_Hamilton)\ **Post date:** [August 2, 2023, 9:39pm UTC](https://discuss.elastic.co/t/elastic-agent-collect-custom-linux-text-file-logs/339593/3 "2023-08-02T21:39:08Z")
Thank you for drawing my attention to this. Yes, I have tried this process and it allows us to do exactly what we wanted. Not sure how we missed this one. Many Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2023, 9:39pm UTC](https://discuss.elastic.co/t/elastic-agent-collect-custom-linux-text-file-logs/339593/4 "2023-08-30T21:39:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.