# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=177 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 178 --- ## [Logstash CSV output generation](https://discuss.elastic.co/t/logstash-csv-output-generation/290285)
**Author:** [@elk\_chaser](https://discuss.elastic.co/u/elk_chaser)\ **Replies:** 1\ **Last updated:** [December 3, 2021, 5:03pm UTC](https://discuss.elastic.co/t/logstash-csv-output-generation/290285 "2021-12-03T17:03:03Z")
I have a logstash csv output report generation config, that I run as a separate logstash instance as and when it is required. I have many csv report to be generated and for each, I have one config file. Everytime I run t… --- ## [LogStash with distributor pattern, relay pipelines problem](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880)
**Author:** [@djehuty](https://discuss.elastic.co/u/djehuty)\ **Replies:** 7\ **Last updated:** [December 3, 2021, 4:12pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880 "2021-12-03T16:12:49Z")
Hi everyone, i recently changed my LogStash configuration from multiple pipelines input to 3 pipelines, relaying on another local pipelines with distributor. Now, i've tried to send and udp json to the udp pipeline, wi… --- ## [Logstash log4j2.properties deleting all the compressed files](https://discuss.elastic.co/t/logstash-log4j2-properties-deleting-all-the-compressed-files/290896)
**Author:** [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Replies:** 0\ **Last updated:** [December 3, 2021, 12:33pm UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-deleting-all-the-compressed-files/290896 "2021-12-03T12:33:08Z")
Hi Team, I am trying to delete the audit files from Elasticsearch logs directory "C:\\test-elk\\Elasticsearch-7.7.0\\data\\logs". My audit files are generating to the mentioned path but unfortunately I am not able to delete… --- ## [Refreshing the access token when communicating with an API](https://discuss.elastic.co/t/refreshing-the-access-token-when-communicating-with-an-api/25384)
**Author:** [@rmarji](https://discuss.elastic.co/u/rmarji)\ **Replies:** 10\ **Last updated:** [December 3, 2021, 9:05am UTC](https://discuss.elastic.co/t/refreshing-the-access-token-when-communicating-with-an-api/25384 "2021-12-03T09:05:26Z")
I'm trying to use Http\_poller input plugin, to get results from an API, it works fine while the access token is valid, however its only valid for one hour, I want to be able to refresh the access token with the refresh t… --- ## [Logstash 7.15 grok patterns directory](https://discuss.elastic.co/t/logstash-7-15-grok-patterns-directory/290826)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 1\ **Last updated:** [December 2, 2021, 8:48pm UTC](https://discuss.elastic.co/t/logstash-7-15-grok-patterns-directory/290826 "2021-12-02T20:48:19Z")
Hi, I was running logstash 7.3, and had configured some grok patterns in a file and placed that file in /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/ After upgrading to logs… --- ## [Systemctl start logstash does nothing](https://discuss.elastic.co/t/systemctl-start-logstash-does-nothing/290814)
**Author:** [@EvanGertis](https://discuss.elastic.co/u/EvanGertis)\ **Replies:** 6\ **Last updated:** [December 2, 2021, 7:36pm UTC](https://discuss.elastic.co/t/systemctl-start-logstash-does-nothing/290814 "2021-12-02T19:36:57Z")
I am working on pulling changes onto a few logstash boxes. The issue is when I run the command systemctl start logstash I get nothing back. Expected: logstash starts with a verbose output. Actual: Any help with t… --- ## [Given Apache logs in ES, how do I get Visitor Stay Length for each URL](https://discuss.elastic.co/t/given-apache-logs-in-es-how-do-i-get-visitor-stay-length-for-each-url/290762)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 3\ **Last updated:** [December 2, 2021, 3:57pm UTC](https://discuss.elastic.co/t/given-apache-logs-in-es-how-do-i-get-visitor-stay-length-for-each-url/290762 "2021-12-02T15:57:45Z")
Given Apache logs in ES, how do I get Visitor Stay Length for each URL Apache logs: 222.105.82.235 - - \[16/Nov/2021:10:04:37 +0530\] "GET /admin/ HTTP/1.1" 301 529 "http://facebook.com/admin/" "Mozilla/5.0 (Windows NT 1… --- ## [Using logstash for transferring data to Azure Eventhub](https://discuss.elastic.co/t/using-logstash-for-transferring-data-to-azure-eventhub/290769)
**Author:** [@aeoker](https://discuss.elastic.co/u/aeoker)\ **Replies:** 0\ **Last updated:** [December 2, 2021, 1:08pm UTC](https://discuss.elastic.co/t/using-logstash-for-transferring-data-to-azure-eventhub/290769 "2021-12-02T13:08:13Z")
Hello Everyone, I need to use logstash for sending data to Azure Event Hub.As much as I found, there is only an input plugin for reading data from Azure Hub. Is there a logstash output plugin for sending data to Az… --- ## [Forwarding logs from 4.7 to external logstash via filebeat](https://discuss.elastic.co/t/forwarding-logs-from-4-7-to-external-logstash-via-filebeat/290751)
**Author:** [@sabarivijay88](https://discuss.elastic.co/u/sabarivijay88)\ **Replies:** 0\ **Last updated:** [December 2, 2021, 10:36am UTC](https://discuss.elastic.co/t/forwarding-logs-from-4-7-to-external-logstash-via-filebeat/290751 "2021-12-02T10:36:17Z")
Hi Team, We have deployed filebeat v7.15 daemon sets on Openshift v4.7 We are able to connect external logstash by using default path: /var/log/containers/\*.log We need to setup only one particular Openshift project a… --- ## [Logstash beats: 69/71 error](https://discuss.elastic.co/t/logstash-beats-69-71-error/290708)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 3\ **Last updated:** [December 1, 2021, 8:55pm UTC](https://discuss.elastic.co/t/logstash-beats-69-71-error/290708 "2021-12-01T20:55:04Z")
Logstash configuration: input { beats { port =\> 9443 host =\> "0.0.0.0" ssl =\> false } } filter { ... } output { ... } stdout{} } It looks like logstash is having an issue with the beats tag as I… --- ## [IAM role issue in s3 output plugin](https://discuss.elastic.co/t/iam-role-issue-in-s3-output-plugin/290702)
**Author:** [@John\_snow](https://discuss.elastic.co/u/John_snow)\ **Replies:** 0\ **Last updated:** [December 1, 2021, 5:56pm UTC](https://discuss.elastic.co/t/iam-role-issue-in-s3-output-plugin/290702 "2021-12-01T17:56:51Z")
Trying to use iam role for cross account S3 Bucket but getting following error Pipeline error {:pipeline\_id=\>"main", :exception=\>#\ … --- ## [Make a loop to read values in logstash](https://discuss.elastic.co/t/make-a-loop-to-read-values-in-logstash/290692)
**Author:** [@elpazo](https://discuss.elastic.co/u/elpazo)\ **Replies:** 1\ **Last updated:** [December 1, 2021, 8:04pm UTC](https://discuss.elastic.co/t/make-a-loop-to-read-values-in-logstash/290692 "2021-12-01T20:04:28Z")
Hello, I have a result like this ' values \[ \[0\] { X = 1 Y = 2 Z = 3 V = 4 } \[1\] { X = 5 Y = 6 Z = 7 V= 8 } ' How can i read this result and put every value in variable (example Var\_X = Values\[0\]\[0\] = 1… --- ## [Schedule csv](https://discuss.elastic.co/t/schedule-csv/290575)
**Author:** [@lnunez](https://discuss.elastic.co/u/lnunez)\ **Replies:** 1\ **Last updated:** [December 1, 2021, 7:43pm UTC](https://discuss.elastic.co/t/schedule-csv/290575 "2021-12-01T19:43:39Z")
It is not possible to schedule the csv using file as input, I've tried also the approach mentioned here: Cron job for CSV filter, which is using exec. But as is mentioned in the documentation: (Cron job for CSV filter) … --- ## [Multiple output sequence](https://discuss.elastic.co/t/multiple-output-sequence/290640)
**Author:** [@zhanggengengen](https://discuss.elastic.co/u/zhanggengengen)\ **Replies:** 1\ **Last updated:** [December 1, 2021, 7:25pm UTC](https://discuss.elastic.co/t/multiple-output-sequence/290640 "2021-12-01T19:25:01Z")
I have two output,the first is http-output,and the second is elasticsearch-output. What i want to do is execute http-output before elasticsearch-output. How can i do that? The problem I have is this I need delete it be… --- ## [Unicode whitespace in configuration files](https://discuss.elastic.co/t/unicode-whitespace-in-configuration-files/290651)
**Author:** [@jmb](https://discuss.elastic.co/u/jmb)\ **Replies:** 2\ **Last updated:** [December 1, 2021, 7:21pm UTC](https://discuss.elastic.co/t/unicode-whitespace-in-configuration-files/290651 "2021-12-01T19:21:28Z")
Is Logstash expected to handle Unicode (UTF-8) in its configuration files? I found an issue today where a pipeline (actually, an Elasticsearch mapping template file) which looks perfectly good fails to load because some… --- ## [How to remove the fields](https://discuss.elastic.co/t/how-to-remove-the-fields/290663)
**Author:** [@rahul\_sirugudi](https://discuss.elastic.co/u/rahul_sirugudi)\ **Replies:** 3\ **Last updated:** [December 1, 2021, 7:19pm UTC](https://discuss.elastic.co/t/how-to-remove-the-fields/290663 "2021-12-01T19:19:28Z")
Hi, I am using filebeat to ship logs to logstash, and creating index. Now while visualising logs from kibana there are many unwanted fields i am seeing like agent.id, cloud.account.id. I want to remove all the fields. I… --- ## [Logstash is not taking in account whitespaces at the end of lines ? (grok filter)](https://discuss.elastic.co/t/logstash-is-not-taking-in-account-whitespaces-at-the-end-of-lines-grok-filter/290449)
**Author:** [@duanra22](https://discuss.elastic.co/u/duanra22)\ **Replies:** 2\ **Last updated:** [December 1, 2021, 4:42pm UTC](https://discuss.elastic.co/t/logstash-is-not-taking-in-account-whitespaces-at-the-end-of-lines-grok-filter/290449 "2021-12-01T16:42:41Z")
Hello Elastic community, Sorry if this question has already been asked but I didn't find something that was close to what I'm looking for. I'm new to logstash and I'm trying to use Grok filter. I have a string of conc… --- ## [How to parse xml log inside of the json format in Logstash](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006)
**Author:** [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\ **Replies:** 15\ **Last updated:** [December 1, 2021, 6:28am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006 "2021-12-01T06:28:09Z")
I will receive a log that contained JSON and XML format I configurated the Logstash to receive TCP in JSON and used the filter when shown dateparsefailure in tags. The JSON part is able to extract, but the xml part can… --- ## [Errors/Problems with logstash in docker-compose](https://discuss.elastic.co/t/errors-problems-with-logstash-in-docker-compose/290616)
**Author:** [@Leandro\_Valim](https://discuss.elastic.co/u/Leandro_Valim)\ **Replies:** 0\ **Last updated:** [December 1, 2021, 12:14am UTC](https://discuss.elastic.co/t/errors-problems-with-logstash-in-docker-compose/290616 "2021-12-01T00:14:22Z")
Hello, I have problems to up my docker-compose enviroment with logstash docker-compose.yml # cat docker-compose.yml version: '2' services: logstash: image: docker.elastic.co/logstash/logstash:7.10.2 containe… --- ## [Logstash + Redis](https://discuss.elastic.co/t/logstash-redis/289100)
**Author:** [@Claudio\_Ract\_Costa](https://discuss.elastic.co/u/Claudio_Ract_Costa)\ **Replies:** 8\ **Last updated:** [November 30, 2021, 8:44pm UTC](https://discuss.elastic.co/t/logstash-redis/289100 "2021-11-30T20:44:38Z")
Hi guys, I am trying to figure out how to use Logstash with Redis Does anyone know how to configure logstash to insert a specific field in message in Redis ? For example, I have a json message with some fields, like: … --- ## [Nessus XML Parsing Errors and No output](https://discuss.elastic.co/t/nessus-xml-parsing-errors-and-no-output/290599)
**Author:** [@James\_Doherty](https://discuss.elastic.co/u/James_Doherty)\ **Replies:** 0\ **Last updated:** [November 30, 2021, 8:19pm UTC](https://discuss.elastic.co/t/nessus-xml-parsing-errors-and-no-output/290599 "2021-11-30T20:19:30Z")
I am having a really difficult time getting my Nessus files to parse correctly if at all. This is my Logstash config: input { file { path =\> "/opt/nessus/\*" mode =\> "read" start\_position =\> "beginning" … --- ## [File ingest timestamp question](https://discuss.elastic.co/t/file-ingest-timestamp-question/290566)
**Author:** [@hagaluly](https://discuss.elastic.co/u/hagaluly)\ **Replies:** 2\ **Last updated:** [November 30, 2021, 3:25pm UTC](https://discuss.elastic.co/t/file-ingest-timestamp-question/290566 "2021-11-30T15:25:45Z")
by default when i ingest a single json file kibana present timestamp at the time of the ingesting process. is there a way to use a timestamp record within the json and present it in kibana as the ingesting time? --- ## [Unable to bind specific network interface to logstash](https://discuss.elastic.co/t/unable-to-bind-specific-network-interface-to-logstash/290556)
**Author:** [@hari97](https://discuss.elastic.co/u/hari97)\ **Replies:** 6\ **Last updated:** [November 30, 2021, 2:32pm UTC](https://discuss.elastic.co/t/unable-to-bind-specific-network-interface-to-logstash/290556 "2021-11-30T14:32:12Z")
I have two interfaces eth0 and eth2. Network traffic of systems reaching to eth2 interface. I have Logstash instance which was working good for network traffic of default network interface (eth0). My requirement is log… --- ## [How to send data via Logstash](https://discuss.elastic.co/t/how-to-send-data-via-logstash/289947)
**Author:** [@smam](https://discuss.elastic.co/u/smam)\ **Replies:** 10\ **Last updated:** [November 30, 2021, 9:30am UTC](https://discuss.elastic.co/t/how-to-send-data-via-logstash/289947 "2021-11-30T09:30:13Z")
Hello, yes. This should be very common but I cannot send data to Elasticsearch-or I cannot find it there. Either way I am haven trouble using the data. What I am doing: Filebeat sends a collection of Log-Files to Logst… --- ## [How add prefix to the fields using kv filter](https://discuss.elastic.co/t/how-add-prefix-to-the-fields-using-kv-filter/238150)
**Author:** [@Ramalakshmi](https://discuss.elastic.co/u/Ramalakshmi)\ **Replies:** 7\ **Last updated:** [November 30, 2021, 9:00am UTC](https://discuss.elastic.co/t/how-add-prefix-to-the-fields-using-kv-filter/238150 "2021-11-30T09:00:20Z")
Hi I am adding prefix to the fields using kv filter, It's working to the some of the fields in the json message. Example: Filed name is resource.network.imageid while adding prefix as aws.datasource using kv filter I … --- ## [Exec input](https://discuss.elastic.co/t/exec-input/290463)
**Author:** [@lnunez](https://discuss.elastic.co/u/lnunez)\ **Replies:** 1\ **Last updated:** [November 29, 2021, 4:54pm UTC](https://discuss.elastic.co/t/exec-input/290463 "2021-11-29T16:54:47Z")
Hi everyone, Just to keep the consistency between the input, I realized that is not possible to just execute manually a pipeline with an exec input without a schedule definition. This is allowed using the JDBC input or … --- ## [Keystore is not possible used in a pipeline after enabling centralized managment](https://discuss.elastic.co/t/keystore-is-not-possible-used-in-a-pipeline-after-enabling-centralized-managment/288641)
**Author:** [@lnunez](https://discuss.elastic.co/u/lnunez)\ **Replies:** 2\ **Last updated:** [November 29, 2021, 4:33pm UTC](https://discuss.elastic.co/t/keystore-is-not-possible-used-in-a-pipeline-after-enabling-centralized-managment/288641 "2021-11-29T16:33:00Z")
Related with: if is enabled the centralized management pipeline, it will deactivated the path settings to be used in cmd, but some pipeline, could have some key-value from the keystore, and as is deactivated the "--pat… --- ## [Logstash doc has same breaking changes across multiple versions](https://discuss.elastic.co/t/logstash-doc-has-same-breaking-changes-across-multiple-versions/290259)
**Author:** [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\ **Replies:** 2\ **Last updated:** [November 29, 2021, 2:52am UTC](https://discuss.elastic.co/t/logstash-doc-has-same-breaking-changes-across-multiple-versions/290259 "2021-11-29T02:52:35Z")
Hello. I was reviewing breaking changes on doc for my upgrade but realized all the changes are same from 7.13 to 7.15 . Should I interpret as "these changes should be aware between ANY version" or something else ? … --- ## [Looking for a way to ingest a file and remove it](https://discuss.elastic.co/t/looking-for-a-way-to-ingest-a-file-and-remove-it/290343)
**Author:** [@hagaluly](https://discuss.elastic.co/u/hagaluly)\ **Replies:** 3\ **Last updated:** [November 28, 2021, 10:38pm UTC](https://discuss.elastic.co/t/looking-for-a-way-to-ingest-a-file-and-remove-it/290343 "2021-11-28T22:38:40Z")
i created a logstash conf file while back and it ingest the filr and removed it. back at the time i did not wanted this behavior but now i do... i only forgot how to do it :slight\_smile: can someone direct me to the c… --- ## [Why my @timestamp shows wrong hour when overwritten?](https://discuss.elastic.co/t/why-my-timestamp-shows-wrong-hour-when-overwritten/290368)
**Author:** [@elk\_chaser](https://discuss.elastic.co/u/elk_chaser)\ **Replies:** 2\ **Last updated:** [November 28, 2021, 5:32pm UTC](https://discuss.elastic.co/t/why-my-timestamp-shows-wrong-hour-when-overwritten/290368 "2021-11-28T17:32:01Z")
echo "Nov 25 11:12:15 test message goes here" | /usr/share/logstash/bin/logstash -e 'input { stdin {} } filter { grok { keep\_empty\_captures =\> true match =\> { "message" =\> "^(?\%{SYSLOGTIMESTAMP:t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=176) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=178)