# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=462 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 463 --- ## [Ignore all files in s3 and read only current files from S3](https://discuss.elastic.co/t/ignore-all-files-in-s3-and-read-only-current-files-from-s3/202223)
**Author:** [@user2416](https://discuss.elastic.co/u/user2416)\ **Replies:** 2\ **Last updated:** [October 15, 2019, 5:45pm UTC](https://discuss.elastic.co/t/ignore-all-files-in-s3-and-read-only-current-files-from-s3/202223 "2019-10-15T17:45:54Z")
how to ignore old files and push only latest log files from S3 using logstash. We are using logstash to push cloudtaril logs from s3 to elasticsearch. Cloudtrail logs are in below format /AWSLogs/CloudTrail/xxxAccount N… --- ## [Logstash event @timestamp adjustment](https://discuss.elastic.co/t/logstash-event-timestamp-adjustment/203616)
**Author:** [@Stefan\_Sabolowitsch](https://discuss.elastic.co/u/Stefan_Sabolowitsch)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 2:50pm UTC](https://discuss.elastic.co/t/logstash-event-timestamp-adjustment/203616 "2019-10-15T14:50:22Z")
Hi there, i want to calculate "event.start" from "event.end" and "event.duration", so a subtraction of "event.end" and "event.duration" "event.end": "2019-10-15T07:34:14.000Z", "event.duration": 30, (sec.) Thereafter… --- ## [Filebeat / Winlogbeat -\> Logstash -\> Elastic '@timestamp' problem](https://discuss.elastic.co/t/filebeat-winlogbeat-logstash-elastic-timestamp-problem/203633)
**Author:** [@TaloN1x](https://discuss.elastic.co/u/TaloN1x)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 2:32pm UTC](https://discuss.elastic.co/t/filebeat-winlogbeat-logstash-elastic-timestamp-problem/203633 "2019-10-15T14:32:48Z")
I am capturing logs from hosts using winlogbeat and filebeat and sending them to the logstash, that is shipping them to the Elastic. The problem I am facing, is that filebeat and winlogbeat as well as logstash are usein… --- ## [Remove timestamp and host frpm logfile](https://discuss.elastic.co/t/remove-timestamp-and-host-frpm-logfile/203480)
**Author:** [@hdma](https://discuss.elastic.co/u/hdma)\ **Replies:** 3\ **Last updated:** [October 15, 2019, 2:25pm UTC](https://discuss.elastic.co/t/remove-timestamp-and-host-frpm-logfile/203480 "2019-10-15T14:25:42Z")
Hello I have a logstash that receive log from other logstash using tcp: input { tcp { port =\> 65009 mode =\> server id =\> "wm-id" tags =\> "wm" codec =\> line } } In output, I put this syslog into file: filter { … --- ## [One kafka topic data is pushing to all elasticsearh](https://discuss.elastic.co/t/one-kafka-topic-data-is-pushing-to-all-elasticsearh/203591)
**Author:** [@jaychaudhary](https://discuss.elastic.co/u/jaychaudhary)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 2:24pm UTC](https://discuss.elastic.co/t/one-kafka-topic-data-is-pushing-to-all-elasticsearh/203591 "2019-10-15T14:24:33Z")
Hello All, I am facing one problem with kafka logstash plugin, I have two kafka topic (ie. Topic-1, topic-2) and two indexes on Elasticsearch (ie. Index-1, index-2). I want to send topic-1 data to index-1 and topic-2 da… --- ## [Logstash使用if做条件判断问题](https://discuss.elastic.co/t/logstash-if/203544)
**Author:** [@shenshensinian](https://discuss.elastic.co/u/shenshensinian)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 2:16pm UTC](https://discuss.elastic.co/t/logstash-if/203544 "2019-10-15T14:16:00Z")
你好，我在filter阶段使用if语句做条件判断的时候出现if语句无效果，也没有报错。版本7.1.1 例句如下。 filter { if \[message\] in \["te1","bb","lkj","hetyafaef"\] grok{ match =\> {"message" =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"} } } --- ## [Logstash elasticsearch filter plugin 401 error](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-401-error/203328)
**Author:** [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Replies:** 2\ **Last updated:** [October 15, 2019, 12:58pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-plugin-401-error/203328 "2019-10-15T12:58:55Z")
Hello Team, We are trying to use elasticsearch filter plugin,But we are getting 401 unauthorised error. Since our elasticsearch is secured , In elasticsearch output section we have configured SSL, cacert, keystore, and… --- ## [Logstash in AWS container with S3 output not working unless S3 bucket is public?](https://discuss.elastic.co/t/logstash-in-aws-container-with-s3-output-not-working-unless-s3-bucket-is-public/203629)
**Author:** [@KalleDania](https://discuss.elastic.co/u/KalleDania)\ **Replies:** 0\ **Last updated:** [October 15, 2019, 11:16am UTC](https://discuss.elastic.co/t/logstash-in-aws-container-with-s3-output-not-working-unless-s3-bucket-is-public/203629 "2019-10-15T11:16:03Z")
So, im hosting ELK in AWS, but it seems my Logstash container wont forward logs to my S3 storage unit even despite me giving it the right permissions. Could this be something to do with Logstash? I have role with access… --- ## [Copy last line of file to another file if the size of the first file is increasing](https://discuss.elastic.co/t/copy-last-line-of-file-to-another-file-if-the-size-of-the-first-file-is-increasing/203601)
**Author:** [@Adrian\_Cincu](https://discuss.elastic.co/u/Adrian_Cincu)\ **Replies:** 0\ **Last updated:** [October 15, 2019, 9:05am UTC](https://discuss.elastic.co/t/copy-last-line-of-file-to-another-file-if-the-size-of-the-first-file-is-increasing/203601 "2019-10-15T09:05:11Z")
Hi, i wrote the output of some logs to a file called interface\_up. I would like to copy the last line of this file to another file compare.txt ( lines in this file will be overwrite each time). Can you help me please? o… --- ## [How to import a config file into Docker Container of Logstash?](https://discuss.elastic.co/t/how-to-import-a-config-file-into-docker-container-of-logstash/203099)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 4\ **Last updated:** [October 15, 2019, 7:46am UTC](https://discuss.elastic.co/t/how-to-import-a-config-file-into-docker-container-of-logstash/203099 "2019-10-15T07:46:27Z")
Hi Logstash Ninja Masters, I recently spun up a Logstash docker container (v 7.4.0), to replace a local-app version of LS on the same Ubuntu box. I‘m confused how to instruct the Docker LS to use the configuration that… --- ## [Help explain Logstash with rollover](https://discuss.elastic.co/t/help-explain-logstash-with-rollover/203572)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 0\ **Last updated:** [October 15, 2019, 7:19am UTC](https://discuss.elastic.co/t/help-explain-logstash-with-rollover/203572 "2019-10-15T07:19:49Z")
Hi, We're using the OSS version of the stack and are ingesting logs in the following way: Syslog -\> Logstash -\> Elasticsearch We want to utilize the Rollover API but we're having difficulty grasping how to implement i… --- ## [Can path of file in input plugins set to D drive/disk path on Windows OS?](https://discuss.elastic.co/t/can-path-of-file-in-input-plugins-set-to-d-drive-disk-path-on-windows-os/202416)
**Author:** [@zxc654951](https://discuss.elastic.co/u/zxc654951)\ **Replies:** 2\ **Last updated:** [October 15, 2019, 5:55am UTC](https://discuss.elastic.co/t/can-path-of-file-in-input-plugins-set-to-d-drive-disk-path-on-windows-os/202416 "2019-10-15T05:55:59Z")
Hello. I can run logstash.bat on Windows normally with path to c drive. I want to change path to d drive, but it won't work. Turn the debug on, and report like: \[2019-10-05T14:49:18,537\]\[DEBUG\]\[logstash.instrument.peri… --- ## [Question for Loadbalancing with logstash](https://discuss.elastic.co/t/question-for-loadbalancing-with-logstash/203560)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 0\ **Last updated:** [October 15, 2019, 4:26am UTC](https://discuss.elastic.co/t/question-for-loadbalancing-with-logstash/203560 "2019-10-15T04:26:33Z")
Hello everyone so i have this kind of setup i have one elastic cluster and 4 logstash, 2 of them are for collecting log from filebeat, and two of them are for processing the log, between the logstash processing and the … --- ## [Add new filter questions](https://discuss.elastic.co/t/add-new-filter-questions/203533)
**Author:** [@sud0](https://discuss.elastic.co/u/sud0)\ **Replies:** 4\ **Last updated:** [October 15, 2019, 3:16am UTC](https://discuss.elastic.co/t/add-new-filter-questions/203533 "2019-10-15T03:16:41Z")
Hi. I'm trying to parse the id filed in my message column so that I can group by id to have a number of unique transactions in my dashboard. Currently, the filter is: if \[type\] == "web\_industry\_log" { muta… --- ## [Question about time addition by Ruby filter](https://discuss.elastic.co/t/question-about-time-addition-by-ruby-filter/203136)
**Author:** [@smash](https://discuss.elastic.co/u/smash)\ **Replies:** 2\ **Last updated:** [October 15, 2019, 1:55am UTC](https://discuss.elastic.co/t/question-about-time-addition-by-ruby-filter/203136 "2019-10-15T01:55:45Z")
The following was correctly output to the "date\_time\_1" field after adding 9 hours to the date and time of "Timestamp".The following is the source ruby { code =\> "event.set('tt',event.get('\[@timestamp\]').time.lo… --- ## [Logstash ver 7.4.0 in Docker :: Set AutoConfig Refresh?](https://discuss.elastic.co/t/logstash-ver-7-4-0-in-docker-set-autoconfig-refresh/203515)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [October 14, 2019, 9:53pm UTC](https://discuss.elastic.co/t/logstash-ver-7-4-0-in-docker-set-autoconfig-refresh/203515 "2019-10-14T21:53:22Z")
Hi ELK Wise Ones… I recently spun up Logstash 7.4.0 as a Docker container. My host machine is a Ubuntu 16.04.4, and my Docker is 17.09.0-ce, build afdb6d4. I was following the procedure to set automatic config reloadin… --- ## [SNS Output - Logstash in EKS - using fine grained Service account IAM roles](https://discuss.elastic.co/t/sns-output-logstash-in-eks-using-fine-grained-service-account-iam-roles/202847)
**Author:** [@chandra2037](https://discuss.elastic.co/u/chandra2037)\ **Replies:** 1\ **Last updated:** [October 14, 2019, 9:32pm UTC](https://discuss.elastic.co/t/sns-output-logstash-in-eks-using-fine-grained-service-account-iam-roles/202847 "2019-10-14T21:32:47Z")
Elasticsearch Version: 7.4.0 Logstash Version: 7.4.0 Deployed in Amazon EKS as a POD using official Logstash Docker container. Trying to publish messages to Amazon SNS using Logstash SNS output plugin. Amazon introdu… --- ## [Elasticsearch as a Logstash input](https://discuss.elastic.co/t/elasticsearch-as-a-logstash-input/203493)
**Author:** [@Ankit\_Nahata](https://discuss.elastic.co/u/Ankit_Nahata)\ **Replies:** 0\ **Last updated:** [October 14, 2019, 4:53pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-logstash-input/203493 "2019-10-14T16:53:12Z")
Hi, Thanks for looking into this.! My problem: I need to get the count of documents in an elastic index every hour and store that back in another elastic index. Same I have to do for oracle and get the counts and store… --- ## [Udp output not working](https://discuss.elastic.co/t/udp-output-not-working/203109)
**Author:** [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\ **Replies:** 1\ **Last updated:** [October 14, 2019, 2:07pm UTC](https://discuss.elastic.co/t/udp-output-not-working/203109 "2019-10-14T14:07:00Z")
Hi admin, I have simple logstash config where i am routing(shipping) logs to some other logstash based on the beat host name, below is the logstash config. I removed the IP, but is not shipping any logs to their server.… --- ## [Pipeline to pipeline distributor, how not to lose events](https://discuss.elastic.co/t/pipeline-to-pipeline-distributor-how-not-to-lose-events/202820)
**Author:** [@nahiko2000](https://discuss.elastic.co/u/nahiko2000)\ **Replies:** 3\ **Last updated:** [October 14, 2019, 12:19pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-distributor-how-not-to-lose-events/202820 "2019-10-14T12:19:46Z")
Hello TL;DR - What pipelines must be set a persistent queues to avoid losing data, in a pepeline-to-pipeline distributor pattern We are using Pipeline to pipeline comunication in Logstash. We use the distributor patter… --- ## [Where can i learning the official grammar of ruby code which apply in logstash script](https://discuss.elastic.co/t/where-can-i-learning-the-official-grammar-of-ruby-code-which-apply-in-logstash-script/203368)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 1\ **Last updated:** [October 14, 2019, 12:10pm UTC](https://discuss.elastic.co/t/where-can-i-learning-the-official-grammar-of-ruby-code-which-apply-in-logstash-script/203368 "2019-10-14T12:10:42Z")
Hi experts, Would you please provide some materials or WEB Link for me that include the introduction of grammar of ruby code which apply in logstash script. Such like this: ruby { code =\> 'event.get("message").sca… --- ## [How to transfer multiple json files having common object to one index in elasticseach](https://discuss.elastic.co/t/how-to-transfer-multiple-json-files-having-common-object-to-one-index-in-elasticseach/203442)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 0\ **Last updated:** [October 14, 2019, 11:58am UTC](https://discuss.elastic.co/t/how-to-transfer-multiple-json-files-having-common-object-to-one-index-in-elasticseach/203442 "2019-10-14T11:58:54Z")
I m trying to transfer more than different json files to a index. each json files have common object. I have already transferred multiple csv files using translate filter. Seems the method I used for csv transfer not co… --- ## [Logstash - ELK stack - conf error](https://discuss.elastic.co/t/logstash-elk-stack-conf-error/203190)
**Author:** [@rishikaswarnkar](https://discuss.elastic.co/u/rishikaswarnkar)\ **Replies:** 1\ **Last updated:** [October 14, 2019, 10:37am UTC](https://discuss.elastic.co/t/logstash-elk-stack-conf-error/203190 "2019-10-14T10:37:22Z")
I am trying to configure the logstash to my ELK stack and i keep running into this error. I have Java version 8. D:\\workplace\\Elasticsearch\\logstash-7.4.0\>cd bin D:\\workplace\\Elasticsearch\\logstash-7.4.0\\bin\>logstash … --- ## [Logstash hive](https://discuss.elastic.co/t/logstash-hive/203406)
**Author:** [@rwr](https://discuss.elastic.co/u/rwr)\ **Replies:** 1\ **Last updated:** [October 14, 2019, 10:13am UTC](https://discuss.elastic.co/t/logstash-hive/203406 "2019-10-14T10:13:34Z")
can logstash connect to hive ? --- ## [I am getting break circuit exception in logstash logs,](https://discuss.elastic.co/t/i-am-getting-break-circuit-exception-in-logstash-logs/200980)
**Author:** [@satyam2593](https://discuss.elastic.co/u/satyam2593)\ **Replies:** 2\ **Last updated:** [October 14, 2019, 10:06am UTC](https://discuss.elastic.co/t/i-am-getting-break-circuit-exception-in-logstash-logs/200980 "2019-10-14T10:06:11Z")
In my cluster i have two Elastic servers and one logstash. what is the solution to avoid these kind of issues and how can i reduce my bulk request count and where to increase the JVM heap size. Below is the error i am ge… --- ## [Logstash and ASAM XIL](https://discuss.elastic.co/t/logstash-and-asam-xil/203423)
**Author:** [@AndyWalker](https://discuss.elastic.co/u/AndyWalker)\ **Replies:** 0\ **Last updated:** [October 14, 2019, 9:59am UTC](https://discuss.elastic.co/t/logstash-and-asam-xil/203423 "2019-10-14T09:59:34Z")
Hello All, we're looking at using Logstash to ingest data from our dSPace rigs using the ASAM XIL API. has anyone had experience with the XIL API and Logstash? Maybe an example pipeline? Thanks. --- ## [Index the content of KV file with Logstash in one document dynamically](https://discuss.elastic.co/t/index-the-content-of-kv-file-with-logstash-in-one-document-dynamically/203210)
**Author:** [@Bouraoui\_Kacem](https://discuss.elastic.co/u/Bouraoui_Kacem)\ **Replies:** 2\ **Last updated:** [October 14, 2019, 9:55am UTC](https://discuss.elastic.co/t/index-the-content-of-kv-file-with-logstash-in-one-document-dynamically/203210 "2019-10-14T09:55:07Z")
I have many properties files with the following structures key = Value , for example : rejets.cache.temporaire.time=60 data.cache.referentiel.time=10 data.cache.temporaire.time=60 data.cache.getinfos.time=200 I want to… --- ## [Netflow Logstash](https://discuss.elastic.co/t/netflow-logstash/202277)
**Author:** [@TheSun](https://discuss.elastic.co/u/TheSun)\ **Replies:** 3\ **Last updated:** [October 14, 2019, 3:10am UTC](https://discuss.elastic.co/t/netflow-logstash/202277 "2019-10-14T03:10:15Z")
Hi, I have problems processing data netflow. I can not create index. This is my configuration, please help me.!!! input { udp { port =\> 1234 type =\> "netflow" codec =\> netflow } } output { if ( \[type\] == "net… --- ## [How to use keystore in Dockerized logstash?](https://discuss.elastic.co/t/how-to-use-keystore-in-dockerized-logstash/199694)
**Author:** [@furiabhavesh](https://discuss.elastic.co/u/furiabhavesh)\ **Replies:** 2\ **Last updated:** [October 13, 2019, 6:35am UTC](https://discuss.elastic.co/t/how-to-use-keystore-in-dockerized-logstash/199694 "2019-10-13T06:35:07Z")
I am referring this help document to setup logstash 7.3.2 using docker and wanted to pass few environment variables which I further want to access in my input block of the.conf files Eg: docker run -e TCP\_PORT=123... i… --- ## [Logstash does not delete processed files](https://discuss.elastic.co/t/logstash-does-not-delete-processed-files/203322)
**Author:** [@Rysiu](https://discuss.elastic.co/u/Rysiu)\ **Replies:** 2\ **Last updated:** [October 13, 2019, 6:21am UTC](https://discuss.elastic.co/t/logstash-does-not-delete-processed-files/203322 "2019-10-13T06:21:58Z")
Hello, My Logstash works fine. However, I can't force deleting old files. Part of the configuration file: input { file { mode =\> read path =\> "/home/user/data/data\_pre/\*\_0" start\_position =\> "beginning" file\_… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=461) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=463)