# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=978 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 979 --- ## [Want to Process Multiple Files that exist in One Folder](https://discuss.elastic.co/t/want-to-process-multiple-files-that-exist-in-one-folder/28384)
**Author:** [@bhutchinson](https://discuss.elastic.co/u/bhutchinson)\ **Replies:** 9\ **Last updated:** [September 1, 2015, 9:05pm UTC](https://discuss.elastic.co/t/want-to-process-multiple-files-that-exist-in-one-folder/28384 "2015-09-01T21:05:10Z")
Windows 7 Elasticsearch v1.7.0 Logstash v1.5.4 I have multiple files that I want to run through Logstash and have Logstash insert into elasticsearch. Each of my files are apache access files. All my files are in one… --- ## [Error when running logstash](https://discuss.elastic.co/t/error-when-running-logstash/86888)
**Author:** [@shane\_lee](https://discuss.elastic.co/u/shane_lee)\ **Replies:** 1\ **Last updated:** [May 24, 2017, 4:37am UTC](https://discuss.elastic.co/t/error-when-running-logstash/86888 "2017-05-24T04:37:54Z")
Hi, I am trying to simply execute logstash via child process in node js. It keeps exiting due to error with log4j ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only error… --- ## [Calculate Date difference in LOGSTASH based on conditional text](https://discuss.elastic.co/t/calculate-date-difference-in-logstash-based-on-conditional-text/86759)
**Author:** [@Senz79](https://discuss.elastic.co/u/Senz79)\ **Replies:** 5\ **Last updated:** [May 24, 2017, 3:47am UTC](https://discuss.elastic.co/t/calculate-date-difference-in-logstash-based-on-conditional-text/86759 "2017-05-24T03:47:04Z")
I have a condition here: I have a series of logs where it starts from startTransaction and ends at endtransaction. Now between there two events there are other logs. My requirement is to capture the log time for the ab… --- ## [Shipping logs via Proxy with Logstash](https://discuss.elastic.co/t/shipping-logs-via-proxy-with-logstash/24723)
**Author:** [@otisg](https://discuss.elastic.co/u/otisg)\ **Replies:** 8\ **Last updated:** [May 24, 2017, 2:12am UTC](https://discuss.elastic.co/t/shipping-logs-via-proxy-with-logstash/24723 "2017-05-24T02:12:34Z")
Hi, How do I tell Logstash to ship logs via a proxy? I searched and searched, but could not find any documentation about shipping via a proxy. Maybe sending data through proxy is simply not supported at this time? T… --- ## [How to have multiple matches for same filter in grok..?](https://discuss.elastic.co/t/how-to-have-multiple-matches-for-same-filter-in-grok/86883)
**Author:** [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\ **Replies:** 0\ **Last updated:** [May 24, 2017, 1:50am UTC](https://discuss.elastic.co/t/how-to-have-multiple-matches-for-same-filter-in-grok/86883 "2017-05-24T01:50:30Z")
I am trying to parse the thread dump data, which is below: "NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable \[0x49aa5000\]java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWa… --- ## [Graphite Output Plugin Question](https://discuss.elastic.co/t/graphite-output-plugin-question/86855)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 2\ **Last updated:** [May 23, 2017, 10:27pm UTC](https://discuss.elastic.co/t/graphite-output-plugin-question/86855 "2017-05-23T22:27:46Z")
Hi, If i use a graphite solution such as a hosted provider, whom needs an API key, I don't see a way to pass this.. Documentation of sending to here: https://www.hostedgraphite.com/docs/ Any help appreciated. Than… --- ## [Logstash 5.4 docker image can't find configuration files](https://discuss.elastic.co/t/logstash-5-4-docker-image-cant-find-configuration-files/85630)
**Author:** [@endersonmaia](https://discuss.elastic.co/u/endersonmaia)\ **Replies:** 3\ **Last updated:** [May 23, 2017, 7:27pm UTC](https://discuss.elastic.co/t/logstash-5-4-docker-image-cant-find-configuration-files/85630 "2017-05-23T19:27:00Z")
I was using docker.elastic.io/logstash:5.3.0 and everything was working fine, I just changed from 5.3.0 to 5.4.0 for the Dockerfile, and now when start it can't find the configuraiton file ? Did anything change in thi… --- ## [Multiline codec reading entire log file as one line](https://discuss.elastic.co/t/multiline-codec-reading-entire-log-file-as-one-line/86858)
**Author:** [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 7:07pm UTC](https://discuss.elastic.co/t/multiline-codec-reading-entire-log-file-as-one-line/86858 "2017-05-23T19:07:20Z")
Hello, I'm having an issue with my multiline codec working correctly. Most of the lines in my file are a single line but some of them do have large stack traces in them at some points. When I run my config file the ent… --- ## [Grok Empty Field Parse Failure](https://discuss.elastic.co/t/grok-empty-field-parse-failure/57462)
**Author:** [@mai\_le](https://discuss.elastic.co/u/mai_le)\ **Replies:** 3\ **Last updated:** [May 23, 2017, 6:47pm UTC](https://discuss.elastic.co/t/grok-empty-field-parse-failure/57462 "2017-05-23T18:47:19Z")
Hello, I'm trying to parse the log below: "------------------------------------------------------------------------------------------------------------------------\\n01/03/2016 19:05:15;01/03/2016 19:05:18;Method;V2.1… --- ## [Multiple Date formats in one file](https://discuss.elastic.co/t/multiple-date-formats-in-one-file/86725)
**Author:** [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Replies:** 3\ **Last updated:** [May 23, 2017, 6:00pm UTC](https://discuss.elastic.co/t/multiple-date-formats-in-one-file/86725 "2017-05-23T18:00:42Z")
Hello, I'm having an issue with date parsing multiple dates in one file. is there anyway to do multiple message matches / GROKS depending on what the log line is? For instance: USING this GROK match =\> { "message… --- ## [How should I format the log to get concatenated field?](https://discuss.elastic.co/t/how-should-i-format-the-log-to-get-concatenated-field/86842)
**Author:** [@Singard123](https://discuss.elastic.co/u/Singard123)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 5:32pm UTC](https://discuss.elastic.co/t/how-should-i-format-the-log-to-get-concatenated-field/86842 "2017-05-23T17:32:16Z")
right now my log looks like this: params={"download\_token"=\\u003e"1495559576" How can I format it or using logstash to parse it so that I can get a field like: params.download\_token: 1495559576 Thanks in advance --- ## [Need help for reading the txt attachments in Mail](https://discuss.elastic.co/t/need-help-for-reading-the-txt-attachments-in-mail/86831)
**Author:** [@pratapkarn](https://discuss.elastic.co/u/pratapkarn)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 4:06pm UTC](https://discuss.elastic.co/t/need-help-for-reading-the-txt-attachments-in-mail/86831 "2017-05-23T16:06:43Z")
Hi All, Can you help me in getting some direction on how to parse the txt or pdf attachments in email. I am using logstash-input-imap for reading the mail from Email account. Mails without attachments is working fi… --- ## [Sample setting with elasticsearch input plugin doesn't work](https://discuss.elastic.co/t/sample-setting-with-elasticsearch-input-plugin-doesnt-work/84764)
**Author:** [@shin0higuchi](https://discuss.elastic.co/u/shin0higuchi)\ **Replies:** 1\ **Last updated:** [May 23, 2017, 3:58pm UTC](https://discuss.elastic.co/t/sample-setting-with-elasticsearch-input-plugin-doesnt-work/84764 "2017-05-23T15:58:43Z")
A sample setting on https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html may not work. Sample input { elasticsearch { hosts =\> "es.production.mysite.org" index =\> "mydata-2018.0… --- ## [Remove a field from an array based on another field's value](https://discuss.elastic.co/t/remove-a-field-from-an-array-based-on-another-fields-value/86474)
**Author:** [@endersonmaia](https://discuss.elastic.co/u/endersonmaia)\ **Replies:** 2\ **Last updated:** [May 23, 2017, 2:51pm UTC](https://discuss.elastic.co/t/remove-a-field-from-an-array-based-on-another-fields-value/86474 "2017-05-23T14:51:40Z")
I need to remove some fields from an array of JSON objects based on the value of another field on the same array element. the input is: { "id": "123456", "pagamentos": \[ {"tipo": "R$", "valor": 23.75, "cartao… --- ## [Change timezone for rotation](https://discuss.elastic.co/t/change-timezone-for-rotation/86580)
**Author:** [@Aled](https://discuss.elastic.co/u/Aled)\ **Replies:** 2\ **Last updated:** [May 23, 2017, 2:15pm UTC](https://discuss.elastic.co/t/change-timezone-for-rotation/86580 "2017-05-23T14:15:44Z")
Hello, i'm using the date-based log rotation with the file output plugin. My problem is that the time is in UTC timezone and i'm in CEST/CET timezone. Consequence is that log are not rotated at midnight as expected. Do… --- ## [Logstash protobuf codec int issue](https://discuss.elastic.co/t/logstash-protobuf-codec-int-issue/86823)
**Author:** [@Roi\_Klein](https://discuss.elastic.co/u/Roi_Klein)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 2:12pm UTC](https://discuss.elastic.co/t/logstash-protobuf-codec-int-issue/86823 "2017-05-23T14:12:05Z")
Hey all, using this proto: \`message foo { optional int64 f = 1; } message boo { optional int64 b = 1; } message aa { oneof bull { foo fa = 1; boo ba = 2; } required int64 b = 3; }… --- ## [Input http plugin and CORS](https://discuss.elastic.co/t/input-http-plugin-and-cors/86815)
**Author:** [@CSchulz](https://discuss.elastic.co/u/CSchulz)\ **Replies:** 1\ **Last updated:** [May 23, 2017, 1:50pm UTC](https://discuss.elastic.co/t/input-http-plugin-and-cors/86815 "2017-05-23T13:50:30Z")
I have enabeld the input http plugin and added some headers for CORS. In general it works. To be secure I have enabled also the authorization, but I am struggling with the OPTIONS request itself. Normally the applicati… --- ## [Logstash aggregate filter doesn't aggregate all messages](https://discuss.elastic.co/t/logstash-aggregate-filter-doesnt-aggregate-all-messages/86802)
**Author:** [@alex88](https://discuss.elastic.co/u/alex88)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 12:02pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-doesnt-aggregate-all-messages/86802 "2017-05-23T12:02:02Z")
i have apache wsgi log: \[Sun Mar 05 14:26:12.858516 2017\] \[wsgi:error\] \[pid 11254:tid 140473096181504\] \[Sun Mar 05 14:26:12.859199 2017\] \[wsgi:error\] \[pid 11254:tid 140473096181504\] /usr/local/lib/python3.5/dist-packag… --- ## [Best practice for reimporting files into Logstash?](https://discuss.elastic.co/t/best-practice-for-reimporting-files-into-logstash/86438)
**Author:** [@mw\_jko](https://discuss.elastic.co/u/mw_jko)\ **Replies:** 3\ **Last updated:** [May 23, 2017, 12:01pm UTC](https://discuss.elastic.co/t/best-practice-for-reimporting-files-into-logstash/86438 "2017-05-23T12:01:28Z")
Hi there, I have trouble with re-importing existing logfiles (600mb each) into logstash or what's the best practice to do this. Our setup is: Multiple hosts with filebeat daemons -\> logstash -\> elasticsearch When s… --- ## [Logstash: If message contains regex add field](https://discuss.elastic.co/t/logstash-if-message-contains-regex-add-field/86799)
**Author:** [@jasdf](https://discuss.elastic.co/u/jasdf)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 11:36am UTC](https://discuss.elastic.co/t/logstash-if-message-contains-regex-add-field/86799 "2017-05-23T11:36:57Z")
Hello, within my Logstash config I'm searching for a regex pattern. If a logfile is found which matches the regex a new field should be added with the found pattern. The if statement is working but I don't know how to … --- ## [Output json](https://discuss.elastic.co/t/output-json/86278)
**Author:** [@shaharz](https://discuss.elastic.co/u/shaharz)\ **Replies:** 2\ **Last updated:** [May 23, 2017, 10:53am UTC](https://discuss.elastic.co/t/output-json/86278 "2017-05-23T10:53:43Z")
hi, Im running logstash to collect json log file. the log file contains lines of logs in json format. here is my configuration input { file { codec =\> "json" path =\> "C:\\Logs\\Client.txt" path =\> "C:\\Logs\\Server.txt… --- ## [Jdbc-streaming using postgres nosql](https://discuss.elastic.co/t/jdbc-streaming-using-postgres-nosql/86793)
**Author:** [@Vittorio](https://discuss.elastic.co/u/Vittorio)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 10:29am UTC](https://discuss.elastic.co/t/jdbc-streaming-using-postgres-nosql/86793 "2017-05-23T10:29:09Z")
Hi there, What about I want to use the nosql feature that postgres has, applied to the logstash filter? How would the statement field look like? --- ## [CSV mutate system load](https://discuss.elastic.co/t/csv-mutate-system-load/86783)
**Author:** [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 9:09am UTC](https://discuss.elastic.co/t/csv-mutate-system-load/86783 "2017-05-23T09:09:23Z")
Hi dear friends, Got one quick question about mutations. I'm parsing CSV log files with logstash and would like to "mutate" some filed values if matching a string. So basically a conf file would containg about 30-40 mu… --- ## [Plugin for Google analytics](https://discuss.elastic.co/t/plugin-for-google-analytics/86777)
**Author:** [@Kamal\_Samra](https://discuss.elastic.co/u/Kamal_Samra)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 7:55am UTC](https://discuss.elastic.co/t/plugin-for-google-analytics/86777 "2017-05-23T07:55:05Z")
Is there any plugin exist for logstash which show google analytics data on kibana? --- ## [Not able to install collate plugin](https://discuss.elastic.co/t/not-able-to-install-collate-plugin/86766)
**Author:** [@deeps123](https://discuss.elastic.co/u/deeps123)\ **Replies:** 2\ **Last updated:** [May 23, 2017, 7:32am UTC](https://discuss.elastic.co/t/not-able-to-install-collate-plugin/86766 "2017-05-23T07:32:06Z")
Hi, With 5.4.0 version of logstash I am not able to install collate plugin. I am getting below errors. Please help. bin/logstash-plugin install logstash-filter-collate Validating logstash-filter-collate Installing lo… --- ## [Logstash assign key value pair into each variables](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583)
**Author:** [@vallikkv](https://discuss.elastic.co/u/vallikkv)\ **Replies:** 8\ **Last updated:** [May 23, 2017, 7:21am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583 "2017-05-23T07:21:05Z")
Hello All, I'm 3 day new to log stash and during this learning process, I tried to filter the unstructured log which contains key & value pairs and want to assign each values to variables. Unstructured log: 017-04-… --- ## [Logstash multiline problem](https://discuss.elastic.co/t/logstash-multiline-problem/86763)
**Author:** [@JIELOGAN](https://discuss.elastic.co/u/JIELOGAN)\ **Replies:** 0\ **Last updated:** [May 23, 2017, 6:27am UTC](https://discuss.elastic.co/t/logstash-multiline-problem/86763 "2017-05-23T06:27:56Z")
hi everyone,i have lots of logs need to be structured,for clear ,there is my data:: ######### ABC 123 abc PPP ZZZZZ ######### I want to: "ABCabcZZZZZ" , i tried the multiline codec and my patt… --- ## [Logstash not showing any output (SOLVED)](https://discuss.elastic.co/t/logstash-not-showing-any-output-solved/28636)
**Author:** [@lookingcloudy](https://discuss.elastic.co/u/lookingcloudy)\ **Replies:** 30\ **Last updated:** [May 23, 2017, 6:59am UTC](https://discuss.elastic.co/t/logstash-not-showing-any-output-solved/28636 "2017-05-23T06:59:26Z")
I'm starting off with Logstash, working through the tutorials. I find the plugin documentation very lacking for someone with no experience with logstash. Here is the config file I am starting with: input { file {… --- ## [Cant append documents on existing index](https://discuss.elastic.co/t/cant-append-documents-on-existing-index/86099)
**Author:** [@nishant2](https://discuss.elastic.co/u/nishant2)\ **Replies:** 3\ **Last updated:** [May 23, 2017, 6:33am UTC](https://discuss.elastic.co/t/cant-append-documents-on-existing-index/86099 "2017-05-23T06:33:13Z")
Hello We are using logstash to sink kafka data to elastic search. When we are trying to insert docs on a non-existing index on elastic search, then logstash creates the index and starts docs there. But when we try t… --- ## [Decrypt data before decoding](https://discuss.elastic.co/t/decrypt-data-before-decoding/86295)
**Author:** [@pollux](https://discuss.elastic.co/u/pollux)\ **Replies:** 1\ **Last updated:** [May 23, 2017, 3:52am UTC](https://discuss.elastic.co/t/decrypt-data-before-decoding/86295 "2017-05-23T03:52:46Z")
Hi, I need to fetch file from ftp. Last time I checked, logstash didn't have an input plug-in for ftp. So I'll mount the ftp and logstash will read from file. The real thing is, can I said to logstash "ok read this f… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=977) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=979)