# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=860 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 861 --- ## [Does http\_poller plugin supports OAUTH 2.0?](https://discuss.elastic.co/t/does-http-poller-plugin-supports-oauth-2-0/110599)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [December 9, 2017, 4:37pm UTC](https://discuss.elastic.co/t/does-http-poller-plugin-supports-oauth-2-0/110599 "2017-12-09T16:37:33Z")
Hi All, I have a API which works on OAuth 2.0 A OAuth 2.0 API requires Authorization as header which holds bearer token This is generated using Refresh Token, ClientID, Client Secret given values.( this are fixed valu… --- ## [Custom plugin - octokit / sawyer / faraday versions](https://discuss.elastic.co/t/custom-plugin-octokit-sawyer-faraday-versions/108921)
**Author:** [@NirKa](https://discuss.elastic.co/u/NirKa)\ **Replies:** 1\ **Last updated:** [December 9, 2017, 2:08pm UTC](https://discuss.elastic.co/t/custom-plugin-octokit-sawyer-faraday-versions/108921 "2017-12-09T14:08:36Z")
Hi, I'm developing a new plugin that requires faraday (~\> 0.9.0). When I try to install the plugins' gem I get conflict error: ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "… --- ## [Logstash going to sleep without pulling files](https://discuss.elastic.co/t/logstash-going-to-sleep-without-pulling-files/110864)
**Author:** [@Shreya\_Chatterjee](https://discuss.elastic.co/u/Shreya_Chatterjee)\ **Replies:** 4\ **Last updated:** [December 9, 2017, 9:54am UTC](https://discuss.elastic.co/t/logstash-going-to-sleep-without-pulling-files/110864 "2017-12-09T09:54:40Z")
log file excerpt: 2017-11-06 11:54:43,333; \[LOG\_LEVEL=ALWAYS, CMPNT\_NM=com.fmr.ifeb.alayer.cache.ehcache.ApplicationCacheImpl, MESSAGE=Initialized cache named 'oscarJdbcDaxCache'\] filebeat.yml --- filebeat.prospector… --- ## [Unable to load csv file into elastic through logstash](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elastic-through-logstash/110882)
**Author:** [@Yuvaraja\_kannan](https://discuss.elastic.co/u/Yuvaraja_kannan)\ **Replies:** 2\ **Last updated:** [December 9, 2017, 7:16am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elastic-through-logstash/110882 "2017-12-09T07:16:56Z")
HI All, I am trying to load a CSV file into elastic by initializing logstash through the below command, .\\bin\\logstash -f C:\\data\\logstash\_cars.config But data is not transferred due to some reason. I am not able to s… --- ## [Logstash 6.0, not logging similar messages from two different hosts](https://discuss.elastic.co/t/logstash-6-0-not-logging-similar-messages-from-two-different-hosts/110902)
**Author:** [@kidn3ys](https://discuss.elastic.co/u/kidn3ys)\ **Replies:** 2\ **Last updated:** [December 9, 2017, 12:23am UTC](https://discuss.elastic.co/t/logstash-6-0-not-logging-similar-messages-from-two-different-hosts/110902 "2017-12-09T00:23:47Z")
Hello, Brand new to ELK but I can't seem to find another log or method to try to narrow down what is happening in this case. Both devices are configured to send log messages to 8514/tcp where logstash is listening. The… --- ## [Logstash consumer group not shown in zookeeper](https://discuss.elastic.co/t/logstash-consumer-group-not-shown-in-zookeeper/110901)
**Author:** [@Nisha\_S](https://discuss.elastic.co/u/Nisha_S)\ **Replies:** 0\ **Last updated:** [December 8, 2017, 10:23pm UTC](https://discuss.elastic.co/t/logstash-consumer-group-not-shown-in-zookeeper/110901 "2017-12-08T22:23:02Z")
Hi! I have a 6 node cluster running kafka broker on 3 machines and a topic with 18 partitions. I started logstash process on 3 out of six nodes with a common consumer group subscribing to this topic. But when I descri… --- ## [LogStash won't bind to any port](https://discuss.elastic.co/t/logstash-wont-bind-to-any-port/110886)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 2\ **Last updated:** [December 8, 2017, 9:57pm UTC](https://discuss.elastic.co/t/logstash-wont-bind-to-any-port/110886 "2017-12-08T21:57:12Z")
I'm running LS 5.4 on Ubuntu 16.04 LTS It's been running happily tailing locally written bro files for several months. I stopped it to do some troubleshooting and testing to add some functionality, but I'm having troub… --- ## [Localhost always used on healthchecks](https://discuss.elastic.co/t/localhost-always-used-on-healthchecks/110857)
**Author:** [@P1llus](https://discuss.elastic.co/u/P1llus)\ **Replies:** 0\ **Last updated:** [December 8, 2017, 2:26pm UTC](https://discuss.elastic.co/t/localhost-always-used-on-healthchecks/110857 "2017-12-08T14:26:54Z")
Hello! I have been trying to figure out why my logstash always reuses localhost for healthchecks, even if not defined ANYWHERE. Here is some snippets from the error log (full gist here: https://www.irccloud.com/pastebin… --- ## [Add NOW timestamp to field w/o use of @timestamp](https://discuss.elastic.co/t/add-now-timestamp-to-field-w-o-use-of-timestamp/110773)
**Author:** [@aaronloesattv](https://discuss.elastic.co/u/aaronloesattv)\ **Replies:** 4\ **Last updated:** [December 8, 2017, 5:45pm UTC](https://discuss.elastic.co/t/add-now-timestamp-to-field-w-o-use-of-timestamp/110773 "2017-12-08T17:45:08Z")
i've been trying to add the "@received\_at" timestamp that indicates when logstash received the log message. this helps us determine if our logging infrastructure is lagging behind our logging events. The logs we receive … --- ## [Logstash - grokfilter is successful using grokdebug, but fails in live](https://discuss.elastic.co/t/logstash-grokfilter-is-successful-using-grokdebug-but-fails-in-live/110823)
**Author:** [@ajwillo](https://discuss.elastic.co/u/ajwillo)\ **Replies:** 14\ **Last updated:** [December 8, 2017, 4:07pm UTC](https://discuss.elastic.co/t/logstash-grokfilter-is-successful-using-grokdebug-but-fails-in-live/110823 "2017-12-08T16:07:23Z")
I have a grok filter that I have built and tested on http://grokdebug.herokuapp.com. the test sample is: \<190\>3259426: \*Dec 7 17:43:09.599 UTC: %SEC-6-IPACCESSLOGP: list ACL\_3P denied tcp 10.10.10.2(8000) -\> 10.20.20.2… --- ## [Nested json field mutate string to number](https://discuss.elastic.co/t/nested-json-field-mutate-string-to-number/110846)
**Author:** [@fpunzo](https://discuss.elastic.co/u/fpunzo)\ **Replies:** 2\ **Last updated:** [December 8, 2017, 2:38pm UTC](https://discuss.elastic.co/t/nested-json-field-mutate-string-to-number/110846 "2017-12-08T14:38:05Z")
Hello, I'm trying to convert a nested json field from a string to an integer. I've tried everything I've found on the web. I'm hoping you can give me some direction as to what I'm doing wrong. I'm trying to convert the r… --- ## [Could logstash distribute data to different nodes?](https://discuss.elastic.co/t/could-logstash-distribute-data-to-different-nodes/110085)
**Author:** [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Replies:** 7\ **Last updated:** [December 8, 2017, 1:58pm UTC](https://discuss.elastic.co/t/could-logstash-distribute-data-to-different-nodes/110085 "2017-12-08T13:58:17Z")
Hi, I want to distribute data into many nodes from logstash. is it feasible? because I want to reduce the risk. If one node crash others can share the loading. thank you in advance ! --- ## [Remove trailing white space in logstash filter](https://discuss.elastic.co/t/remove-trailing-white-space-in-logstash-filter/110819)
**Author:** [@platso588](https://discuss.elastic.co/u/platso588)\ **Replies:** 6\ **Last updated:** [December 8, 2017, 12:25pm UTC](https://discuss.elastic.co/t/remove-trailing-white-space-in-logstash-filter/110819 "2017-12-08T12:25:53Z")
I am trying to add filter for the below log format: \[24/Oct/2017 15:04:53 \] cluster WARNING Picking RM HA: ha The filter I have added is: \[%{MONTHDAY:logDate}/%{MONTH:logMonth}/%{YEAR:logYear} %{TIME:logTime} \]%… --- ## [Logstash in docker input jdbc plugin not updating last\_run\_metadata\_path](https://discuss.elastic.co/t/logstash-in-docker-input-jdbc-plugin-not-updating-last-run-metadata-path/37411)
**Author:** [@sripri](https://discuss.elastic.co/u/sripri)\ **Replies:** 4\ **Last updated:** [December 8, 2017, 12:15pm UTC](https://discuss.elastic.co/t/logstash-in-docker-input-jdbc-plugin-not-updating-last-run-metadata-path/37411 "2017-12-08T12:15:42Z")
Hi I am trying to use the input jdbc plugin jdbc { type =\> "logs" jdbc\_driver\_library =\> "/opt/logstash/driver/ojdbc6.jar" jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_connection\_string =\> … --- ## [Help please.. breaking changes?](https://discuss.elastic.co/t/help-please-breaking-changes/108010)
**Author:** [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Replies:** 15\ **Last updated:** [December 8, 2017, 8:52am UTC](https://discuss.elastic.co/t/help-please-breaking-changes/108010 "2017-12-08T08:52:58Z")
Upgrading to 6.0.0 of the ELK stack Can anyone please help me why i get this entrys over and over in the Logstash log \[2017-11-16T20:49:36,973\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticse… --- ## [Keep variables in Logstash](https://discuss.elastic.co/t/keep-variables-in-logstash/110501)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 6\ **Last updated:** [December 8, 2017, 8:43am UTC](https://discuss.elastic.co/t/keep-variables-in-logstash/110501 "2017-12-08T08:43:04Z")
I have an xml file which I am currently ingesting and it is quite nested. To simplify it, I am splitting the filelds as follows: \<(Header|parameter|differentparameter) This is configured in the input section, with mult… --- ## [Index template not taking effect on index created](https://discuss.elastic.co/t/index-template-not-taking-effect-on-index-created/110464)
**Author:** [@pope843](https://discuss.elastic.co/u/pope843)\ **Replies:** 5\ **Last updated:** [December 8, 2017, 8:27am UTC](https://discuss.elastic.co/t/index-template-not-taking-effect-on-index-created/110464 "2017-12-08T08:27:00Z")
Hello guys, i have a problem on my index creation, the i can see that mappings/settings is created based on the template as checked (GET \_template). but the created index doesnt get the mappings/settings when created. H… --- ## [Logstash input/ouput limits per/sec](https://discuss.elastic.co/t/logstash-input-ouput-limits-per-sec/110239)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 4\ **Last updated:** [December 8, 2017, 4:48am UTC](https://discuss.elastic.co/t/logstash-input-ouput-limits-per-sec/110239 "2017-12-08T04:48:55Z")
hi everyone. could someone point out the logstash limit, plz? let me say my scenario first. i use cloudwatch\_logs input \> do 50-line grok match filter \> elasticsearch output base on my script, i would like to know fe… --- ## [Copy part of a field into new field](https://discuss.elastic.co/t/copy-part-of-a-field-into-new-field/110456)
**Author:** [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Replies:** 10\ **Last updated:** [December 8, 2017, 4:41am UTC](https://discuss.elastic.co/t/copy-part-of-a-field-into-new-field/110456 "2017-12-08T04:41:34Z")
Alright, I've got another question is it possible to copy part of one field into another? I.E. I have NewProcessName - C:\\Windows\\SysWOW64\\dllhost.exe I want to move everything after last backslash to new field Susp… --- ## [Grok Help Request - Get FileDate from File Name into new field FileDate](https://discuss.elastic.co/t/grok-help-request-get-filedate-from-file-name-into-new-field-filedate/110404)
**Author:** [@sconrod](https://discuss.elastic.co/u/sconrod)\ **Replies:** 11\ **Last updated:** [December 8, 2017, 1:01am UTC](https://discuss.elastic.co/t/grok-help-request-get-filedate-from-file-name-into-new-field-filedate/110404 "2017-12-08T01:01:47Z")
Hi, I have a lot of csv files I am importing using the CSV filter but they have not timestamp included in the logs but I do have the date the log was produced in the filename, therefore, I want to create a new filedate f… --- ## [Logstash kafka input combining topics and topics\_pattern](https://discuss.elastic.co/t/logstash-kafka-input-combining-topics-and-topics-pattern/110747)
**Author:** [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Replies:** 0\ **Last updated:** [December 7, 2017, 9:09pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-combining-topics-and-topics-pattern/110747 "2017-12-07T21:09:01Z")
We are collection logs from different kafka topics which are been created dynamically. The common pattern of the topic naming is application\_name-log\_type-logs. We have also some important topics too like qafilebeat, prf… --- ## [Logstash authentication with TCP](https://discuss.elastic.co/t/logstash-authentication-with-tcp/110558)
**Author:** [@NenadP](https://discuss.elastic.co/u/NenadP)\ **Replies:** 5\ **Last updated:** [December 7, 2017, 8:31pm UTC](https://discuss.elastic.co/t/logstash-authentication-with-tcp/110558 "2017-12-07T20:31:38Z")
I am delivering logs from Pivotal web service (based on Cloud Foundry) to remote ELK. On Pivotal I've created log drain service which delivers logs to logstash via syslog TCP connection. Everything works fine but appare… --- ## [Logstash-Output-Mongodb to Mongo 3.4](https://discuss.elastic.co/t/logstash-output-mongodb-to-mongo-3-4/109997)
**Author:** [@jshipley-zyston](https://discuss.elastic.co/u/jshipley-zyston)\ **Replies:** 1\ **Last updated:** [December 7, 2017, 8:03pm UTC](https://discuss.elastic.co/t/logstash-output-mongodb-to-mongo-3-4/109997 "2017-12-07T20:03:31Z")
I am attempting to use the MongoDB Logstash Output plugin to output some data to the online MongoDB 3.4 (Atlas). Every time Logstash tries to send the data to the MongoDB I receive the following error: \[2017-11-24T22:2… --- ## [Logstash permission problem](https://discuss.elastic.co/t/logstash-permission-problem/110175)
**Author:** [@Tal\_Druckmann](https://discuss.elastic.co/u/Tal_Druckmann)\ **Replies:** 16\ **Last updated:** [December 7, 2017, 5:49pm UTC](https://discuss.elastic.co/t/logstash-permission-problem/110175 "2017-12-07T17:49:08Z")
I am running logstash on windows 2016 vm . I use it from administrator command line. I keep getting this message. It won't let me save my visualization on kibana and keeps me from shutting it down (ctrl+c). This is the … --- ## [Easier ingestion of CSV files](https://discuss.elastic.co/t/easier-ingestion-of-csv-files/110665)
**Author:** [@scch](https://discuss.elastic.co/u/scch)\ **Replies:** 2\ **Last updated:** [December 7, 2017, 2:50pm UTC](https://discuss.elastic.co/t/easier-ingestion-of-csv-files/110665 "2017-12-07T14:50:28Z")
Is there a way to ingest csv file, where a csv field can automatically map to a specified column irrespective of its location in csv file ? and also, how csv filter can automatically drop the header line during increme… --- ## [Grok Parse Failure - Logger](https://discuss.elastic.co/t/grok-parse-failure-logger/110674)
**Author:** [@Rebelpyr7](https://discuss.elastic.co/u/Rebelpyr7)\ **Replies:** 3\ **Last updated:** [December 7, 2017, 2:35pm UTC](https://discuss.elastic.co/t/grok-parse-failure-logger/110674 "2017-12-07T14:35:18Z")
Ok so the baseline of the log that I am trying to parse is the results of a command line speed test. I am using the logger command to export the results to my ELK server. I've done the leg work and gotten a pattern to wo… --- ## [Process array elements as objects](https://discuss.elastic.co/t/process-array-elements-as-objects/110682)
**Author:** [@Peter\_Boer](https://discuss.elastic.co/u/Peter_Boer)\ **Replies:** 2\ **Last updated:** [December 7, 2017, 2:24pm UTC](https://discuss.elastic.co/t/process-array-elements-as-objects/110682 "2017-12-07T14:24:19Z")
I am trying to add json data to my elasticsearch with arrays in its field. My setup works, but I want the arrays to become separate fields. Is this possible in logstash or should i create a json parser to fix my problem? … --- ## [Logstash Error: Address already in use](https://discuss.elastic.co/t/logstash-error-address-already-in-use/110677)
**Author:** [@Jonathan\_Kosgei](https://discuss.elastic.co/u/Jonathan_Kosgei)\ **Replies:** 2\ **Last updated:** [December 7, 2017, 2:23pm UTC](https://discuss.elastic.co/t/logstash-error-address-already-in-use/110677 "2017-12-07T14:23:11Z")
I have the following config input { beats { 'port' =\> '5044' } } output { cloudwatch { 'region'=\> 'us-west-2' } } I have only one conf file in /usr/share/logstash/pipeline/ bash-4.2$ ls /usr/share/logstash/pi… --- ## [S3 input configuration in logstash to get the logs present in zip folder in the bucket](https://discuss.elastic.co/t/s3-input-configuration-in-logstash-to-get-the-logs-present-in-zip-folder-in-the-bucket/110670)
**Author:** [@npasha](https://discuss.elastic.co/u/npasha)\ **Replies:** 0\ **Last updated:** [December 7, 2017, 12:32pm UTC](https://discuss.elastic.co/t/s3-input-configuration-in-logstash-to-get-the-logs-present-in-zip-folder-in-the-bucket/110670 "2017-12-07T12:32:49Z")
I can get the file present in s3 bucket after configuring logstash, but i dont know how to get the files present in zip or tar.gz folders in s3 bucket. my logstash s3 input configuration input { s3 { bucket =\> "xxxxx… --- ## [Logstash is not populating custom index](https://discuss.elastic.co/t/logstash-is-not-populating-custom-index/110388)
**Author:** [@rromanovic](https://discuss.elastic.co/u/rromanovic)\ **Replies:** 2\ **Last updated:** [December 7, 2017, 10:48am UTC](https://discuss.elastic.co/t/logstash-is-not-populating-custom-index/110388 "2017-12-07T10:48:12Z")
Hi all, I have a problem when I try to populate custom index. First, I create index using following syntax: PUT testIndex { "mappings": { "txs": { "properties": { "date": { "type": "date", "format": "dd-MM-yyyy… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=859) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=861)