# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1110 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1111 --- ## [Logstash http.port auto-increment?](https://discuss.elastic.co/t/logstash-http-port-auto-increment/57370)
**Author:** [@ave19](https://discuss.elastic.co/u/ave19)\ **Replies:** 3\ **Last updated:** [August 26, 2016, 4:10pm UTC](https://discuss.elastic.co/t/logstash-http-port-auto-increment/57370 "2016-08-26T16:10:00Z")
Hey! I have a case where I run two logstash instances on the same system. In v5.0.0-alpha4, when the second one starts, it complains that the port 9600 is already taken. How about trying 9601? :smiley: -ave --- ## [\[Solved\]Tcp input/output](https://discuss.elastic.co/t/solved-tcp-input-output/27411)
**Author:** [@DavidL](https://discuss.elastic.co/u/DavidL)\ **Replies:** 3\ **Last updated:** [August 26, 2016, 3:01pm UTC](https://discuss.elastic.co/t/solved-tcp-input-output/27411 "2016-08-26T15:01:04Z")
I have two logstash instances on two machines, trying to receive json from to the other, In the receiver I have: input { tcp { port =\> 9302 codec =\> "json" } } And in the sender I have outp… --- ## [Invalid Config File](https://discuss.elastic.co/t/invalid-config-file/58859)
**Author:** [@kmk3173](https://discuss.elastic.co/u/kmk3173)\ **Replies:** 31\ **Last updated:** [August 26, 2016, 2:52pm UTC](https://discuss.elastic.co/t/invalid-config-file/58859 "2016-08-26T14:52:54Z")
When I run --configtest I get the following error: Reason: Expected one of #, at line 45, column 4 after output { if \[type\] == "test2" { elasticsearch { hosts =\> "localhost:… --- ## [Twitter Logstash - undefined method \`filter' for nil:NilClass](https://discuss.elastic.co/t/twitter-logstash-undefined-method-filter-for-nil-nilclass/59045)
**Author:** [@Azlirn](https://discuss.elastic.co/u/Azlirn)\ **Replies:** 0\ **Last updated:** [August 26, 2016, 2:06pm UTC](https://discuss.elastic.co/t/twitter-logstash-undefined-method-filter-for-nil-nilclass/59045 "2016-08-26T14:06:29Z")
Hi all, I am working on utilizing the ELK stack to do some live Twitter research and I am running into some issues when utilizing large tracks. I know by Twitter's standards, the track size has a limit of 400 terms wi… --- ## [Why does logstash/elasticsearch re-put index mappings when I restart logstash?](https://discuss.elastic.co/t/why-does-logstash-elasticsearch-re-put-index-mappings-when-i-restart-logstash/58997)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 2\ **Last updated:** [August 26, 2016, 1:11pm UTC](https://discuss.elastic.co/t/why-does-logstash-elasticsearch-re-put-index-mappings-when-i-restart-logstash/58997 "2016-08-26T13:11:45Z")
Every time I restart logstash, It appears that it attempts to re-put a ton of index mappings. This turns out to be a problem for us, because we have a pretty large cluster, and so these calls take a little while to proce… --- ## [Log4j 2 json parsing with logstash 1.4.2](https://discuss.elastic.co/t/log4j-2-json-parsing-with-logstash-1-4-2/58906)
**Author:** [@madhawa1984](https://discuss.elastic.co/u/madhawa1984)\ **Replies:** 4\ **Last updated:** [August 26, 2016, 12:56pm UTC](https://discuss.elastic.co/t/log4j-2-json-parsing-with-logstash-1-4-2/58906 "2016-08-26T12:56:51Z")
I am using log4j 2 along with json pattern layout to create the logs and using the logstash agent 1.4.2 to port the logs in to a redis. In my logs i am trying to log some json payloads which includes some nested json. A… --- ## [Indexing documents to elasticsearch monthly?](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793)
**Author:** [@i5513](https://discuss.elastic.co/u/i5513)\ **Replies:** 4\ **Last updated:** [August 26, 2016, 12:35pm UTC](https://discuss.elastic.co/t/indexing-documents-to-elasticsearch-monthly/58793 "2016-08-26T12:35:25Z")
Hello, We have a 3 node ELK cluster with 83000000 documents and 53 gb Our cluster is populated by logstash who create a index by day We have this scheme since 2014 january Now elasticsearch is going slow. It have 484… --- ## [Log stash-jdbc input pool configuration](https://discuss.elastic.co/t/log-stash-jdbc-input-pool-configuration/58823)
**Author:** [@ggp](https://discuss.elastic.co/u/ggp)\ **Replies:** 1\ **Last updated:** [August 26, 2016, 9:38am UTC](https://discuss.elastic.co/t/log-stash-jdbc-input-pool-configuration/58823 "2016-08-26T09:38:24Z")
i have configured jdbc input to pull data from postgresql, is there any way we can configure connection pool here, or we need to run multiple log stash processes? --- ## [Metrics on Throttle Plugin](https://discuss.elastic.co/t/metrics-on-throttle-plugin/58841)
**Author:** [@devkmsg](https://discuss.elastic.co/u/devkmsg)\ **Replies:** 1\ **Last updated:** [August 26, 2016, 9:26am UTC](https://discuss.elastic.co/t/metrics-on-throttle-plugin/58841 "2016-08-26T09:26:38Z")
I'm interested in knowing how many times a particular key was throttled. I've been poking at this problem for the better part of a week now and the only thing I have been able to come up with is being able to determine … --- ## [Logstash Elasticsearch Output Plugin (Index)](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-index/59007)
**Author:** [@jamesl](https://discuss.elastic.co/u/jamesl)\ **Replies:** 1\ **Last updated:** [August 26, 2016, 7:35am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-index/59007 "2016-08-26T07:35:03Z")
Hi. In the following configuration I would like to name my index and document\_type to have the 'type' field automatically put in instead of having to manually write it out. Extract of output LS file: } else if \[type\] =… --- ## [Field marked as number, but index is having it as string](https://discuss.elastic.co/t/field-marked-as-number-but-index-is-having-it-as-string/58952)
**Author:** [@rmalired](https://discuss.elastic.co/u/rmalired)\ **Replies:** 2\ **Last updated:** [August 26, 2016, 3:55am UTC](https://discuss.elastic.co/t/field-marked-as-number-but-index-is-having-it-as-string/58952 "2016-08-26T03:55:35Z")
Hi, I running into an issue where my grok expression stays a particular field as NUMBER, but when the data is loaded into elastic search it is getting marked as string. below is the configuration grok Pattern WEBLOGIC… --- ## [Logstash not reading file when started as a service](https://discuss.elastic.co/t/logstash-not-reading-file-when-started-as-a-service/58877)
**Author:** [@obudiman](https://discuss.elastic.co/u/obudiman)\ **Replies:** 7\ **Last updated:** [August 25, 2016, 11:36pm UTC](https://discuss.elastic.co/t/logstash-not-reading-file-when-started-as-a-service/58877 "2016-08-25T23:36:15Z")
We have the following logstash configuration file which has been tested successfully using the configtest option, input { file { path =\> "/etc/httpd/logs/clickstream.log" } } filter { … --- ## [Longitude and latitude for logs](https://discuss.elastic.co/t/longitude-and-latitude-for-logs/58964)
**Author:** [@anoopmk007](https://discuss.elastic.co/u/anoopmk007)\ **Replies:** 5\ **Last updated:** [August 25, 2016, 11:28pm UTC](https://discuss.elastic.co/t/longitude-and-latitude-for-logs/58964 "2016-08-25T23:28:11Z")
hello , My logs doesn't have any option to longitude and latitude information. Currently these client logs for one location copied to the server at same location and from there logstash sending to elsastcisearch serve… --- ## [Logstash service restart : reading file from beginning](https://discuss.elastic.co/t/logstash-service-restart-reading-file-from-beginning/58974)
**Author:** [@anoopmk007](https://discuss.elastic.co/u/anoopmk007)\ **Replies:** 1\ **Last updated:** [August 25, 2016, 8:23pm UTC](https://discuss.elastic.co/t/logstash-service-restart-reading-file-from-beginning/58974 "2016-08-25T20:23:34Z")
Hello, I am facing some issues with logstash service restart. Every time when restart the service , then logstash start reading logs files from beginning. Any idea how to resolve this? logstash configuration:- … --- ## [Logstash adding extra comma to tags](https://discuss.elastic.co/t/logstash-adding-extra-comma-to-tags/58863)
**Author:** [@Josh\_Reichardt](https://discuss.elastic.co/u/Josh_Reichardt)\ **Replies:** 9\ **Last updated:** [August 25, 2016, 7:01pm UTC](https://discuss.elastic.co/t/logstash-adding-extra-comma-to-tags/58863 "2016-08-25T19:01:37Z")
I just started seeing this issue, I'm not quite sure what's up. I switched one of my environments to use the logstash+tcp functionality that recently got added to the logspout-logstash forwarder and noticed that an extra… --- ## [Adding field for string length](https://discuss.elastic.co/t/adding-field-for-string-length/58958)
**Author:** [@megamax](https://discuss.elastic.co/u/megamax)\ **Replies:** 2\ **Last updated:** [August 25, 2016, 6:26pm UTC](https://discuss.elastic.co/t/adding-field-for-string-length/58958 "2016-08-25T18:26:38Z")
Hello, Newbie here. I'm trying to add a field to hold the string length of another field in my input. I want to visualize the string lengths (to show average email message sizes by user for example). I have tried the… --- ## [Not able to create index after first index uisng imap input plugin](https://discuss.elastic.co/t/not-able-to-create-index-after-first-index-uisng-imap-input-plugin/58810)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 5\ **Last updated:** [August 25, 2016, 3:56pm UTC](https://discuss.elastic.co/t/not-able-to-create-index-after-first-index-uisng-imap-input-plugin/58810 "2016-08-25T15:56:38Z")
Hi All I created a index using imap plugin. It went through successfully first time Then i realized i should change some fields type using template and i deleted the index I re ran the logstash using same config file… --- ## [Infinite Loop Of Logs](https://discuss.elastic.co/t/infinite-loop-of-logs/58949)
**Author:** [@Nitz](https://discuss.elastic.co/u/Nitz)\ **Replies:** 0\ **Last updated:** [August 25, 2016, 3:00pm UTC](https://discuss.elastic.co/t/infinite-loop-of-logs/58949 "2016-08-25T15:00:01Z")
Hi, I'm using ELK stack, where Filebeat (v5.0.0-alpha3) collects my logs and sends them to Kafka (v8.2.1), Then Logstash (v2.1.1) consumes those logs, parses / filters them and outputting them to Elasticsearch (v2.1.1) … --- ## [Logstash docker with kafka input not consuming data](https://discuss.elastic.co/t/logstash-docker-with-kafka-input-not-consuming-data/58908)
**Author:** [@cesarpball](https://discuss.elastic.co/u/cesarpball)\ **Replies:** 4\ **Last updated:** [August 25, 2016, 2:40pm UTC](https://discuss.elastic.co/t/logstash-docker-with-kafka-input-not-consuming-data/58908 "2016-08-25T14:40:22Z")
Hello, I am having problems to make the Logstash docker working as consumer with a kafka input. I can see the tcp socket in the zookeeper node but I can't get any data in the consumer. When I look the consumer (offse… --- ## [S3 dynamic folder naming in yyyy/mm/dd](https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879)
**Author:** [@lrem006](https://discuss.elastic.co/u/lrem006)\ **Replies:** 1\ **Last updated:** [August 25, 2016, 2:19pm UTC](https://discuss.elastic.co/t/s3-dynamic-folder-naming-in-yyyy-mm-dd/58879 "2016-08-25T14:19:21Z")
Someone on another website proposed a mechanism for dynamically creating folders in the YYYY/mm/dd for the s3 output plugin. Effectively changing the function "write\_on\_bucket" to the following starting few lines. This… --- ## [Logstash config output regex condition on integer](https://discuss.elastic.co/t/logstash-config-output-regex-condition-on-integer/58909)
**Author:** [@metalcapone](https://discuss.elastic.co/u/metalcapone)\ **Replies:** 2\ **Last updated:** [August 25, 2016, 12:22pm UTC](https://discuss.elastic.co/t/logstash-config-output-regex-condition-on-integer/58909 "2016-08-25T12:22:22Z")
Hello all! I'm working on indexing IIS logs in ES using Logstash. Here is my config output section, where I try to seperate the http code response (status field) in two indices: output { if \[type\] == "mywebsite" … --- ## [Logstash TCP SSL Certificate Error](https://discuss.elastic.co/t/logstash-tcp-ssl-certificate-error/58832)
**Author:** [@jaykumar](https://discuss.elastic.co/u/jaykumar)\ **Replies:** 2\ **Last updated:** [August 25, 2016, 12:11pm UTC](https://discuss.elastic.co/t/logstash-tcp-ssl-certificate-error/58832 "2016-08-25T12:11:23Z")
I am trying to setup TCP SSL input, below is my configuration: input { tcp { port =\> 9000 host =\> "0.0.0.0" mode =\> "server" ssl\_enable =\> true … --- ## [Escaping return carriage in json message body](https://discuss.elastic.co/t/escaping-return-carriage-in-json-message-body/58717)
**Author:** [@kalim](https://discuss.elastic.co/u/kalim)\ **Replies:** 1\ **Last updated:** [August 25, 2016, 8:17am UTC](https://discuss.elastic.co/t/escaping-return-carriage-in-json-message-body/58717 "2016-08-25T08:17:53Z")
Hi, Am using logstash-input-azureeventhub plugin to get data from Azure event hub. Am getting json parse error on return carriages when trying to import data. Please see below: Message: {"app\_Name":"KalimTest","Valu… --- ## [UK Post Code to geo data in Logstash?](https://discuss.elastic.co/t/uk-post-code-to-geo-data-in-logstash/43210)
**Author:** [@LeeSyd](https://discuss.elastic.co/u/LeeSyd)\ **Replies:** 9\ **Last updated:** [August 25, 2016, 8:13am UTC](https://discuss.elastic.co/t/uk-post-code-to-geo-data-in-logstash/43210 "2016-08-25T08:13:08Z")
I'm looking for a way to add Geo location data to documents based on UK post code (no IP data, address data only). I've searched but have been unable to find any results and have to admit that I am not a developer so a… --- ## [Extract substring from log field and translate it via YAML file](https://discuss.elastic.co/t/extract-substring-from-log-field-and-translate-it-via-yaml-file/58897)
**Author:** [@fantaghost](https://discuss.elastic.co/u/fantaghost)\ **Replies:** 3\ **Last updated:** [August 25, 2016, 7:52am UTC](https://discuss.elastic.co/t/extract-substring-from-log-field-and-translate-it-via-yaml-file/58897 "2016-08-25T07:52:37Z")
Hi All, I need to use an external YAML file to translate a field based on substring: YAML File structure: "testa": "application aaa" "testb": "application bbb" "testc": "application ccc" LOG Field content… --- ## [Two different grok pattern in a single config file](https://discuss.elastic.co/t/two-different-grok-pattern-in-a-single-config-file/58892)
**Author:** [@harshita\_nolakha](https://discuss.elastic.co/u/harshita_nolakha)\ **Replies:** 2\ **Last updated:** [August 25, 2016, 7:25am UTC](https://discuss.elastic.co/t/two-different-grok-pattern-in-a-single-config-file/58892 "2016-08-25T07:25:40Z")
can i use the two different grok pattern in single config file? --- ## [Logstash 5 Alpha4 with Xpack getting error in log](https://discuss.elastic.co/t/logstash-5-alpha4-with-xpack-getting-error-in-log/55474)
**Author:** [@gaurav\_S\_javheri](https://discuss.elastic.co/u/gaurav_S_javheri)\ **Replies:** 6\ **Last updated:** [August 25, 2016, 2:26am UTC](https://discuss.elastic.co/t/logstash-5-alpha4-with-xpack-getting-error-in-log/55474 "2016-08-25T02:26:22Z")
elasticsearch is secured with xpack security and hooked with ldap which is working fine. Even user has admin right in role\_mapping. Below is configuration output { elasticsearch { hosts =\> \['localhost:9200'\] … --- ## [Io/console not supported when starting logstash 1.5.0 on Win 7](https://discuss.elastic.co/t/io-console-not-supported-when-starting-logstash-1-5-0-on-win-7/966)
**Author:** [@mobcdi](https://discuss.elastic.co/u/mobcdi)\ **Replies:** 8\ **Last updated:** [August 24, 2016, 9:25pm UTC](https://discuss.elastic.co/t/io-console-not-supported-when-starting-logstash-1-5-0-on-win-7/966 "2016-08-24T21:25:39Z")
I'm trying to get ELK setup on a Win 7 machine using Java 1.8 . I've got E and K and marvel running but I can't get logstash to startup. From a windows command console running with admin previlages on a folder on the d… --- ## [Combining Multiple Config Files](https://discuss.elastic.co/t/combining-multiple-config-files/58840)
**Author:** [@kmk3173](https://discuss.elastic.co/u/kmk3173)\ **Replies:** 7\ **Last updated:** [August 24, 2016, 9:11pm UTC](https://discuss.elastic.co/t/combining-multiple-config-files/58840 "2016-08-24T21:11:13Z")
Hi, How would I be able to combine multiple config files to push data to elasticsearch. I currently have 2 csv files that have different column names and data results. I want to be able to run multiple config files a… --- ## [Reduce duplicate or similar log entries](https://discuss.elastic.co/t/reduce-duplicate-or-similar-log-entries/29029)
**Author:** [@pdcleyn](https://discuss.elastic.co/u/pdcleyn)\ **Replies:** 3\ **Last updated:** [August 24, 2016, 8:56pm UTC](https://discuss.elastic.co/t/reduce-duplicate-or-similar-log-entries/29029 "2016-08-24T20:56:47Z")
I would like to setup a logstash configuation which acts similar to the rsyslog repeated message reduction. Basically, based on parsing I would like to ship to elasticsearch only one message stating that in the previous … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1109) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1111)