# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=471 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 472 --- ## [Unable to bind 5000 port for logstash](https://discuss.elastic.co/t/unable-to-bind-5000-port-for-logstash/201184)
**Author:** [@jacobmathews](https://discuss.elastic.co/u/jacobmathews)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 8:32am UTC](https://discuss.elastic.co/t/unable-to-bind-5000-port-for-logstash/201184 "2019-09-26T08:32:12Z")
hi...I am not able to connect to my localhost 5000 port in which I have configured by logstash…I am attaching a screenshot of my "filebeat test output",logstash input conf and filebeat.yml configurations.kindly help me w… --- ## [Logstash aggregate maps to be used in a cluster](https://discuss.elastic.co/t/logstash-aggregate-maps-to-be-used-in-a-cluster/201183)
**Author:** [@Gokul\_Raj](https://discuss.elastic.co/u/Gokul_Raj)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 8:29am UTC](https://discuss.elastic.co/t/logstash-aggregate-maps-to-be-used-in-a-cluster/201183 "2019-09-26T08:29:11Z")
I'm trying to know if logstash aggregate maps can be used in a cluster where there are 3 nodes of logstash with multiple pipelines . Read that aggregate maps are shared at pipeline level, will it work if I specify a comm… --- ## [Is any exception handling mechanism in Logstash when dealing with multiple Elasticsearch in the output section to overcome the Elasticsearch failures?](https://discuss.elastic.co/t/is-any-exception-handling-mechanism-in-logstash-when-dealing-with-multiple-elasticsearch-in-the-output-section-to-overcome-the-elasticsearch-failures/201145)
**Author:** [@geethusureshkrishna](https://discuss.elastic.co/u/geethusureshkrishna)\ **Replies:** 3\ **Last updated:** [September 26, 2019, 7:14am UTC](https://discuss.elastic.co/t/is-any-exception-handling-mechanism-in-logstash-when-dealing-with-multiple-elasticsearch-in-the-output-section-to-overcome-the-elasticsearch-failures/201145 "2019-09-26T07:14:11Z")
We are trying to write data parallelly to 2 different clusters let us say, cluster A and cluster B from Logstash. In the sample Logstash configuration is given below. input { elasticsearch{ hosts… --- ## [\[ES7.0.1\] Field 'cn'(keyword&text type) is not extracted as expected from message field](https://discuss.elastic.co/t/es7-0-1-field-cn-keyword-text-type-is-not-extracted-as-expected-from-message-field/199560)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 6\ **Last updated:** [September 26, 2019, 5:42am UTC](https://discuss.elastic.co/t/es7-0-1-field-cn-keyword-text-type-is-not-extracted-as-expected-from-message-field/199560 "2019-09-26T05:42:00Z")
Hi Team, in prod elk, there's below message with containing the value of cn field is GRIFOLS, S.A. 2019-09-11 17:29:30,930 module=SCM fa=TS at=SCM.TS.LIST\_SAVED\_SEARCH si=566F57EC24877AC67B8D36655D69D490.vsa3029283 ci… --- ## [\[logstash\]what does "\\1" mean in gsub?](https://discuss.elastic.co/t/logstash-what-does-1-mean-in-gsub/199946)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 11\ **Last updated:** [September 26, 2019, 5:41am UTC](https://discuss.elastic.co/t/logstash-what-does-1-mean-in-gsub/199946 "2019-09-26T05:41:40Z")
Hi team, what does "\\1" mean in below configuration and how to use it? I can;t find the official documentation which can explain it. mutate { gsub =\> \["message","(?\<=SERVICEPERFDATA::)procs=(\\d+);\\S+", "\\1"\] } Is be… --- ## [Input via Log4J SocketAppender fails due to charachter problems](https://discuss.elastic.co/t/input-via-log4j-socketappender-fails-due-to-charachter-problems/201136)
**Author:** [@Dilan\_Sachintha](https://discuss.elastic.co/u/Dilan_Sachintha)\ **Replies:** 0\ **Last updated:** [September 26, 2019, 4:46am UTC](https://discuss.elastic.co/t/input-via-log4j-socketappender-fails-due-to-charachter-problems/201136 "2019-09-26T04:46:29Z")
I'm using Log4J v2.x SocketAppender to send logs to logstash directly. The system was previously designed to send the logs via filebeat. Since I changed the setting, logstash now cannot read my inputs. \[2019-09-26T10:06… --- ## [Logstash JDBC query for LIKE operator](https://discuss.elastic.co/t/logstash-jdbc-query-for-like-operator/200949)
**Author:** [@Donny\_Nightfrost](https://discuss.elastic.co/u/Donny_Nightfrost)\ **Replies:** 1\ **Last updated:** [September 26, 2019, 2:49am UTC](https://discuss.elastic.co/t/logstash-jdbc-query-for-like-operator/200949 "2019-09-26T02:49:13Z")
Hi guys, I have a jdbc static query like this: local\_lookups =\> \[ { query =\> "select \* from servers WHERE ip LIKE = '%:ip%'" parameters =\> {ip =\> "\[from\_ip\]"} target =\> "server" } … --- ## [How can I remove the object in nested array by logstash?](https://discuss.elastic.co/t/how-can-i-remove-the-object-in-nested-array-by-logstash/201006)
**Author:** [@living](https://discuss.elastic.co/u/living)\ **Replies:** 5\ **Last updated:** [September 26, 2019, 2:24am UTC](https://discuss.elastic.co/t/how-can-i-remove-the-object-in-nested-array-by-logstash/201006 "2019-09-26T02:24:46Z")
Hi, The log output like this: { "@timestamp" =\> 2019-09-19T17:30:48.984Z, "characterid" =\> "20036815", "inventory\_item" =\> \[ \[ 0\] { "item\_equipped" =\> true, … --- ## [Netflow in Logstash does not find the logstash.yml file](https://discuss.elastic.co/t/netflow-in-logstash-does-not-find-the-logstash-yml-file/199115)
**Author:** [@eemtzc](https://discuss.elastic.co/u/eemtzc)\ **Replies:** 24\ **Last updated:** [September 26, 2019, 2:21am UTC](https://discuss.elastic.co/t/netflow-in-logstash-does-not-find-the-logstash-yml-file/199115 "2019-09-26T02:21:37Z")
Hello everyone, I am trying to use Netflow on my cluster, so I configured the logstash.yml file as describe on the articules as follows, this directory is located on /etc/logstash: modules: -name: netflow var.input.u… --- ## [Unable to connect to Logstash from Remote](https://discuss.elastic.co/t/unable-to-connect-to-logstash-from-remote/201100)
**Author:** [@James\_Nobles](https://discuss.elastic.co/u/James_Nobles)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 8:10pm UTC](https://discuss.elastic.co/t/unable-to-connect-to-logstash-from-remote/201100 "2019-09-25T20:10:23Z")
I've have installed ELK 6.8.3 on CENTOS 6.10. Elasticsearch and Kibana appear to work on so far. I can upload csv files in Kibana and create visualizations. This issue is sending syslog data to logstash. The goal is to … --- ## [Replace LAT and LON values](https://discuss.elastic.co/t/replace-lat-and-lon-values/201089)
**Author:** [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Replies:** 4\ **Last updated:** [September 25, 2019, 7:52pm UTC](https://discuss.elastic.co/t/replace-lat-and-lon-values/201089 "2019-09-25T19:52:02Z")
my goal is to not plot the "dot" down to spcific state or city, just country. Sample data: "geoip" =\> { "latitude" =\> 44.6628, "location" =\> { "lon" =\> -93.1539, "lat" =\> 44.6628 … --- ## [Logstash Service does not start after running the Netflow module](https://discuss.elastic.co/t/logstash-service-does-not-start-after-running-the-netflow-module/201097)
**Author:** [@eemtzc](https://discuss.elastic.co/u/eemtzc)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 7:46pm UTC](https://discuss.elastic.co/t/logstash-service-does-not-start-after-running-the-netflow-module/201097 "2019-09-25T19:46:01Z")
Ok, here is what happened. Originally we had a pipeline to receive logs from our firewalls devices to our ELK stack, and everything was working perfect. Then we will like to add, to the same logstash server, the Netflo… --- ## [Message: Missed Logs Details: 10000](https://discuss.elastic.co/t/message-missed-logs-details-10000/201086)
**Author:** [@sbarry](https://discuss.elastic.co/u/sbarry)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 4:38pm UTC](https://discuss.elastic.co/t/message-missed-logs-details-10000/201086 "2019-09-25T16:38:15Z")
We are missing a lot of logs coming from cloud foundry into kibana, and on the ingestor vms we get this error message Message: Missed Logs Details: 10000 Tried scaling the ingestor vms but still missing logs. Any idea… --- ## [Merge information from separated databases into a single document](https://discuss.elastic.co/t/merge-information-from-separated-databases-into-a-single-document/201078)
**Author:** [@ricardo.almeida](https://discuss.elastic.co/u/ricardo.almeida)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 3:45pm UTC](https://discuss.elastic.co/t/merge-information-from-separated-databases-into-a-single-document/201078 "2019-09-25T15:45:54Z")
I would like to merge information from two different tables in different database. In table t\_a from database db\_a I have information about the users, like username, Name, Phone Number, etc. In table t\_b from database … --- ## [Transform datetime to @timestamp](https://discuss.elastic.co/t/transform-datetime-to-timestamp/201062)
**Author:** [@fernrguez](https://discuss.elastic.co/u/fernrguez)\ **Replies:** 4\ **Last updated:** [September 25, 2019, 3:22pm UTC](https://discuss.elastic.co/t/transform-datetime-to-timestamp/201062 "2019-09-25T15:22:00Z")
I'm trying to use a datetime field (startTime) from the input xml logs and use it as @timestamp but something is not properly set as it idenfies the field as string but is not able to apply the transformation leaving the… --- ## [Logstash and Postgres Error](https://discuss.elastic.co/t/logstash-and-postgres-error/201074)
**Author:** [@emy.comin](https://discuss.elastic.co/u/emy.comin)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 2:54pm UTC](https://discuss.elastic.co/t/logstash-and-postgres-error/201074 "2019-09-25T14:54:09Z")
Hey everyone, I had several problems with the configuration of logstash, jumping from post to post, but the error I had now I do not have much information and now began to move cok ELK, I have little knowledge ... Could … --- ## [Logstash unacceptable code point error](https://discuss.elastic.co/t/logstash-unacceptable-code-point-error/201073)
**Author:** [@mrstone](https://discuss.elastic.co/u/mrstone)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 2:50pm UTC](https://discuss.elastic.co/t/logstash-unacceptable-code-point-error/201073 "2019-09-25T14:50:55Z")
Hi We had a period, when number of our Logstash scripts refused to run. The error reported in the log file was as follows: Pipeline aborted due to error {:pipeline\_id=\>"xxxxx", :exception=\>#\ **Author:** [@wmassingham](https://discuss.elastic.co/u/wmassingham)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 2:36pm UTC](https://discuss.elastic.co/t/how-do-i-run-netflow-setup-when-using-elasticsearch-as-config-store-in-azure/201070 "2019-09-25T14:36:26Z") I'm trying to get netflow set up for the SIEM tools in my Azure instance. I have data coming into ES through a logstash pipeline (summarized here): input { udp { port =\> 2055 codec =\> netflow type =\> netf… --- ## [Limit on number of convert statements?](https://discuss.elastic.co/t/limit-on-number-of-convert-statements/201060)
**Author:** [@gyterpena](https://discuss.elastic.co/u/gyterpena)\ **Replies:** 1\ **Last updated:** [September 25, 2019, 2:33pm UTC](https://discuss.elastic.co/t/limit-on-number-of-convert-statements/201060 "2019-09-25T14:33:51Z")
When I use mutate convert with 45 different convert statements logstash fails to start. When I split it into two different convert groups it works fine. Below is error message when it fails. Seems like there is limit on… --- ## [Google\_pubsub plugin via Internet Proxy](https://discuss.elastic.co/t/google-pubsub-plugin-via-internet-proxy/201054)
**Author:** [@Thanawan\_Sachdev](https://discuss.elastic.co/u/Thanawan_Sachdev)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 2:16pm UTC](https://discuss.elastic.co/t/google-pubsub-plugin-via-internet-proxy/201054 "2019-09-25T14:16:12Z")
Hi, I have logstash running inside closed network. We need to connect to internet via internet proxy. We want to use google\_pubsub plugin. We have already set environment variable HTTP\_PROXY but it doesn't work. Any hel… --- ## [Clarification needed on File filter](https://discuss.elastic.co/t/clarification-needed-on-file-filter/200801)
**Author:** [@SathishPrakasam](https://discuss.elastic.co/u/SathishPrakasam)\ **Replies:** 1\ **Last updated:** [September 25, 2019, 1:55pm UTC](https://discuss.elastic.co/t/clarification-needed-on-file-filter/200801 "2019-09-25T13:55:02Z")
Hi, We are planning to use logstash in our project. I have 200 different files to be processed, and the files are updated every 15 minutes. My question is how many file filters can be used in input plugin? input { … --- ## [Logstash not able to connect to elasticsearch docker host](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-elasticsearch-docker-host/200866)
**Author:** [@akapit](https://discuss.elastic.co/u/akapit)\ **Replies:** 3\ **Last updated:** [September 25, 2019, 1:47pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-elasticsearch-docker-host/200866 "2019-09-25T13:47:34Z")
Hi I have deployed ELK in a Docker swarm. Elasticsearch and kibana are in the same node, when logstash is in another one. The issue is as follows, logstash is not able to connect to elasticsearch using the docker host… --- ## [Logstsh output Kafka AUTHENTICATION\_FAILED](https://discuss.elastic.co/t/logstsh-output-kafka-authentication-failed/201026)
**Author:** [@Darshan\_Parab](https://discuss.elastic.co/u/Darshan_Parab)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 12:15pm UTC](https://discuss.elastic.co/t/logstsh-output-kafka-authentication-failed/201026 "2019-09-25T12:15:28Z")
Hi All, I'm trying to send logs to Kerberized kafka from logstash. After staring a pipeline I'm getting below mentioned message which. i assume, tells me that Kafka Producer on logstash is able to communicate and authen… --- ## [Logstash to Elasticsearch - data update](https://discuss.elastic.co/t/logstash-to-elasticsearch-data-update/200987)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 0\ **Last updated:** [September 25, 2019, 7:53am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-data-update/200987 "2019-09-25T07:53:17Z")
I am trying to update an existing data in my Elasticsearch Index. Im using the below Logstash configuration: input { jdbc { jdbc\_connection\_string =\> "jdbc:sqlserver://DB01:1433;databasename=mydb;integratedS… --- ## [\[Resolved\]\[Kibana7.0.1\] Error occurs " Request Timeout after 30000ms" when starting kibana](https://discuss.elastic.co/t/resolved-kibana7-0-1-error-occurs-request-timeout-after-30000ms-when-starting-kibana/200968)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 3\ **Last updated:** [September 25, 2019, 10:38am UTC](https://discuss.elastic.co/t/resolved-kibana7-0-1-error-occurs-request-timeout-after-30000ms-when-starting-kibana/200968 "2019-09-25T10:38:29Z")
Hi Team, I was running into below error when starting kibana, and ES side has no error and exception. log \[06:49:29.171\] \[info\]\[status\]\[plugin:spaces@7.0.1\] Status changed from yellow to green - Ready error \[06:4… --- ## [Ruby exception occurred: invalid date when passing date from logstash to ruby](https://discuss.elastic.co/t/ruby-exception-occurred-invalid-date-when-passing-date-from-logstash-to-ruby/200689)
**Author:** [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Replies:** 1\ **Last updated:** [September 25, 2019, 2:27am UTC](https://discuss.elastic.co/t/ruby-exception-occurred-invalid-date-when-passing-date-from-logstash-to-ruby/200689 "2019-09-25T02:27:53Z")
Hi All, Have a sample set of data as below orders type createdon dispatchedon deliveredon C015 2 31/07/2019 18:09 31/07/2019 18:15 11/09/2019 16:28 C018 3 06/08/2019 11:46 11/09/2019 12:24 11/09/2019 12:26 C019… --- ## [Convert date to 2019-09-11T15:47:00Z from 11/09/2019 15:47](https://discuss.elastic.co/t/convert-date-to-2019-09-11t1500z-from-11-09-2019-15-47/200826)
**Author:** [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Replies:** 5\ **Last updated:** [September 25, 2019, 2:25am UTC](https://discuss.elastic.co/t/convert-date-to-2019-09-11t1500z-from-11-09-2019-15-47/200826 "2019-09-25T02:25:46Z")
Hi all can someone help me how to convert the column value in csv from the date 11/09/2019 15:47 to 2019-09-11T15:47:00Z format Thanks --- ## [Grok & firewall logs](https://discuss.elastic.co/t/grok-firewall-logs/200749)
**Author:** [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Replies:** 6\ **Last updated:** [September 24, 2019, 8:42pm UTC](https://discuss.elastic.co/t/grok-firewall-logs/200749 "2019-09-24T20:42:12Z")
Hi, I have some firewall logs sent to "syslog" and i have filebeat read the syslog, send it to Logstash then to Elasticsearch. with this filter(1): grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp}" } } … --- ## [Load avro export from BigQuery](https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 1\ **Last updated:** [September 24, 2019, 7:10pm UTC](https://discuss.elastic.co/t/load-avro-export-from-bigquery/200918 "2019-09-24T19:10:05Z")
I would like to load avro export from BigQuery. Ideally without making any changes to that file. File has the following content: Objavro.codecnullavro.schemaØ {"type":"record","name":"Root","fields":\[{"name":"id","type… --- ## [Zeek Filebeat Module to Logstash JSON](https://discuss.elastic.co/t/zeek-filebeat-module-to-logstash-json/200539)
**Author:** [@millap](https://discuss.elastic.co/u/millap)\ **Replies:** 1\ **Last updated:** [September 24, 2019, 6:43pm UTC](https://discuss.elastic.co/t/zeek-filebeat-module-to-logstash-json/200539 "2019-09-24T18:43:09Z")
Hi All, I wonder if anyone can offer any assistance. We've recently moved to using the Zeek Filebeat module on some remote sensors, as these integrate nicely with the SIEM feature of Kibana, however, the old function we… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=470) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=472)