# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=182 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 183 --- ## [Parse Vsftp log (Filebeat + Logstash)](https://discuss.elastic.co/t/parse-vsftp-log-filebeat-logstash/288040)
**Author:** [@Hecha](https://discuss.elastic.co/u/Hecha)\ **Replies:** 2\ **Last updated:** [November 9, 2021, 8:21pm UTC](https://discuss.elastic.co/t/parse-vsftp-log-filebeat-logstash/288040 "2021-11-09T20:21:12Z")
Hello guys! I am trying to parse the logs of my FTP server (Vsftpd) with logstash but I am having trouble. The logs follow the following format Fri Oct 29 17:16:17 2021 \[pid 22947\] CONNECT: Client "::ffff:10.0.1.6" Fri… --- ## [Where would my syslogs be?](https://discuss.elastic.co/t/where-would-my-syslogs-be/288819)
**Author:** [@MKirby](https://discuss.elastic.co/u/MKirby)\ **Replies:** 0\ **Last updated:** [November 9, 2021, 8:19pm UTC](https://discuss.elastic.co/t/where-would-my-syslogs-be/288819 "2021-11-09T20:19:28Z")
Good Afternoon Elastic team. I have proof via packetbeat that my syslogs are flowing to my Logstash server. I can also netstat and confirm the ports are open and listening on the configured port. I have the following … --- ## [ParNew and ConcurrentMarkSweep seen in logs](https://discuss.elastic.co/t/parnew-and-concurrentmarksweep-seen-in-logs/288806)
**Author:** [@MKirby](https://discuss.elastic.co/u/MKirby)\ **Replies:** 4\ **Last updated:** [November 9, 2021, 7:37pm UTC](https://discuss.elastic.co/t/parnew-and-concurrentmarksweep-seen-in-logs/288806 "2021-11-09T19:37:22Z")
Good Afternoon everyone; I am seeing the following lines when I place my Logstash in Debug mode to collect logs. I have a feeling that it is what is stopping me from seeing any syslog that has been forwarded to the logs… --- ## [Get the first line of \[message\] field coming from winlogbeat](https://discuss.elastic.co/t/get-the-first-line-of-message-field-coming-from-winlogbeat/288768)
**Author:** [@yquirion](https://discuss.elastic.co/u/yquirion)\ **Replies:** 6\ **Last updated:** [November 9, 2021, 7:24pm UTC](https://discuss.elastic.co/t/get-the-first-line-of-message-field-coming-from-winlogbeat/288768 "2021-11-09T19:24:30Z")
Hello, I'm trying to get the first line of the message field because this is an important information to keep. The \[message\] line, when I set the output to file using rudydebug option looks like this: "message" =\> "A… --- ## [Logstashpipelines, Internal API server error, path\_info=\>"/\_node/stats"](https://discuss.elastic.co/t/logstashpipelines-internal-api-server-error-path-info-node-stats/288735)
**Author:** [@Melany](https://discuss.elastic.co/u/Melany)\ **Replies:** 1\ **Last updated:** [November 9, 2021, 6:03pm UTC](https://discuss.elastic.co/t/logstashpipelines-internal-api-server-error-path-info-node-stats/288735 "2021-11-09T18:03:42Z")
Hello, when I start my logstashpipelines service I am getting the following error: \[2021-11-09T10:05:16,268\]\[ERROR\]\[logstash.agent \] Internal API server error {:status=\>500, :request\_method=\>"GET", :path\_info=\>"/\_node/s… --- ## [How to replace all groups of number in a string using gsub](https://discuss.elastic.co/t/how-to-replace-all-groups-of-number-in-a-string-using-gsub/288761)
**Author:** [@gustavon](https://discuss.elastic.co/u/gustavon)\ **Replies:** 1\ **Last updated:** [November 9, 2021, 5:49pm UTC](https://discuss.elastic.co/t/how-to-replace-all-groups-of-number-in-a-string-using-gsub/288761 "2021-11-09T17:49:00Z")
Hi, I have an uri on nginx log like this: /xyz/attachment/thumbnail/61355754/90/90 and others, so I need to use gsub to replace all number to something else, /xyz/attachment/thumbnail/{}/{}/{} for example, I tried to us… --- ## [Grok Named capture group as metadata](https://discuss.elastic.co/t/grok-named-capture-group-as-metadata/288731)
**Author:** [@sirReeall](https://discuss.elastic.co/u/sirReeall)\ **Replies:** 1\ **Last updated:** [November 9, 2021, 5:47pm UTC](https://discuss.elastic.co/t/grok-named-capture-group-as-metadata/288731 "2021-11-09T17:47:57Z")
Hello, I was wondering if it is possible to set a named capture group as metadata rather then a field sent to Elastic? I'm parsing some data that has these permutations for the same field: 1m 125s 345ms 125s 345ms 345… --- ## [Alternatives to Multiline Codec for logging Java traces with TCP input?](https://discuss.elastic.co/t/alternatives-to-multiline-codec-for-logging-java-traces-with-tcp-input/288784)
**Author:** [@bennetth](https://discuss.elastic.co/u/bennetth)\ **Replies:** 0\ **Last updated:** [November 9, 2021, 3:43pm UTC](https://discuss.elastic.co/t/alternatives-to-multiline-codec-for-logging-java-traces-with-tcp-input/288784 "2021-11-09T15:43:08Z")
Hello all, I have an issue where I am trying to log Java traces for a TCP input. However, it appears that Logstash truncates the trace to just the first line of the trace. I have attempted to use the multiline codec for… --- ## [%{message} is always empty](https://discuss.elastic.co/t/message-is-always-empty/288765)
**Author:** [@Robert\_Neagu](https://discuss.elastic.co/u/Robert_Neagu)\ **Replies:** 1\ **Last updated:** [November 9, 2021, 2:32pm UTC](https://discuss.elastic.co/t/message-is-always-empty/288765 "2021-11-09T14:32:56Z")
I want to send an email notification if there are any errors inside the Logstash import process. For this I'm using the email plugin inside the output section output { email { to =\> "mail@gmail.com" from =\> "server@… --- ## [Missing Converter handling for full class name=org.jruby.ext.date.RubyDateTime](https://discuss.elastic.co/t/missing-converter-handling-for-full-class-name-org-jruby-ext-date-rubydatetime/288757)
**Author:** [@Miria\_Bernardes](https://discuss.elastic.co/u/Miria_Bernardes)\ **Replies:** 1\ **Last updated:** [November 9, 2021, 12:44pm UTC](https://discuss.elastic.co/t/missing-converter-handling-for-full-class-name-org-jruby-ext-date-rubydatetime/288757 "2021-11-09T12:44:22Z")
Hey guys, all right? I need help with an issue I've been experiencing when collecting data using jdbc\_streaming, running conf locally through docker the data collection by logstash is performed successfully, but when ap… --- ## [Alternative to ingest-convert.sh for YML](https://discuss.elastic.co/t/alternative-to-ingest-convert-sh-for-yml/288597)
**Author:** [@IsaacKr](https://discuss.elastic.co/u/IsaacKr)\ **Replies:** 4\ **Last updated:** [November 9, 2021, 11:42am UTC](https://discuss.elastic.co/t/alternative-to-ingest-convert-sh-for-yml/288597 "2021-11-09T11:42:27Z")
We are using Logstash to ship logs to a cloud based SIEM. We'd like to utilize Filebeat to assist in the parsing, however it seems that Filebeat only provides the ingest pipelines for Elasticsearch. The ingest-convert.sh… --- ## [Data are processing slowly from the Logstash persistent queue to ES](https://discuss.elastic.co/t/data-are-processing-slowly-from-the-logstash-persistent-queue-to-es/288741)
**Author:** [@jenifer\_raja](https://discuss.elastic.co/u/jenifer_raja)\ **Replies:** 0\ **Last updated:** [November 9, 2021, 10:14am UTC](https://discuss.elastic.co/t/data-are-processing-slowly-from-the-logstash-persistent-queue-to-es/288741 "2021-11-09T10:14:22Z")
We are facing an issue with Logstash parsing, Initially parsing speed is very high and then it decreases gradually, because of this we have around 10 gb of data in queue. We tried increasing swap size, increasing workers… --- ## [Heartbeat input plugin](https://discuss.elastic.co/t/heartbeat-input-plugin/288621)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 2\ **Last updated:** [November 9, 2021, 9:50am UTC](https://discuss.elastic.co/t/heartbeat-input-plugin/288621 "2021-11-09T09:50:40Z")
Hiya all, if i want to change the input plugins listen port for heartbeat how can i do this? the guide doesn't list it as an option input { heartbeat { port =\> 2600 } } generates this error \[ERROR\]\[logstash.… --- ## [Elasticsearch filter in logstash = random results](https://discuss.elastic.co/t/elasticsearch-filter-in-logstash-random-results/288718)
**Author:** [@Rnx](https://discuss.elastic.co/u/Rnx)\ **Replies:** 0\ **Last updated:** [November 9, 2021, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-filter-in-logstash-random-results/288718 "2021-11-09T07:52:23Z")
Hi, I'm using Elasticsearch filter plugin in logstash to enrich records and calculate time difference between two events from different logs. It works, but randomly - it returns search results for some records, but not … --- ## [How to manage large messages with a few memory?](https://discuss.elastic.co/t/how-to-manage-large-messages-with-a-few-memory/288677)
**Author:** [@Jesus\_Alberto\_Carril](https://discuss.elastic.co/u/Jesus_Alberto_Carril)\ **Replies:** 4\ **Last updated:** [November 9, 2021, 3:55am UTC](https://discuss.elastic.co/t/how-to-manage-large-messages-with-a-few-memory/288677 "2021-11-09T03:55:17Z")
I have a problem. My JVM logstash only has 512M in it configurations: -Xms256m -Xmx512m Logstash read messages from mylog.json and send them to my broker. BUT when a super large message is written in my mylog.json the… --- ## [Daylight saving time impacting only part of the environment](https://discuss.elastic.co/t/daylight-saving-time-impacting-only-part-of-the-environment/288622)
**Author:** [@phdsantos](https://discuss.elastic.co/u/phdsantos)\ **Replies:** 5\ **Last updated:** [November 8, 2021, 9:03pm UTC](https://discuss.elastic.co/t/daylight-saving-time-impacting-only-part-of-the-environment/288622 "2021-11-08T21:03:14Z")
Hello everyone, we are facing a strange behavior in ELK environment related to daylight saving time; the details are below: Platform: kibana 6.7.2 Logstash 5.3.0 Elasticsearch 6.7.2 Scenario: We have Weblogic domai… --- ## [Multiple data sources over same port](https://discuss.elastic.co/t/multiple-data-sources-over-same-port/288598)
**Author:** [@IsaacKr](https://discuss.elastic.co/u/IsaacKr)\ **Replies:** 2\ **Last updated:** [November 8, 2021, 5:05pm UTC](https://discuss.elastic.co/t/multiple-data-sources-over-same-port/288598 "2021-11-08T17:05:31Z")
We are ingesting data from multiple data sources over port 514 and using Logstash to ship them out to multiple destinations. Is there a way to tag ingested data based on source IP so that it can be forwarded to the corre… --- ## [Jdbc\_streming and prepared statement](https://discuss.elastic.co/t/jdbc-streming-and-prepared-statement/288647)
**Author:** [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\ **Replies:** 0\ **Last updated:** [November 8, 2021, 2:56pm UTC](https://discuss.elastic.co/t/jdbc-streming-and-prepared-statement/288647 "2021-11-08T14:56:21Z")
Hi , I'm trying to use a prepared statement in jdbc\_streaming filter, I setuped the prepared statement on mySql instance using this commands: PREPARE lookup\_persone FROM 'SELECT persone.id, persone.nome AS nome\_person… --- ## [Logstash TCP Input with SSL failing with non descript error](https://discuss.elastic.co/t/logstash-tcp-input-with-ssl-failing-with-non-descript-error/288312)
**Author:** [@AlanMark](https://discuss.elastic.co/u/AlanMark)\ **Replies:** 14\ **Last updated:** [November 8, 2021, 8:23am UTC](https://discuss.elastic.co/t/logstash-tcp-input-with-ssl-failing-with-non-descript-error/288312 "2021-11-08T08:23:20Z")
I'm trying to set up a TCP Input with an SSL certificate, but no matter how I configure it, i keep getting a non descript error. This is running on Logstash 7.9.1 OSS. My config is the following: input { tcp {… --- ## [Install Logstash 7.13](https://discuss.elastic.co/t/install-logstash-7-13/288358)
**Author:** [@Nikparab](https://discuss.elastic.co/u/Nikparab)\ **Replies:** 4\ **Last updated:** [November 8, 2021, 6:14am UTC](https://discuss.elastic.co/t/install-logstash-7-13/288358 "2021-11-08T06:14:15Z")
I have aws opensearch cluster of version 1.0 I want to send the logs to this cluster using logstash. But when we install the logstash it install the latest version of logstash and which is not supported by opensearch … --- ## [Bug on logstash input for gz files](https://discuss.elastic.co/t/bug-on-logstash-input-for-gz-files/288547)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 0\ **Last updated:** [November 6, 2021, 7:05pm UTC](https://discuss.elastic.co/t/bug-on-logstash-input-for-gz-files/288547 "2021-11-06T19:05:42Z")
Hi I've observed that logstash with input plugin doesn't proceed in the correct way gz files in sincedb. I saw that logstash doesn't put all logs info to sincedb but only the header without names. If I have restarted lo… --- ## [How to make logstash to automatically detect new pipleines configuration and execute them?](https://discuss.elastic.co/t/how-to-make-logstash-to-automatically-detect-new-pipleines-configuration-and-execute-them/288541)
**Author:** [@Husnain](https://discuss.elastic.co/u/Husnain)\ **Replies:** 4\ **Last updated:** [November 6, 2021, 7:06pm UTC](https://discuss.elastic.co/t/how-to-make-logstash-to-automatically-detect-new-pipleines-configuration-and-execute-them/288541 "2021-11-06T19:06:47Z")
Hii.I am using logstash to ingest data from four different data sources from i.e Elasticsearch,mongodb,textfile and mysql.My requirement is,that I will specify a directory path and logstash will read pipeline configurati… --- ## [How to send logs from different filebeat to logstash](https://discuss.elastic.co/t/how-to-send-logs-from-different-filebeat-to-logstash/288305)
**Author:** [@root\_linux](https://discuss.elastic.co/u/root_linux)\ **Replies:** 2\ **Last updated:** [November 6, 2021, 2:27pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-different-filebeat-to-logstash/288305 "2021-11-06T14:27:05Z")
I have configured Elasticsearch, kibana and logstash on same server. I want to send logs from multiple filebeat clients to logstash with different index name. How can I configure it? --- ## [JDBC Input plugin installation](https://discuss.elastic.co/t/jdbc-input-plugin-installation/288510)
**Author:** [@SANDRA\_ROY\_ARICATT](https://discuss.elastic.co/u/SANDRA_ROY_ARICATT)\ **Replies:** 1\ **Last updated:** [November 5, 2021, 7:00pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-installation/288510 "2021-11-05T19:00:06Z")
I have the new Mac M1 Silicon. I need to get data from MySQL workbench into Elasticsearch. I came to know that it can be achieved using Logstash, which in turn uses JDBC input plugin for getting data from database into… --- ## [Dealing with invalid json](https://discuss.elastic.co/t/dealing-with-invalid-json/288179)
**Author:** [@caseydm](https://discuss.elastic.co/u/caseydm)\ **Replies:** 6\ **Last updated:** [November 5, 2021, 6:01pm UTC](https://discuss.elastic.co/t/dealing-with-invalid-json/288179 "2021-11-05T18:01:32Z")
I'm trying to import some json data that is in a column in redshift. Some of the records have a field that includes invalid json escape sequences, such as this: "work\_title": "The Discrete and Semi-continuous Fr\\'echet … --- ## [Logstash group time series by decade](https://discuss.elastic.co/t/logstash-group-time-series-by-decade/288462)
**Author:** [@caseydm](https://discuss.elastic.co/u/caseydm)\ **Replies:** 4\ **Last updated:** [November 5, 2021, 5:48pm UTC](https://discuss.elastic.co/t/logstash-group-time-series-by-decade/288462 "2021-11-05T17:48:03Z")
I am ingesting academic articles and splitting the index by year: output { elasticsearch { hosts =\> \["${ES\_HOST\_PROD}"\] index =\> "works-%{year}" user =\> "${ES\_USER\_PROD}" password =\> … --- ## [Logstash is reloading the same documents again and again from elasticsearch index](https://discuss.elastic.co/t/logstash-is-reloading-the-same-documents-again-and-again-from-elasticsearch-index/288315)
**Author:** [@Husnain](https://discuss.elastic.co/u/Husnain)\ **Replies:** 5\ **Last updated:** [November 5, 2021, 1:47pm UTC](https://discuss.elastic.co/t/logstash-is-reloading-the-same-documents-again-and-again-from-elasticsearch-index/288315 "2021-11-05T13:47:20Z")
Hii.I am pulling documents from one Elasticsearch index into another Elasticsearch index using logstash.I am running logstash as a service by configuring "logstash.yml" file.The index from which I am pulling data,that in… --- ## [How to aggregate multiple events coming from different logs with slight different timestamp, when the only field is timestamp to combine those?](https://discuss.elastic.co/t/how-to-aggregate-multiple-events-coming-from-different-logs-with-slight-different-timestamp-when-the-only-field-is-timestamp-to-combine-those/287848)
**Author:** [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)\ **Replies:** 10\ **Last updated:** [November 5, 2021, 1:05pm UTC](https://discuss.elastic.co/t/how-to-aggregate-multiple-events-coming-from-different-logs-with-slight-different-timestamp-when-the-only-field-is-timestamp-to-combine-those/287848 "2021-11-05T13:05:14Z")
Hi, I have this situation: "How to aggregate multiple events coming from different logs with slight different timestamp, when the only common field could be the timesamp? I need to combine all data points into one dat… --- ## [Logstash output creating error](https://discuss.elastic.co/t/logstash-output-creating-error/288161)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 25\ **Last updated:** [November 5, 2021, 9:39am UTC](https://discuss.elastic.co/t/logstash-output-creating-error/288161 "2021-11-05T09:39:45Z")
Hi all, I have a weird error i can't nail down. Running 7.15.1 ES / Logstash I have a simple Logstash conf thats listening on a port for beats data and it should just push it to an index. input { beats { p… --- ## [Need to configure message timestamp as timestamp in elastic stack](https://discuss.elastic.co/t/need-to-configure-message-timestamp-as-timestamp-in-elastic-stack/288191)
**Author:** [@Mobin](https://discuss.elastic.co/u/Mobin)\ **Replies:** 6\ **Last updated:** [November 5, 2021, 5:06am UTC](https://discuss.elastic.co/t/need-to-configure-message-timestamp-as-timestamp-in-elastic-stack/288191 "2021-11-05T05:06:26Z")
We have created an elastic stack to monitor logs from the Linux clients. On Linux clients, filebeat is configured to ship the logs to the elastic stack. When we are searching for the logs using elastic discover option @… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=181) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=183)