# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=423 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 424 --- ## [Logstash issue with dropping rows with a condition check on empty elements](https://discuss.elastic.co/t/logstash-issue-with-dropping-rows-with-a-condition-check-on-empty-elements/214132)
**Author:** [@Nick11](https://discuss.elastic.co/u/Nick11)\ **Replies:** 7\ **Last updated:** [January 8, 2020, 2:51pm UTC](https://discuss.elastic.co/t/logstash-issue-with-dropping-rows-with-a-condition-check-on-empty-elements/214132 "2020-01-08T14:51:51Z")
I have a CSV file with the following 2 rows (sample) reservation date, reservationID Jan 6th, res:id:adbcj-oksok-gjkk Jan 10th, Mar 10th, res:id:kkbcj-oksok-gjkk My ask is to drop empty rows and apply a grok filter on … --- ## [/usr/share/logstash/bin/system-install: line 88: #: command not found](https://discuss.elastic.co/t/usr-share-logstash-bin-system-install-line-88-command-not-found/214242)
**Author:** [@leprovokateur](https://discuss.elastic.co/u/leprovokateur)\ **Replies:** 1\ **Last updated:** [January 8, 2020, 2:48pm UTC](https://discuss.elastic.co/t/usr-share-logstash-bin-system-install-line-88-command-not-found/214242 "2020-01-08T14:48:05Z")
Hi, On upgrading from 7.5.0 to 7.5.1 I receive the error: Using provided startup.options file: /etc/logstash/startup.options OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 an… --- ## [Syslog input configuration for Logstash](https://discuss.elastic.co/t/syslog-input-configuration-for-logstash/211510)
**Author:** [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\ **Replies:** 10\ **Last updated:** [January 8, 2020, 2:46pm UTC](https://discuss.elastic.co/t/syslog-input-configuration-for-logstash/211510 "2020-01-08T14:46:23Z")
I want to send syslog files to my logstash, but not sure how to incorporate it with my beats input. When I have tried adding my beats stop working. Below is my logstash file. input { beats { port =\> 5044 } } … --- ## [Why stdout output shows message =\> "?"](https://discuss.elastic.co/t/why-stdout-output-shows-message/214079)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 10\ **Last updated:** [January 8, 2020, 2:35pm UTC](https://discuss.elastic.co/t/why-stdout-output-shows-message/214079 "2020-01-08T14:35:04Z")
Hello all, i am very new to logstash. I am just testing some filters to extract content from txt files without any structure. Why is this happening? On Windows enviroment logstash config file input { file { p… --- ## [Logstash goes down || Solution](https://discuss.elastic.co/t/logstash-goes-down-solution/213887)
**Author:** [@rkcharlie](https://discuss.elastic.co/u/rkcharlie)\ **Replies:** 1\ **Last updated:** [January 8, 2020, 1:44pm UTC](https://discuss.elastic.co/t/logstash-goes-down-solution/213887 "2020-01-08T13:44:28Z")
Hello Team, I am trying to push data from oracle to ES and I am able to do that as well. My problem when I am pushing those data and that time my ES goes down then I am not able to figure out that from where I need to p… --- ## [Delete documents after sync](https://discuss.elastic.co/t/delete-documents-after-sync/214236)
**Author:** [@tushar\_chevulkar](https://discuss.elastic.co/u/tushar_chevulkar)\ **Replies:** 0\ **Last updated:** [January 8, 2020, 12:54pm UTC](https://discuss.elastic.co/t/delete-documents-after-sync/214236 "2020-01-08T12:54:07Z")
I have a logstash config where I am putting data from one to another input { elasticsearch { hosts =\> \[ "http://localhost:9200" \] index =\> "node\_2" docinfo =\> true } } output { elasticsearch { hos… --- ## [Issue to connect AWS RDS mysql from Logstash](https://discuss.elastic.co/t/issue-to-connect-aws-rds-mysql-from-logstash/214233)
**Author:** [@vijoyem](https://discuss.elastic.co/u/vijoyem)\ **Replies:** 0\ **Last updated:** [January 8, 2020, 12:23pm UTC](https://discuss.elastic.co/t/issue-to-connect-aws-rds-mysql-from-logstash/214233 "2020-01-08T12:23:32Z")
Hello, I am setting up logstash conf file to fetch the logs from my AWS RDS MYSQL database. Please let me know what needs to be corrected. ELK is configured in the AWS ec2 single machine. All services are up and running… --- ## [Logstash parsing with combinedapachelog pattern for nginx](https://discuss.elastic.co/t/logstash-parsing-with-combinedapachelog-pattern-for-nginx/214232)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 0\ **Last updated:** [January 8, 2020, 12:23pm UTC](https://discuss.elastic.co/t/logstash-parsing-with-combinedapachelog-pattern-for-nginx/214232 "2020-01-08T12:23:12Z")
Hello everyone!! I ran into a problem with parsing my log messages with logstash. I have set ELK stack for geoip monitoring and my logstash config file looks like that: input { beats { port =\> 5044 } } filter… --- ## [Logstash jdbc string column value in sql. how to split individual field and value](https://discuss.elastic.co/t/logstash-jdbc-string-column-value-in-sql-how-to-split-individual-field-and-value/213890)
**Author:** [@saravana\_hariharan](https://discuss.elastic.co/u/saravana_hariharan)\ **Replies:** 4\ **Last updated:** [January 8, 2020, 10:52am UTC](https://discuss.elastic.co/t/logstash-jdbc-string-column-value-in-sql-how-to-split-individual-field-and-value/213890 "2020-01-08T10:52:45Z")
sql column value concatenated in {"ID":"P0001","Name":"sam","Createdby:124","Createdtime":"157312334321"} In the above line is my sample data in one clolumn value of sql table how to split this column to individual f… --- ## [Logstash throws config error, even though 'input' is listed as the first parameter on config file](https://discuss.elastic.co/t/logstash-throws-config-error-even-though-input-is-listed-as-the-first-parameter-on-config-file/214154)
**Author:** [@sg\_shreyas](https://discuss.elastic.co/u/sg_shreyas)\ **Replies:** 1\ **Last updated:** [January 8, 2020, 8:59am UTC](https://discuss.elastic.co/t/logstash-throws-config-error-even-though-input-is-listed-as-the-first-parameter-on-config-file/214154 "2020-01-08T08:59:59Z")
I am getting config error while launching Logstash. I am using 7.5.1 version. Same config file worked earlier. I had to setup ELK stash on another VM since older VM was decommissioned, and I am facing this error here. I… --- ## [Logstash Pipeline for Microsoft SQL Server to Elasticsearch](https://discuss.elastic.co/t/logstash-pipeline-for-microsoft-sql-server-to-elasticsearch/212649)
**Author:** [@jporter](https://discuss.elastic.co/u/jporter)\ **Replies:** 9\ **Last updated:** [January 8, 2020, 8:01am UTC](https://discuss.elastic.co/t/logstash-pipeline-for-microsoft-sql-server-to-elasticsearch/212649 "2020-01-08T08:01:43Z")
I have a Microsoft SQL Server which I wish to ( contuously) push into Elasticsearch. In my base directory I have a docker-compose.yml file version: "3.7" services: backend: build: ./flask … --- ## [Grok Pattern works fine in Debugger but in the Logtstash](https://discuss.elastic.co/t/grok-pattern-works-fine-in-debugger-but-in-the-logtstash/214026)
**Author:** [@sabyasachi](https://discuss.elastic.co/u/sabyasachi)\ **Replies:** 3\ **Last updated:** [January 8, 2020, 12:26am UTC](https://discuss.elastic.co/t/grok-pattern-works-fine-in-debugger-but-in-the-logtstash/214026 "2020-01-08T00:26:09Z")
My Pattern: %{MONTH:FP\_Month} %{NUMBER:FP\_Date} %{TIME} %{IP:FP\_LogSource} %{WORD}:%{INT}.%{INT}|%{WORD}|%{WORD}|%{WORD}.%{WORD}.%{WORD}|%{WORD}:%{WORD:FP\_Transaction\_Status}|%{WORD}=%{NUMBER:FP\_Severity}\\%{WORD}=%{NUMBE… --- ## [Looking for best practice recommendations for SysLog data retrieval?](https://discuss.elastic.co/t/looking-for-best-practice-recommendations-for-syslog-data-retrieval/214139)
**Author:** [@zombo](https://discuss.elastic.co/u/zombo)\ **Replies:** 0\ **Last updated:** [January 7, 2020, 10:34pm UTC](https://discuss.elastic.co/t/looking-for-best-practice-recommendations-for-syslog-data-retrieval/214139 "2020-01-07T22:34:51Z")
I'm planning to receive SysLog data from various network devices that I'm not able to directly install beats on and trying to figure out the best way to go about it. I know Beats is being leveraged more and see that it s… --- ## [Fields in documents are using a literal '.' instead of hierarchical JSON](https://discuss.elastic.co/t/fields-in-documents-are-using-a-literal-instead-of-hierarchical-json/214122)
**Author:** [@kdp494](https://discuss.elastic.co/u/kdp494)\ **Replies:** 2\ **Last updated:** [January 7, 2020, 8:11pm UTC](https://discuss.elastic.co/t/fields-in-documents-are-using-a-literal-instead-of-hierarchical-json/214122 "2020-01-07T20:11:50Z")
I am having an issue where the output from grok and other filters are producing fields with a literal '.' separator as opposed to the expected hierarchical JSON. For example, I use the following pattern in my grok filte… --- ## [Not able to see logs in the index](https://discuss.elastic.co/t/not-able-to-see-logs-in-the-index/213399)
**Author:** [@dustlesspuma](https://discuss.elastic.co/u/dustlesspuma)\ **Replies:** 4\ **Last updated:** [January 7, 2020, 7:45pm UTC](https://discuss.elastic.co/t/not-able-to-see-logs-in-the-index/213399 "2020-01-07T19:45:56Z")
I have two mutate filters created one to get all the /var/log/messages to type \> security and other mutate filter to get all the logs from one kind of hosts to type \> host\_type. I am not able to see the /var/log/message… --- ## [Removing special characters from field name](https://discuss.elastic.co/t/removing-special-characters-from-field-name/213788)
**Author:** [@Rob\_wylde](https://discuss.elastic.co/u/Rob_wylde)\ **Replies:** 3\ **Last updated:** [January 7, 2020, 2:03pm UTC](https://discuss.elastic.co/t/removing-special-characters-from-field-name/213788 "2020-01-07T14:03:50Z")
if "Holding Dealer Organic ID" { mutate { add\_tag =\> \["holding\_dealer"\] } mutate { lowercas… --- ## [JDBC issue after upgrade to 7.5](https://discuss.elastic.co/t/jdbc-issue-after-upgrade-to-7-5/214096)
**Author:** [@ginu](https://discuss.elastic.co/u/ginu)\ **Replies:** 2\ **Last updated:** [January 7, 2020, 4:48pm UTC](https://discuss.elastic.co/t/jdbc-issue-after-upgrade-to-7-5/214096 "2020-01-07T16:48:29Z")
Hi all, We have a logstash pipeline which is configured to collect data from Vertica Analytic v9.0.1-20190201 (https://www.vertica.com/)... it all worked well with the logstash version 7.4. with the configureation jdbc… --- ## [Sql\_last\_value, store value even no document indexed](https://discuss.elastic.co/t/sql-last-value-store-value-even-no-document-indexed/213695)
**Author:** [@pilo](https://discuss.elastic.co/u/pilo)\ **Replies:** 5\ **Last updated:** [January 7, 2020, 4:34pm UTC](https://discuss.elastic.co/t/sql-last-value-store-value-even-no-document-indexed/213695 "2020-01-07T16:34:36Z")
Hello everyone, I'm using logstash jdbc to transform my data in postgresql to Elasticsearch. Because my sql query is too large, about 1 millions records, i have to use schedule to transform about 5 row each time, and i… --- ## [Capture multiple information for one match grok](https://discuss.elastic.co/t/capture-multiple-information-for-one-match-grok/214069)
**Author:** [@focheur91300](https://discuss.elastic.co/u/focheur91300)\ **Replies:** 2\ **Last updated:** [January 7, 2020, 3:42pm UTC](https://discuss.elastic.co/t/capture-multiple-information-for-one-match-grok/214069 "2020-01-07T15:42:56Z")
Hello, I am currently stuck on a parsing topic. I am looking with a single match to recover several information from the same information to already capture. I'm working on DNS log: Jan 7 13:49:02 192.168.10.254 unb… --- ## [How to loop through array in Logstash?](https://discuss.elastic.co/t/how-to-loop-through-array-in-logstash/213852)
**Author:** [@Ahmet\_Kartal](https://discuss.elastic.co/u/Ahmet_Kartal)\ **Replies:** 3\ **Last updated:** [January 7, 2020, 2:52pm UTC](https://discuss.elastic.co/t/how-to-loop-through-array-in-logstash/213852 "2020-01-07T14:52:40Z")
Hello, I have this data: "url" =\> \[ \[0\] \[ \[0\] "", \[1\] "data", \[2\] "data1" \[3\] "9462" \], What I want is that I need to get last element of list as assignment\_1 ( I need to… --- ## [Logstash - seconds part is missing for ES plugin](https://discuss.elastic.co/t/logstash-seconds-part-is-missing-for-es-plugin/214048)
**Author:** [@psiva017](https://discuss.elastic.co/u/psiva017)\ **Replies:** 0\ **Last updated:** [January 7, 2020, 12:25pm UTC](https://discuss.elastic.co/t/logstash-seconds-part-is-missing-for-es-plugin/214048 "2020-01-07T12:25:35Z")
I need to monitor the ES. I tries with following logstash config. input{ elasticsearch { hosts =\> \["http://localhost:9200"\] index =\> "my\_index" query =\> '{"query":{"match\_all": {}}}' schedule =\> "0-59 \* \* \* \*" } … --- ## [Logstash - Get index name from filename](https://discuss.elastic.co/t/logstash-get-index-name-from-filename/213891)
**Author:** [@Lubos\_Marek](https://discuss.elastic.co/u/Lubos_Marek)\ **Replies:** 3\ **Last updated:** [January 7, 2020, 10:13am UTC](https://discuss.elastic.co/t/logstash-get-index-name-from-filename/213891 "2020-01-07T10:13:35Z")
Hello, I am loading files by logstash and I would like to know if it possible to set index from filename. For example my files are: system1-yyyyMMdd.csv (system1-20200106.csv) system2-yyyyMMdd.csv (system2-20200106.c… --- ## [CosmosDB configuration](https://discuss.elastic.co/t/cosmosdb-configuration/213997)
**Author:** [@Croos](https://discuss.elastic.co/u/Croos)\ **Replies:** 0\ **Last updated:** [January 7, 2020, 7:43am UTC](https://discuss.elastic.co/t/cosmosdb-configuration/213997 "2020-01-07T07:43:50Z")
Hi, Currently, I have the following config file for logstash. input { jdbc { #https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-record\_last\_run jdbc\_connection\_strin… --- ## [What does this error mean](https://discuss.elastic.co/t/what-does-this-error-mean/213971)
**Author:** [@volcano](https://discuss.elastic.co/u/volcano)\ **Replies:** 0\ **Last updated:** [January 7, 2020, 4:02am UTC](https://discuss.elastic.co/t/what-does-this-error-mean/213971 "2020-01-07T04:02:16Z")
what does this logstash error mean "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Can't merge a non object mapping \[batch-upload-usage.costcalculation.elapsed-time\] with an object mapping \[bat… --- ## [Default index lifecycle management period](https://discuss.elastic.co/t/default-index-lifecycle-management-period/213977)
**Author:** [@suraj\_kumar3](https://discuss.elastic.co/u/suraj_kumar3)\ **Replies:** 0\ **Last updated:** [January 7, 2020, 5:08am UTC](https://discuss.elastic.co/t/default-index-lifecycle-management-period/213977 "2020-01-07T05:08:27Z")
What is the default lifecycle of indexes of .monitoring-kibana and .monitoring-es? --- ## [Watcher to monitor Logstash Pipeline "Events Emitted Rate"](https://discuss.elastic.co/t/watcher-to-monitor-logstash-pipeline-events-emitted-rate/213958)
**Author:** [@SSS1](https://discuss.elastic.co/u/SSS1)\ **Replies:** 1\ **Last updated:** [January 7, 2020, 4:16am UTC](https://discuss.elastic.co/t/watcher-to-monitor-logstash-pipeline-events-emitted-rate/213958 "2020-01-07T04:16:52Z")
Hi, I need to create a Watcher to alert when a pipeline's "Events Emitted Rate" value drops below a certain threshold. Is this possible to do in the current monitoring tools? I've tried to run querying myself, but can… --- ## [Logstash JDBC and Firebird](https://discuss.elastic.co/t/logstash-jdbc-and-firebird/213393)
**Author:** [@Osiel](https://discuss.elastic.co/u/Osiel)\ **Replies:** 3\ **Last updated:** [January 7, 2020, 2:54am UTC](https://discuss.elastic.co/t/logstash-jdbc-and-firebird/213393 "2020-01-07T02:54:44Z")
I need some help setting up logstash with jdbc and firebird 2.5 database, I'm using version 7.5 of elastic and logstash. What happens is that I always need to overwrite the information in my indexes, ie will always have… --- ## [Multiple grok parsing not extracting all the fields](https://discuss.elastic.co/t/multiple-grok-parsing-not-extracting-all-the-fields/213764)
**Author:** [@nmoham](https://discuss.elastic.co/u/nmoham)\ **Replies:** 4\ **Last updated:** [January 7, 2020, 2:28am UTC](https://discuss.elastic.co/t/multiple-grok-parsing-not-extracting-all-the-fields/213764 "2020-01-07T02:28:51Z")
Trying to extract some fields from the msgbody field using grok , but only the first field in the gork gets extracted. interested Fields - corId, controller, httpStatusText and uri Sample Data - 2020-01-03 10:44:17,02… --- ## [Hardware requirement for logstash](https://discuss.elastic.co/t/hardware-requirement-for-logstash/213831)
**Author:** [@Jayanth\_S\_N](https://discuss.elastic.co/u/Jayanth_S_N)\ **Replies:** 2\ **Last updated:** [January 6, 2020, 11:22pm UTC](https://discuss.elastic.co/t/hardware-requirement-for-logstash/213831 "2020-01-06T23:22:39Z")
What will be the best suitable hardware configuration for logstash, we are reading the data from Apache Kafka and adding a few filters in logstash config file on the different sources of data and then data will be sent t… --- ## [Error with outside gem for custom plugin, bug in pipeline](https://discuss.elastic.co/t/error-with-outside-gem-for-custom-plugin-bug-in-pipeline/213915)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 0\ **Last updated:** [January 6, 2020, 4:14pm UTC](https://discuss.elastic.co/t/error-with-outside-gem-for-custom-plugin-bug-in-pipeline/213915 "2020-01-06T16:14:38Z")
Hi All I am developing a very small plugin, in which in rb file, i am using a ruby gem require 'crack' i installed this crack gem, but still i get this error I installed the external gem like this /opt/logstash/bin… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=422) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=424)