# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=489 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 490 --- ## [Jdbc plugin ships Repeated data from mongodb](https://discuss.elastic.co/t/jdbc-plugin-ships-repeated-data-from-mongodb/189914)
**Author:** [@chandu5565](https://discuss.elastic.co/u/chandu5565)\ **Replies:** 15\ **Last updated:** [August 22, 2019, 7:16am UTC](https://discuss.elastic.co/t/jdbc-plugin-ships-repeated-data-from-mongodb/189914 "2019-08-22T07:16:43Z")
Hi, I am using logstash JDBC plugin to ship data from Mongo DB to elasticsearch. The data stored in the Mongo DB is in JSON documents. When logstash try to fetch the data every 1 min its getting repeated data. Total doc… --- ## [Changing Default Logstash Port - 5044](https://discuss.elastic.co/t/changing-default-logstash-port-5044/196091)
**Author:** [@rakesh15](https://discuss.elastic.co/u/rakesh15)\ **Replies:** 2\ **Last updated:** [August 22, 2019, 7:02am UTC](https://discuss.elastic.co/t/changing-default-logstash-port-5044/196091 "2019-08-22T07:02:15Z")
Hi All, I would like to change the default logstash port from 5044 to 6044. Can any one please share some pointers for the same. I'm trying to run multiple instances of logstash on same machine. Is running Logstash usi… --- ## [Elk cloud](https://discuss.elastic.co/t/elk-cloud/196236)
**Author:** [@pavan3](https://discuss.elastic.co/u/pavan3)\ **Replies:** 0\ **Last updated:** [August 22, 2019, 5:33am UTC](https://discuss.elastic.co/t/elk-cloud/196236 "2019-08-22T05:33:46Z")
how to export a data from mssq to elk cloud --- ## [Identify data upload completed by file plugin in logstasg](https://discuss.elastic.co/t/identify-data-upload-completed-by-file-plugin-in-logstasg/196074)
**Author:** [@ManuMP](https://discuss.elastic.co/u/ManuMP)\ **Replies:** 2\ **Last updated:** [August 22, 2019, 5:23am UTC](https://discuss.elastic.co/t/identify-data-upload-completed-by-file-plugin-in-logstasg/196074 "2019-08-22T05:23:16Z")
We use the following configuration to upload log entries in all log files to elastic search. The OS is Windows 10. We are using elastic search to identify patterns in the log files after completing log data uploaded by … --- ## [Unable to ingest CSV file into Elasticsearch through Logstash](https://discuss.elastic.co/t/unable-to-ingest-csv-file-into-elasticsearch-through-logstash/196036)
**Author:** [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\ **Replies:** 4\ **Last updated:** [August 22, 2019, 3:58am UTC](https://discuss.elastic.co/t/unable-to-ingest-csv-file-into-elasticsearch-through-logstash/196036 "2019-08-22T03:58:52Z")
C:\\Users\\ramya.t\\logstash-7.3.0\\bin\>logstash -f C:\\Users\\ramya.t\\logstash-7.3.0\\bin\\log.conf Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed … --- ## [Logstash index lifecycle management](https://discuss.elastic.co/t/logstash-index-lifecycle-management/196142)
**Author:** [@Jiri\_Safar](https://discuss.elastic.co/u/Jiri_Safar)\ **Replies:** 1\ **Last updated:** [August 22, 2019, 3:03am UTC](https://discuss.elastic.co/t/logstash-index-lifecycle-management/196142 "2019-08-22T03:03:32Z")
Hi guys, I would like to set "index lifecycle management " on LOGSTASH output. It would means that everything that goes via this output will get "ndex lifecycle management" which was created in Kibana. This is how my … --- ## [Flatten nested json array](https://discuss.elastic.co/t/flatten-nested-json-array/196192)
**Author:** [@rokitpower](https://discuss.elastic.co/u/rokitpower)\ **Replies:** 2\ **Last updated:** [August 21, 2019, 9:34pm UTC](https://discuss.elastic.co/t/flatten-nested-json-array/196192 "2019-08-21T21:34:07Z")
I'm trying to flatten this json object with nested arrays into individual objects at the top level. It's a little weird because the array elements have different elements { "MyData": { "identifier" : "ab… --- ## [CSV parse/read error when dumped into a folder](https://discuss.elastic.co/t/csv-parse-read-error-when-dumped-into-a-folder/196185)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 0\ **Last updated:** [August 21, 2019, 7:33pm UTC](https://discuss.elastic.co/t/csv-parse-read-error-when-dumped-into-a-folder/196185 "2019-08-21T19:33:32Z")
I have the following case: I configured logstash to monitor a specific root and subdirectories if there is any folder created with a csv file in it then it should read it and push it to elasticsearch. Here is how I conf… --- ## [Translate filter for Azure subscription ID](https://discuss.elastic.co/t/translate-filter-for-azure-subscription-id/196177)
**Author:** [@spate](https://discuss.elastic.co/u/spate)\ **Replies:** 0\ **Last updated:** [August 21, 2019, 6:39pm UTC](https://discuss.elastic.co/t/translate-filter-for-azure-subscription-id/196177 "2019-08-21T18:39:15Z")
Sorry I can't figure out how to format this properly. logstash 1:7.3.0-1 I am pulling in data from an Azure event hub using the logstash azure\_event\_hubs plugin. Without any modification, the data displayed in Kibana s… --- ## [Remove the pri-tag from rsyslog messages](https://discuss.elastic.co/t/remove-the-pri-tag-from-rsyslog-messages/196161)
**Author:** [@dwenger](https://discuss.elastic.co/u/dwenger)\ **Replies:** 2\ **Last updated:** [August 21, 2019, 5:57pm UTC](https://discuss.elastic.co/t/remove-the-pri-tag-from-rsyslog-messages/196161 "2019-08-21T17:57:35Z")
Hi, I'm trying to put postfix logs to elasticsearch. I'm stuck with parsing the pri-tag - in fact, I want to remove it from the message. Here's a line from logs being sent: \<22\>Aug 21 18:34:14 mailserver postfix/qmgr… --- ## [Problem with mutate - convert and mutate - rename](https://discuss.elastic.co/t/problem-with-mutate-convert-and-mutate-rename/196150)
**Author:** [@franco.federico](https://discuss.elastic.co/u/franco.federico)\ **Replies:** 3\ **Last updated:** [August 21, 2019, 5:41pm UTC](https://discuss.elastic.co/t/problem-with-mutate-convert-and-mutate-rename/196150 "2019-08-21T17:41:11Z")
Hi all I make a filter to intercept the log of squid. I make a pattern with grok debug that run and convert the various fields in fields in elasticsearch. I'm trying to apply ECS Schema and for 2 fields I have first a… --- ## [Error : "tags" =\> \[ \[0\] "\_grokparsefailure" \]..Grok filter error](https://discuss.elastic.co/t/error-tags-0-grokparsefailure-grok-filter-error/195384)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 21\ **Last updated:** [August 21, 2019, 5:16pm UTC](https://discuss.elastic.co/t/error-tags-0-grokparsefailure-grok-filter-error/195384 "2019-08-21T17:16:21Z")
Hello, I am using grok filters for parsing messages and I can see parsing failures but can't figure out what's causing the issue. Grok Filter: filter { # Create a copy of original message mutate { … --- ## [Sending logs from on-prem logstash to hosted enterprise logstash](https://discuss.elastic.co/t/sending-logs-from-on-prem-logstash-to-hosted-enterprise-logstash/196149)
**Author:** [@IanJazz](https://discuss.elastic.co/u/IanJazz)\ **Replies:** 0\ **Last updated:** [August 21, 2019, 3:42pm UTC](https://discuss.elastic.co/t/sending-logs-from-on-prem-logstash-to-hosted-enterprise-logstash/196149 "2019-08-21T15:42:09Z")
Hi guys, I'm just reaching out for someone to point me in the right direction with a relatively simple query I have. We have an elastic.co hosted ELK stack and I am in the process of building a logstash instance in our… --- ## [Get last folder form path in Logstash](https://discuss.elastic.co/t/get-last-folder-form-path-in-logstash/195991)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 8:26pm UTC](https://discuss.elastic.co/t/get-last-folder-form-path-in-logstash/195991 "2019-08-20T20:26:48Z")
I need to extract the name of the last folder in the path where the file plugin caught a csv file and then create a field that has the value of the folder, which it is a taskId in my case. Here is an example of what I'm… --- ## [Logstash date filter convert @timestamp to timestamp in log file issue](https://discuss.elastic.co/t/logstash-date-filter-convert-timestamp-to-timestamp-in-log-file-issue/196100)
**Author:** [@dcrocker](https://discuss.elastic.co/u/dcrocker)\ **Replies:** 2\ **Last updated:** [August 21, 2019, 2:49pm UTC](https://discuss.elastic.co/t/logstash-date-filter-convert-timestamp-to-timestamp-in-log-file-issue/196100 "2019-08-21T14:49:18Z")
I am trying to convert the log file time stamp to @timestamp, but it is not working. Following is the date in the job log: "2019-08-20T11:52:18.4906519-04:00" Following is the date filter: date { # use timestamp from… --- ## [Index not created, logstash](https://discuss.elastic.co/t/index-not-created-logstash/196110)
**Author:** [@hunter84](https://discuss.elastic.co/u/hunter84)\ **Replies:** 0\ **Last updated:** [August 21, 2019, 12:38pm UTC](https://discuss.elastic.co/t/index-not-created-logstash/196110 "2019-08-21T12:38:13Z")
Hello! I have windows 2012 server, it runs an ELK stack. I am trying to parse using logstash, the index has not yet been created in elastiсsearch, and it does not appear after running the config file. input { file { … --- ## [Matching direction in dissect](https://discuss.elastic.co/t/matching-direction-in-dissect/195995)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 3\ **Last updated:** [August 21, 2019, 12:29pm UTC](https://discuss.elastic.co/t/matching-direction-in-dissect/195995 "2019-08-21T12:29:01Z")
I have log lines like the following: src=192.168.100.1:502 rest of message src=\[::1\]:304 message details The format is \:\. I could use dissect easily except for the case shown in the second line. This… --- ## [Getting error while running logstash](https://discuss.elastic.co/t/getting-error-while-running-logstash/196035)
**Author:** [@jainabhishek0406](https://discuss.elastic.co/u/jainabhishek0406)\ **Replies:** 1\ **Last updated:** [August 21, 2019, 12:25pm UTC](https://discuss.elastic.co/t/getting-error-while-running-logstash/196035 "2019-08-21T12:25:53Z")
facing issue and getting below error \[2019-08-19T14:27:41,342\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"Java: :JavaLang::NoSuchMethodErr… --- ## [Logstash throwing error .Unexpected character (‘\<’ (code 60))](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005)
**Author:** [@prabhat\_ranjan](https://discuss.elastic.co/u/prabhat_ranjan)\ **Replies:** 11\ **Last updated:** [August 21, 2019, 12:14pm UTC](https://discuss.elastic.co/t/logstash-throwing-error-unexpected-character-code-60/195005 "2019-08-21T12:14:58Z")
Hi team, I am using ELK on the cloud. Using filebeat to send data to logstash. Logstash sends to ES which pushes it in Kibana.It appears most of the data is properly parsed and sent to Kibana successfully. There are ins… --- ## [Parse json as string](https://discuss.elastic.co/t/parse-json-as-string/196069)
**Author:** [@Ivan\_Martos](https://discuss.elastic.co/u/Ivan_Martos)\ **Replies:** 1\ **Last updated:** [August 21, 2019, 12:07pm UTC](https://discuss.elastic.co/t/parse-json-as-string/196069 "2019-08-21T12:07:12Z")
I'm having issues with parsing JSON as String using json filter Sample stdout output from logstash without any filters looks like this { "agent": { "type": "filebeat", "ephemeral\_id": "0a8384b3-c4df… --- ## [Connect kinsta mysql](https://discuss.elastic.co/t/connect-kinsta-mysql/196089)
**Author:** [@zernalda](https://discuss.elastic.co/u/zernalda)\ **Replies:** 0\ **Last updated:** [August 21, 2019, 11:00am UTC](https://discuss.elastic.co/t/connect-kinsta-mysql/196089 "2019-08-21T11:00:05Z")
I want to connect my kinsta DB to visualize on kibana. but error when i run .conf in logstash. its my .conf = input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://35.187.353.208/db?useSSL=true&verifyServ… --- ## [Updated JDBC connection string but not seeing changes pull through](https://discuss.elastic.co/t/updated-jdbc-connection-string-but-not-seeing-changes-pull-through/196080)
**Author:** [@jared-nelson](https://discuss.elastic.co/u/jared-nelson)\ **Replies:** 0\ **Last updated:** [August 21, 2019, 10:01am UTC](https://discuss.elastic.co/t/updated-jdbc-connection-string-but-not-seeing-changes-pull-through/196080 "2019-08-21T10:01:50Z")
Hi everyone, New to ELK so will do my best to explain. We're using the JDBC input plugin and recently changed the MySQL server the conf file refers to in the connection string. However I wanted to confirm what steps ar… --- ## [Logstash Twitter plugin issues](https://discuss.elastic.co/t/logstash-twitter-plugin-issues/195950)
**Author:** [@gur79](https://discuss.elastic.co/u/gur79)\ **Replies:** 1\ **Last updated:** [August 21, 2019, 9:23am UTC](https://discuss.elastic.co/t/logstash-twitter-plugin-issues/195950 "2019-08-21T09:23:07Z")
Hello Guys. Yesterday i tried to use Twitter Logstash input plugin to connect to Twitter Stream API to get tweets contain word "Twitter". the Question is, I run the Logstash for 2 hours and it didn't even return one st… --- ## [Add new field from message](https://discuss.elastic.co/t/add-new-field-from-message/195737)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 5\ **Last updated:** [August 21, 2019, 6:51am UTC](https://discuss.elastic.co/t/add-new-field-from-message/195737 "2019-08-21T06:51:11Z")
Hi, I wonder if someone can help. I have looked online and I haven't found a solution. We have an event that passes a message like "error detected in request error code: \[error code\] please check" Is there a way to be … --- ## [Logstash 7.3.0 node stats API does not list all pipelines](https://discuss.elastic.co/t/logstash-7-3-0-node-stats-api-does-not-list-all-pipelines/195724)
**Author:** [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Replies:** 1\ **Last updated:** [August 21, 2019, 5:16am UTC](https://discuss.elastic.co/t/logstash-7-3-0-node-stats-api-does-not-list-all-pipelines/195724 "2019-08-21T05:16:10Z")
Hi, We have a logstash deployment where we use the Node Stats API to independently track logstash performance/status. Till 7.2.0, the API behaved as documented, but after update to 7.3.0, we are seeing inconsistent repo… --- ## [Logstash config not creating an index in elasticsearch](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736)
**Author:** [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\ **Replies:** 6\ **Last updated:** [August 21, 2019, 2:01am UTC](https://discuss.elastic.co/t/logstash-config-not-creating-an-index-in-elasticsearch/195736 "2019-08-21T02:01:13Z")
Hey elastic team, ive been trying to write a filter for json files so that it gets indexed into elasticsearch but its not creating an index in elastic search. There is no error in the logstash config file. This is a sam… --- ## [Duplicate records in indexes](https://discuss.elastic.co/t/duplicate-records-in-indexes/194443)
**Author:** [@SirJune](https://discuss.elastic.co/u/SirJune)\ **Replies:** 4\ **Last updated:** [August 20, 2019, 9:00pm UTC](https://discuss.elastic.co/t/duplicate-records-in-indexes/194443 "2019-08-20T21:00:06Z")
HI, I have logstash running connecting/reading records from MySQL DB. I have 7 records in a timeframe (say: 01:00:00 - 01:01:00) I have this config file: tracking\_column =\> "datetime" use\_column\_value =\>true stateme… --- ## [Process - Reshape a CSV file using logstash](https://discuss.elastic.co/t/process-reshape-a-csv-file-using-logstash/195270)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 29\ **Last updated:** [August 20, 2019, 8:42pm UTC](https://discuss.elastic.co/t/process-reshape-a-csv-file-using-logstash/195270 "2019-08-20T20:42:42Z")
Is there a way where I can process any csv file dynamically detect the data\_type for the columns, either float or string. as an example, I have this csv file table and I need to dynamically turn it into this schema a… --- ## [Input file reads only newly added files](https://discuss.elastic.co/t/input-file-reads-only-newly-added-files/195956)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 1\ **Last updated:** [August 20, 2019, 8:18pm UTC](https://discuss.elastic.co/t/input-file-reads-only-newly-added-files/195956 "2019-08-20T20:18:43Z")
I have a service that outputs two files into a file system, csv and manifest files, both have same file name but different extension. I need to build a logstash config file that does the following: Once the files are … --- ## [Unable to extract the message filed from the logstash filter](https://discuss.elastic.co/t/unable-to-extract-the-message-filed-from-the-logstash-filter/195807)
**Author:** [@Arun\_Soman](https://discuss.elastic.co/u/Arun_Soman)\ **Replies:** 6\ **Last updated:** [August 20, 2019, 7:02pm UTC](https://discuss.elastic.co/t/unable-to-extract-the-message-filed-from-the-logstash-filter/195807 "2019-08-20T19:02:54Z")
I am trying to split my message into different fields . I used Grok filer on logstash but I am still getting the same message content in the Kibana discover dash board. My message as follows. 2019-8-16T18:37:16.45 CT3 … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=488) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=490)