# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=742 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 743 --- ## [Dateparsefailure on conversion](https://discuss.elastic.co/t/dateparsefailure-on-conversion/135592)
**Author:** [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Replies:** 2\ **Last updated:** [June 12, 2018, 7:56pm UTC](https://discuss.elastic.co/t/dateparsefailure-on-conversion/135592 "2018-06-12T19:56:55Z")
Hi All. I have the following timestamp 20180612 21:07:10.197 and I am using the this match: date { # 2018-06-04T07:25:13.943Z # 20180605 16:37:34.966 match =\> \['ts', 'yyyymmdd hh:mm:ss.SSS'\] tim… --- ## [Restart logstash-Beats](https://discuss.elastic.co/t/restart-logstash-beats/135569)
**Author:** [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\ **Replies:** 1\ **Last updated:** [June 12, 2018, 4:46pm UTC](https://discuss.elastic.co/t/restart-logstash-beats/135569 "2018-06-12T16:46:13Z")
Hello, I have a question that seems naive at first but is very relevant. Imagine that I have 300 agents (Beats), which are linked with a logstash. after modifying the loogstash conf file (add filter or other), do I hav… --- ## [Having trouble assigning value from field A to field B](https://discuss.elastic.co/t/having-trouble-assigning-value-from-field-a-to-field-b/135546)
**Author:** [@JimP](https://discuss.elastic.co/u/JimP)\ **Replies:** 3\ **Last updated:** [June 12, 2018, 4:38pm UTC](https://discuss.elastic.co/t/having-trouble-assigning-value-from-field-a-to-field-b/135546 "2018-06-12T16:38:53Z")
We have a field \[env\_id\] which is sent from filebeat. In logstash, I want to assign this value to field, \[topic\]\[env\], but am failing miserably. Here's the latest that I've tried which doesn't work. This should be str… --- ## [Mssql data into elasticsearch using logstash](https://discuss.elastic.co/t/mssql-data-into-elasticsearch-using-logstash/135556)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 2\ **Last updated:** [June 12, 2018, 3:34pm UTC](https://discuss.elastic.co/t/mssql-data-into-elasticsearch-using-logstash/135556 "2018-06-12T15:34:38Z")
Hey Guys, I have been trying to upload data from mssql database into elasticsearch using logstash. Have succeeded with uploading some smaller table informations, when i try to upload tables like EVENTS which has around… --- ## [Logstash using incorrect address](https://discuss.elastic.co/t/logstash-using-incorrect-address/134641)
**Author:** [@jarnold](https://discuss.elastic.co/u/jarnold)\ **Replies:** 2\ **Last updated:** [June 7, 2018, 8:58am UTC](https://discuss.elastic.co/t/logstash-using-incorrect-address/134641 "2018-06-07T08:58:19Z")
Logstash appears to be using localhost:9200 to connect to my elasticsearch instance instead of its proper IP address, logstash error output here; Elasticsearch Unreachable: \[http://localhost:9200/\]\[Manticore::SocketExce… --- ## [UTF-16 Broken since logstash 6](https://discuss.elastic.co/t/utf-16-broken-since-logstash-6/135558)
**Author:** [@maigel](https://discuss.elastic.co/u/maigel)\ **Replies:** 0\ **Last updated:** [June 12, 2018, 2:24pm UTC](https://discuss.elastic.co/t/utf-16-broken-since-logstash-6/135558 "2018-06-12T14:24:09Z")
Hello, I recently migrated our cluster from ELK2 to ELK6. One of our logstash configs is used to parse some powershell CSV output, which is in UTF-16LE (Little-endian UTF-16 Unicode text, with very long lines, with CR… --- ## [JDBC input: connection OK, request OK, but no result](https://discuss.elastic.co/t/jdbc-input-connection-ok-request-ok-but-no-result/135540)
**Author:** [@AurelienG](https://discuss.elastic.co/u/AurelienG)\ **Replies:** 4\ **Last updated:** [June 12, 2018, 2:19pm UTC](https://discuss.elastic.co/t/jdbc-input-connection-ok-request-ok-but-no-result/135540 "2018-06-12T14:19:35Z")
Hello, I think the title says a lot already but here are some details. I have a PostgreSQL database, which I try to get data from with the JDBC input plugin. There doesn't seem to be a problem in my Logstash config file… --- ## [Daily log file + archived .zip](https://discuss.elastic.co/t/daily-log-file-archived-zip/135554)
**Author:** [@Bfoucher](https://discuss.elastic.co/u/Bfoucher)\ **Replies:** 0\ **Last updated:** [June 12, 2018, 2:16pm UTC](https://discuss.elastic.co/t/daily-log-file-archived-zip/135554 "2018-06-12T14:16:47Z")
Hello, I have an application that feeds several log files, which I'm doing fine parsing at the moment. Each day, at 7am, the log file (nameOfFile.log) gets zipped into nameOfFile.20180612.log.zip. These zip are kept fo… --- ## [Running Logstash in a Docker container: data directory permissions](https://discuss.elastic.co/t/running-logstash-in-a-docker-container-data-directory-permissions/135028)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 10\ **Last updated:** [June 12, 2018, 1:56pm UTC](https://discuss.elastic.co/t/running-logstash-in-a-docker-container-data-directory-permissions/135028 "2018-06-12T13:56:18Z")
Hello, I'm trying to run a Docker container with this setup: docker run --name logstash6 -d -v /srv/logstash6/pipeline/:/usr/share/logstash/pipeline/ -v /srv/logstash6/data/:/usr/share/logstash/data/ docker.elastic.co/l… --- ## [Grok pattern test](https://discuss.elastic.co/t/grok-pattern-test/134835)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 2\ **Last updated:** [June 6, 2018, 3:53pm UTC](https://discuss.elastic.co/t/grok-pattern-test/134835 "2018-06-06T15:53:06Z")
hi everyone , i wanna do filter grok for some expression the filter work but there is only one probleme there is the txt : we have detected abuse from the IP address ( 197.230.107.154 ), which according to a whois lo… --- ## [Yourself to create a partition file filter for this FTP log but but you cannot be do you can be do not yourself. yourself thanks to many](https://discuss.elastic.co/t/yourself-to-create-a-partition-file-filter-for-this-ftp-log-but-but-you-cannot-be-do-you-can-be-do-not-yourself-yourself-thanks-to-many/135521)
**Author:** [@ngocky](https://discuss.elastic.co/u/ngocky)\ **Replies:** 0\ **Last updated:** [June 12, 2018, 11:08am UTC](https://discuss.elastic.co/t/yourself-to-create-a-partition-file-filter-for-this-ftp-log-but-but-you-cannot-be-do-you-can-be-do-not-yourself-yourself-thanks-to-many/135521 "2018-06-12T11:08:40Z")
\[l%C3%B2%20ftp\] --- ## [Start two logstash and one of them create default template and one not](https://discuss.elastic.co/t/start-two-logstash-and-one-of-them-create-default-template-and-one-not/135279)
**Author:** [@zeroliuhuang](https://discuss.elastic.co/u/zeroliuhuang)\ **Replies:** 4\ **Last updated:** [June 12, 2018, 11:00am UTC](https://discuss.elastic.co/t/start-two-logstash-and-one-of-them-create-default-template-and-one-not/135279 "2018-06-12T11:00:02Z")
Version: logstash-5.4.1 Operating System: CentOS 6.9 Config File (if you have sensitive info, please remove it): input { beats { port =\> 5044 # codec =\> multiline { # pattern =\> "^# User@H… --- ## [SQl to Elasticsearch pipeline is breaking?](https://discuss.elastic.co/t/sql-to-elasticsearch-pipeline-is-breaking/135514)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 0\ **Last updated:** [June 12, 2018, 10:38am UTC](https://discuss.elastic.co/t/sql-to-elasticsearch-pipeline-is-breaking/135514 "2018-06-12T10:38:24Z")
Hi all.. I'm trying to load data from sql to elasticsearch.While loading the data we have around 1 lakh records but after 2500 records are inserted to elasticsearch pipeline is ended.Can anyone help me out.:smile: --- ## [Input Filter: Syslog vs TCP/UDP](https://discuss.elastic.co/t/input-filter-syslog-vs-tcp-udp/135437)
**Author:** [@mrahim](https://discuss.elastic.co/u/mrahim)\ **Replies:** 2\ **Last updated:** [June 12, 2018, 9:26am UTC](https://discuss.elastic.co/t/input-filter-syslog-vs-tcp-udp/135437 "2018-06-12T09:26:26Z")
Hi, I am looking to create an ELK pipeline for a variety of Syslog messages. While I thought it would make sense to use the Syslog input filter in Logstash, I read this piece about why Syslog might not be the most benefi… --- ## [Is there any way to "pull" logs into logstash, rather than "pushing" them from Filebeat?](https://discuss.elastic.co/t/is-there-any-way-to-pull-logs-into-logstash-rather-than-pushing-them-from-filebeat/132371)
**Author:** [@jarnold](https://discuss.elastic.co/u/jarnold)\ **Replies:** 3\ **Last updated:** [June 12, 2018, 9:21am UTC](https://discuss.elastic.co/t/is-there-any-way-to-pull-logs-into-logstash-rather-than-pushing-them-from-filebeat/132371 "2018-06-12T09:21:59Z")
I've been tasked with implementing an Elastic stack cluster to deal with monitoring for some server logs - Some of the servers we need to monitor are based on an external network. Due to our security policy, I can't mak… --- ## [Need help in Logstash](https://discuss.elastic.co/t/need-help-in-logstash/135388)
**Author:** [@Parinita\_Rani](https://discuss.elastic.co/u/Parinita_Rani)\ **Replies:** 1\ **Last updated:** [June 12, 2018, 9:04am UTC](https://discuss.elastic.co/t/need-help-in-logstash/135388 "2018-06-12T09:04:08Z")
C:\\Program Files\\logstash-6.2.4\\bin\>logstash -f logstash\_car.config Sending Logstash's logs to C:/Program Files/logstash-6.2.4/logs which is now con figured via log4j2.properties \[2018-06-11T07:54:38,798\]\[INFO \]\[logst… --- ## [Logstash 6.2.4 Ruby Issue](https://discuss.elastic.co/t/logstash-6-2-4-ruby-issue/133794)
**Author:** [@MariumHassan](https://discuss.elastic.co/u/MariumHassan)\ **Replies:** 4\ **Last updated:** [June 12, 2018, 8:30am UTC](https://discuss.elastic.co/t/logstash-6-2-4-ruby-issue/133794 "2018-06-12T08:30:01Z")
Hi, I have a Logstash script in which I have created some ruby functions. I have placed them at the beginning of my logstash filter so in this way, they are declared before they are called. The script works fine when I… --- ## [How to parse JSON field obtained from split filter](https://discuss.elastic.co/t/how-to-parse-json-field-obtained-from-split-filter/135372)
**Author:** [@gonzalo](https://discuss.elastic.co/u/gonzalo)\ **Replies:** 4\ **Last updated:** [June 12, 2018, 8:05am UTC](https://discuss.elastic.co/t/how-to-parse-json-field-obtained-from-split-filter/135372 "2018-06-12T08:05:20Z")
Hello all, I have used the split filter in order to parse a multiline XML and as a result, one of the fields obtained is the one called parsed.logEntry.logTimestamp as you can see in the following image: \[imagen\] What… --- ## [Csv character limitation per row](https://discuss.elastic.co/t/csv-character-limitation-per-row/135309)
**Author:** [@bobbytan](https://discuss.elastic.co/u/bobbytan)\ **Replies:** 11\ **Last updated:** [June 12, 2018, 8:02am UTC](https://discuss.elastic.co/t/csv-character-limitation-per-row/135309 "2018-06-12T08:02:53Z")
Hello, I am new to the ELK,i have noticed that certain rows of my CSV file that exceed about 250 characters are having issues being visualized in kibana after importing using logstash. Is there a way to solve this? --- ## [Replace multiple value in a specified field](https://discuss.elastic.co/t/replace-multiple-value-in-a-specified-field/134780)
**Author:** [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Replies:** 5\ **Last updated:** [June 12, 2018, 6:59am UTC](https://discuss.elastic.co/t/replace-multiple-value-in-a-specified-field/134780 "2018-06-12T06:59:25Z")
Hi, I want to replace 6 to TCP and 17 to UDP in the PROTOCOL field. this is my logstash config: input{ tcp{ host =\> "163.19.X.X" port =\> 5510 codec =\> json … --- ## [No configuration found in the configured sources](https://discuss.elastic.co/t/no-configuration-found-in-the-configured-sources/135420)
**Author:** [@mrahim](https://discuss.elastic.co/u/mrahim)\ **Replies:** 1\ **Last updated:** [June 12, 2018, 6:43am UTC](https://discuss.elastic.co/t/no-configuration-found-in-the-configured-sources/135420 "2018-06-12T06:43:58Z")
I am looking to create a simple logstash configuration in the Linux command line according to the first example shown here Logstash Configuration Examples | Logstash Reference \[6.3\] | Elastic . Here is all that is uncom… --- ## [Logstash-input-jdbc append where clause?](https://discuss.elastic.co/t/logstash-input-jdbc-append-where-clause/135442)
**Author:** [@dmongit](https://discuss.elastic.co/u/dmongit)\ **Replies:** 1\ **Last updated:** [June 12, 2018, 6:07am UTC](https://discuss.elastic.co/t/logstash-input-jdbc-append-where-clause/135442 "2018-06-12T06:07:02Z")
I want to reuse some query files. I'm using jdbc statement\_filepath for a logstash input that is NOT on a schedule. I have another jdbc input with statement\_filepath with a very similar query that IS on a schedule. In fa… --- ## [Date format match](https://discuss.elastic.co/t/date-format-match/135191)
**Author:** [@Roque\_Moyano](https://discuss.elastic.co/u/Roque_Moyano)\ **Replies:** 4\ **Last updated:** [June 12, 2018, 6:02am UTC](https://discuss.elastic.co/t/date-format-match/135191 "2018-06-12T06:02:38Z")
Hi there I just read and saw a lot of examples but in can't solve it , I have my filter of this way: filter { grok { match =\> { "message" =\>"\[%{TIMESTAMP\_ISO8601:date\_log}\] | \[%{WORD:ENV}\] | \[%{WORD:APPLICATION}\] | \[… --- ## [Not able to create an indexes for files stored in S3](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759)
**Author:** [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)\ **Replies:** 10\ **Last updated:** [June 12, 2018, 4:44am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759 "2018-06-12T04:44:58Z")
Hi Team, Able to execute and create an index in kibana if I keep the \*.csv in local machine but, I don't if I keep them in S3 bucket. Below are the config details. input { s3 { access\_key\_id =\> "Access KEY" secret\_a… --- ## [Remove logstash default timestamp and host info](https://discuss.elastic.co/t/remove-logstash-default-timestamp-and-host-info/135333)
**Author:** [@simple365](https://discuss.elastic.co/u/simple365)\ **Replies:** 2\ **Last updated:** [June 12, 2018, 1:42am UTC](https://discuss.elastic.co/t/remove-logstash-default-timestamp-and-host-info/135333 "2018-06-12T01:42:14Z")
Hi there: My env info Logstash 2.4.1 Kafka 2.10-0.8.2.1 My logstash config is input { kafka { group\_id =\> "salo" zk\_connect=\>"192.168.5.126:2181,192.168.5.21:2181,192.168.5.225:2181" topic\_id=\>"info\_flow" } } fi… --- ## [Mysql connector jdbc driver not loaded](https://discuss.elastic.co/t/mysql-connector-jdbc-driver-not-loaded/134808)
**Author:** [@dmongit](https://discuss.elastic.co/u/dmongit)\ **Replies:** 1\ **Last updated:** [June 12, 2018, 12:01am UTC](https://discuss.elastic.co/t/mysql-connector-jdbc-driver-not-loaded/134808 "2018-06-12T00:01:33Z")
Hi, I see tons of posts related to this. But, I still have the same issue after running through several of the solutions in these posts. I used the full path for the jar, and it's not an empty file. I'm running windows… --- ## [Having trouble forwarding indexes from Elasticsearch to logstash](https://discuss.elastic.co/t/having-trouble-forwarding-indexes-from-elasticsearch-to-logstash/135395)
**Author:** [@StillReclaim](https://discuss.elastic.co/u/StillReclaim)\ **Replies:** 3\ **Last updated:** [June 11, 2018, 8:36pm UTC](https://discuss.elastic.co/t/having-trouble-forwarding-indexes-from-elasticsearch-to-logstash/135395 "2018-06-11T20:36:41Z")
Hello, I'm trying to send all information from one elasticsearch node to another in different clusters through logstash. My current inputs.conf is below. It sort of works, but it outputs indexes that are named: %{\[@meta… --- ## [Contructing JSON array output from logstash](https://discuss.elastic.co/t/contructing-json-array-output-from-logstash/135418)
**Author:** [@Hanciv](https://discuss.elastic.co/u/Hanciv)\ **Replies:** 4\ **Last updated:** [June 11, 2018, 7:55pm UTC](https://discuss.elastic.co/t/contructing-json-array-output-from-logstash/135418 "2018-06-11T19:55:31Z")
Hi, I am reading a log file where I have key value paired events. I extract the key value fields using KV filter and then construct a JSON manually like the below. "\[application\]\[environment\]" =\> "%{environment}" "\[ap… --- ## [Sometimes "Ruby exception occurred: no implicit conversion of nil into String"](https://discuss.elastic.co/t/sometimes-ruby-exception-occurred-no-implicit-conversion-of-nil-into-string/135401)
**Author:** [@vb4t](https://discuss.elastic.co/u/vb4t)\ **Replies:** 14\ **Last updated:** [June 11, 2018, 7:40pm UTC](https://discuss.elastic.co/t/sometimes-ruby-exception-occurred-no-implicit-conversion-of-nil-into-string/135401 "2018-06-11T19:40:10Z")
Hello, I use following ruby code in ruby filter to remove empty fields and fields containing "-" from fields nested in "parentfield" . The code works 99% time well, but sometimes I get ruby exception "Ruby exception occ… --- ## [How to drop "](https://discuss.elastic.co/t/how-to-drop/135410)
**Author:** [@sysdradmin](https://discuss.elastic.co/u/sysdradmin)\ **Replies:** 4\ **Last updated:** [June 11, 2018, 6:26pm UTC](https://discuss.elastic.co/t/how-to-drop/135410 "2018-06-11T18:26:55Z")
hello, We have an error when parsing with logstash : /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385\] csv - Error parsing csv {:field=\>"message", :source=\>"6 ...., hello "word",,,,,,,,,,,,, " :exception=\>… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=741) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=743)