# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=838 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 839 --- ## [How to Change Date Format in Logstash](https://discuss.elastic.co/t/how-to-change-date-format-in-logstash/115603)
**Author:** [@Saman.KH](https://discuss.elastic.co/u/Saman.KH)\ **Replies:** 7\ **Last updated:** [January 19, 2018, 7:48am UTC](https://discuss.elastic.co/t/how-to-change-date-format-in-logstash/115603 "2018-01-19T07:48:13Z")
I use ELK 6.1.1 and File beat to send some applications log files to logstash. Logstash automatically, try to changing some of my fields type to Date and this change is not suitable for me. I need just numbers in date… --- ## [Logstash: max\_bytes\_length\_exceeded exception](https://discuss.elastic.co/t/logstash-max-bytes-length-exceeded-exception/115874)
**Author:** [@annaK](https://discuss.elastic.co/u/annaK)\ **Replies:** 3\ **Last updated:** [January 19, 2018, 7:42am UTC](https://discuss.elastic.co/t/logstash-max-bytes-length-exceeded-exception/115874 "2018-01-19T07:42:43Z")
Hi all, I have a short question, which might be very stupid indeed. I have a document that got 34321 byte of lenght. So I get a response like response=\>{"index"=\>{"\_index"=\>"kafka-test-car-2018.01.15", "\_type"=\>"logs"… --- ## [Ruby filter - how to access JSON fields when using the json codec](https://discuss.elastic.co/t/ruby-filter-how-to-access-json-fields-when-using-the-json-codec/116071)
**Author:** [@anon24174512](https://discuss.elastic.co/u/anon24174512)\ **Replies:** 5\ **Last updated:** [January 19, 2018, 6:46am UTC](https://discuss.elastic.co/t/ruby-filter-how-to-access-json-fields-when-using-the-json-codec/116071 "2018-01-19T06:46:14Z")
Dear all, I have two payload.longitude and payload.latitude fields I can display in Kibana. There type in the elasticsearch mapping is float. I want to divide them by 1000000 and set the values of the corresponding lat… --- ## [JSON codec creating unreadable data](https://discuss.elastic.co/t/json-codec-creating-unreadable-data/116149)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [January 19, 2018, 6:41am UTC](https://discuss.elastic.co/t/json-codec-creating-unreadable-data/116149 "2018-01-19T06:41:41Z")
Hi, For whatever reason the JSON codec suddenly is coding messages in such a way that only the first message in a file is readable. The forum doesn't allow uploading files so I can't attach the actual file but the file… --- ## [Getting NoMethodError: undefined method \`\<' for nil:NilClass](https://discuss.elastic.co/t/getting-nomethoderror-undefined-method-for-nil-nilclass/116167)
**Author:** [@vrael](https://discuss.elastic.co/u/vrael)\ **Replies:** 0\ **Last updated:** [January 19, 2018, 6:04am UTC](https://discuss.elastic.co/t/getting-nomethoderror-undefined-method-for-nil-nilclass/116167 "2018-01-19T06:04:55Z")
I'm trying to upload a csv file via logstash and end up with this error. Any Solutions? Here is my logstash.conf file: input { file { path =\> "/home/sarthak/Downloads/day.csv" start\_position =\> "beginning" sincedb\_… --- ## [JDBC update issue](https://discuss.elastic.co/t/jdbc-update-issue/116157)
**Author:** [@pavuk](https://discuss.elastic.co/u/pavuk)\ **Replies:** 0\ **Last updated:** [January 19, 2018, 4:23am UTC](https://discuss.elastic.co/t/jdbc-update-issue/116157 "2018-01-19T04:23:20Z")
Hi, At the moment I have following configuration input { jdbc { jdbc\_driver\_library =\> "/usr/share/java/mysql-connector-java.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" jdbc\_connection\_string =\> "jd… --- ## [Specify index when using elasticsearch logstash filter](https://discuss.elastic.co/t/specify-index-when-using-elasticsearch-logstash-filter/1928)
**Author:** [@lwintergerst](https://discuss.elastic.co/u/lwintergerst)\ **Replies:** 5\ **Last updated:** [January 19, 2018, 3:26am UTC](https://discuss.elastic.co/t/specify-index-when-using-elasticsearch-logstash-filter/1928 "2018-01-19T03:26:01Z")
Is it possible to specify an Index when I use the elasticsearch logstash filter? I can't possibly let it search through everything. My google searches did not bring me any further, since googling for something with the… --- ## [Time to timestamp issue](https://discuss.elastic.co/t/time-to-timestamp-issue/113455)
**Author:** [@uvarisubash](https://discuss.elastic.co/u/uvarisubash)\ **Replies:** 10\ **Last updated:** [January 18, 2018, 9:54pm UTC](https://discuss.elastic.co/t/time-to-timestamp-issue/113455 "2018-01-18T21:54:58Z")
Hi All, I use filter to extract the @time into a field of its own called "@time" message ==\> "@time": "2017-12-04T17:44:34" "@timestamp" =\> 2017-12-28T12:29:22.096Z, .conf file content input { file { path =\> "/hom… --- ## [Speed up the pipeline-2](https://discuss.elastic.co/t/speed-up-the-pipeline-2/115582)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 4\ **Last updated:** [January 18, 2018, 9:44pm UTC](https://discuss.elastic.co/t/speed-up-the-pipeline-2/115582 "2018-01-18T21:44:07Z")
I have set up of 3 machines: Single ES process setup with 20gb heap size and 16 processors a logstash process with 20gb heap size connecting to the ES on machine 1 filebeat pumping data to the logstash process on machi… --- ## [Multiline - pattern repeats in content](https://discuss.elastic.co/t/multiline-pattern-repeats-in-content/116113)
**Author:** [@vanian](https://discuss.elastic.co/u/vanian)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 6:01pm UTC](https://discuss.elastic.co/t/multiline-pattern-repeats-in-content/116113 "2018-01-18T18:01:17Z")
Hello, I'm using Logstash to parse some incident reports and attempting to use the multiline codec with my CSV files. The problem I'm running into is that in certain fields, the pattern that I'm using to determine a new … --- ## [JSON Logs Parsed with JSON codec but not JSON filter](https://discuss.elastic.co/t/json-logs-parsed-with-json-codec-but-not-json-filter/115186)
**Author:** [@cappy](https://discuss.elastic.co/u/cappy)\ **Replies:** 2\ **Last updated:** [January 18, 2018, 4:36pm UTC](https://discuss.elastic.co/t/json-logs-parsed-with-json-codec-but-not-json-filter/115186 "2018-01-18T16:36:20Z")
Hi, I'm running Filebeat and Logstash 5.6.5 and was having a bit of an issue with the json filter plugin provided for use with Logstash. I am pulling in multiline JSON logs with Filebeat, and I have an input configured… --- ## [Multiline Filter Help to find specific field](https://discuss.elastic.co/t/multiline-filter-help-to-find-specific-field/116090)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 1\ **Last updated:** [January 18, 2018, 3:48pm UTC](https://discuss.elastic.co/t/multiline-filter-help-to-find-specific-field/116090 "2018-01-18T15:48:50Z")
I have an xml file that I need to ingest with Logstash, with specific fields. It has structure: \< Test\> \< Type\> \< Data\> .......... .......... \< /Data\> \< /Type\> \< /Test\> I am trying to use multiline codec, in ord… --- ## [Miltiline filter pattern](https://discuss.elastic.co/t/miltiline-filter-pattern/116089)
**Author:** [@Telstra\_Aus](https://discuss.elastic.co/u/Telstra_Aus)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 3:27pm UTC](https://discuss.elastic.co/t/miltiline-filter-pattern/116089 "2018-01-18T15:27:53Z")
Hi, can you please help me with multiline filters, below is the exception that i need to capture \<\> \[Jan 17 17:04:24\] \[\[ACTIVE\] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'\] \[CM\] \[ABPBatchUser… --- ## [Getting "Pipeline aborted due to error" in tutorial pipeline & 401 Error](https://discuss.elastic.co/t/getting-pipeline-aborted-due-to-error-in-tutorial-pipeline-401-error/115436)
**Author:** [@tzure](https://discuss.elastic.co/u/tzure)\ **Replies:** 3\ **Last updated:** [January 15, 2018, 9:00am UTC](https://discuss.elastic.co/t/getting-pipeline-aborted-due-to-error-in-tutorial-pipeline-401-error/115436 "2018-01-15T09:00:16Z")
Update at the next msg - found out it's receiving a msg of 401 error. Hi, I'm using the tutorial pipeline here and everything was okay, until I got to the part where I wanted to connect my pipeline to elastic search. … --- ## [Grok Filtering - Two Groks One Feed](https://discuss.elastic.co/t/grok-filtering-two-groks-one-feed/115915)
**Author:** [@Stormdizzle](https://discuss.elastic.co/u/Stormdizzle)\ **Replies:** 5\ **Last updated:** [January 18, 2018, 1:59pm UTC](https://discuss.elastic.co/t/grok-filtering-two-groks-one-feed/115915 "2018-01-18T13:59:02Z")
I am trying to figure out how to make the Grok filtering work properly, but it appears that I am not having such luck. I looked at other documents but I can't seem to figure it out so hopefully someone smarter than me c… --- ## [How to rename the field “event\_data.TargetUserName”](https://discuss.elastic.co/t/how-to-rename-the-field-event-data-targetusername/116064)
**Author:** [@camelliuxu](https://discuss.elastic.co/u/camelliuxu)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 1:36pm UTC](https://discuss.elastic.co/t/how-to-rename-the-field-event-data-targetusername/116064 "2018-01-18T13:36:57Z")
Hello everyone, I am using the winlogbeat in Windows Server security event analysis, a hash field "event\_data" was generated, now I want to rename the "TargetUserName" field in the "event\_data" field using logstash filt… --- ## [Sincedb not written when EOF not reached yet](https://discuss.elastic.co/t/sincedb-not-written-when-eof-not-reached-yet/116059)
**Author:** [@padakwaak](https://discuss.elastic.co/u/padakwaak)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 1:13pm UTC](https://discuss.elastic.co/t/sincedb-not-written-when-eof-not-reached-yet/116059 "2018-01-18T13:13:40Z")
Hi, I'm using LS 6.0.0 and ES 6.0.0 and my SinceDB files are only created when Logstash reaches the end of the file. In my use case my log files are about 10GB each and I've configured the file input to scan from the "b… --- ## [Filter Ruby configuration Error](https://discuss.elastic.co/t/filter-ruby-configuration-error/115996)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 1\ **Last updated:** [January 18, 2018, 11:27am UTC](https://discuss.elastic.co/t/filter-ruby-configuration-error/115996 "2018-01-18T11:27:31Z")
I have data that is coming in via Logstash with ActiveMQ bus that looks like this: "DataType1" =\> { "system:subsystem123" =\> \[ \[0\] {"field": "test}\], "system:subsystem234" =\> \[ \[0\] { "field": "test"}\], "system:subs… --- ## [Multiple Pipelines and stdout](https://discuss.elastic.co/t/multiple-pipelines-and-stdout/116026)
**Author:** [@MarkusT](https://discuss.elastic.co/u/MarkusT)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 11:01am UTC](https://discuss.elastic.co/t/multiple-pipelines-and-stdout/116026 "2018-01-18T11:01:18Z")
Hey there I'm dealing with the following problem: Í have multiple pipelines defined, in each config file there is a rubydebug output to the console (stdout). When I run those in a multiple Pipeline, none is shown in con… --- ## [Logstash outputs kafka - KafkaProducer.send() failed](https://discuss.elastic.co/t/logstash-outputs-kafka-kafkaproducer-send-failed/115696)
**Author:** [@elkuserh](https://discuss.elastic.co/u/elkuserh)\ **Replies:** 1\ **Last updated:** [January 18, 2018, 10:03am UTC](https://discuss.elastic.co/t/logstash-outputs-kafka-kafkaproducer-send-failed/115696 "2018-01-18T10:03:52Z")
We have a logstash config, which is used to send data to a Kafka broker. The logstash config is as follows: input { … --- ## [\[Logstash jms input\] Unable to connect to ActiveMQ topic](https://discuss.elastic.co/t/logstash-jms-input-unable-to-connect-to-activemq-topic/115170)
**Author:** [@Pavel\_Sapozhnikov](https://discuss.elastic.co/u/Pavel_Sapozhnikov)\ **Replies:** 3\ **Last updated:** [January 18, 2018, 9:53am UTC](https://discuss.elastic.co/t/logstash-jms-input-unable-to-connect-to-activemq-topic/115170 "2018-01-18T09:53:53Z")
Hi I am trying to connect to an ActiveMQ topic using logstash-jms-input plugin and I am unable to. When I run logstash it will create a queue with the name that I give it in destination but never a topic. Here are my co… --- ## [Conditional Output to http not working](https://discuss.elastic.co/t/conditional-output-to-http-not-working/115956)
**Author:** [@premkumar\_narashiman](https://discuss.elastic.co/u/premkumar_narashiman)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 11:27pm UTC](https://discuss.elastic.co/t/conditional-output-to-http-not-working/115956 "2018-01-17T23:27:58Z")
Conditional Output to stdout or http not working consistently. Only one condition works and behavior seems to tied to input file that's getting updated first. Does anyone see any issue with the configuration? please pro… --- ## [Logstash not sending data to ElasticSearch - multiple outputs based on tags](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch-multiple-outputs-based-on-tags/116002)
**Author:** [@Milad\_Hamid](https://discuss.elastic.co/u/Milad_Hamid)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 8:53am UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch-multiple-outputs-based-on-tags/116002 "2018-01-18T08:53:08Z")
Hi, My host shipping logs via filebeat 6.1.1 to our logstash 6.1.1 server, and i can see the parsed logs on the screen by using stdout { codec =\> rubydebug }. But logs data not sending to Elasticsearch 6.1.1 In my logs… --- ## [Remove HTML markup from logstash input](https://discuss.elastic.co/t/remove-html-markup-from-logstash-input/116003)
**Author:** [@Aleksander\_de](https://discuss.elastic.co/u/Aleksander_de)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 8:57am UTC](https://discuss.elastic.co/t/remove-html-markup-from-logstash-input/116003 "2018-01-18T08:57:24Z")
Hi All! I am struggling with data manipulation in logstash version 5.1, where some of the data come to me from open text fields with HTML options. Most of the time it comes with only one marker \> \ XYZ \ and I am… --- ## [Logstash only reading first JSON line](https://discuss.elastic.co/t/logstash-only-reading-first-json-line/115997)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [January 18, 2018, 8:34am UTC](https://discuss.elastic.co/t/logstash-only-reading-first-json-line/115997 "2018-01-18T08:34:49Z")
Hi, Suddenly my S3 input stopped processing complete files. This morning the same files in the same bucket were processed just fine, but after deleting some files, the remaining files suddenly only have the first JSON l… --- ## [Parsing and combining different CSV files in logstash](https://discuss.elastic.co/t/parsing-and-combining-different-csv-files-in-logstash/113587)
**Author:** [@anchasis](https://discuss.elastic.co/u/anchasis)\ **Replies:** 11\ **Last updated:** [January 18, 2018, 7:44am UTC](https://discuss.elastic.co/t/parsing-and-combining-different-csv-files-in-logstash/113587 "2018-01-18T07:44:52Z")
Hi, I am looking for some pointers on how to best tackle this problem. I have two csv files, roughly with 20 columns each. One of the CSV files contains a field (hostname) that is part of a field of the second csv file … --- ## [Logstash missing events - monitoring ftp logs](https://discuss.elastic.co/t/logstash-missing-events-monitoring-ftp-logs/115362)
**Author:** [@premkumar\_narashiman](https://discuss.elastic.co/u/premkumar_narashiman)\ **Replies:** 3\ **Last updated:** [January 17, 2018, 11:31pm UTC](https://discuss.elastic.co/t/logstash-missing-events-monitoring-ftp-logs/115362 "2018-01-17T23:31:11Z")
I am an Newbie to ELK using to stash for processing ftp logs (vsftpd, sftp). I am facing missing event issue with Logstash version: 6.1.0, Test case: uploaded 15 files using ftp - only 7 lines logged in logstash-output… --- ## [Having issue with DNS filter](https://discuss.elastic.co/t/having-issue-with-dns-filter/115940)
**Author:** [@amb1s1](https://discuss.elastic.co/u/amb1s1)\ **Replies:** 7\ **Last updated:** [January 17, 2018, 10:34pm UTC](https://discuss.elastic.co/t/having-issue-with-dns-filter/115940 "2018-01-17T22:34:56Z")
I'm using logstash as a NetFlow collector, and it is working well for a start. I'm getting all the flow, and I want to use filters to be able to resolve ips. I create a simple filter (my first one, but it is not working… --- ## [Logstash Unable to maintain connection to ES](https://discuss.elastic.co/t/logstash-unable-to-maintain-connection-to-es/114974)
**Author:** [@Austin\_E](https://discuss.elastic.co/u/Austin_E)\ **Replies:** 7\ **Last updated:** [January 17, 2018, 9:18pm UTC](https://discuss.elastic.co/t/logstash-unable-to-maintain-connection-to-es/114974 "2018-01-17T21:18:46Z")
I setup a stock ES node and a stock Logstash node on the same physical machine(myelastic). I've made 7 changes to the node (node name, cluster name, host, logpath, datapath, locking memory, 31gb of RAM). Kibana can work … --- ## [Filebeat and Logstash with log rotation](https://discuss.elastic.co/t/filebeat-and-logstash-with-log-rotation/115866)
**Author:** [@uliana\_andreeva](https://discuss.elastic.co/u/uliana_andreeva)\ **Replies:** 1\ **Last updated:** [January 17, 2018, 8:42pm UTC](https://discuss.elastic.co/t/filebeat-and-logstash-with-log-rotation/115866 "2018-01-17T20:42:00Z")
Hi there! I'm totally new in ELK framework. I use Filebeat on CentOs 7 to ship logs to Logstash, here is my filebeat.yml: filebeat.prospectors: - type: log paths: - /opt/tomcat/logs/calendar/log.log … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=837) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=839)