# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=684 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 685 --- ## [Big Data to Elastic with Logstash](https://discuss.elastic.co/t/big-data-to-elastic-with-logstash/148836)
**Author:** [@Olha1](https://discuss.elastic.co/u/Olha1)\ **Replies:** 3\ **Last updated:** [September 18, 2018, 9:35am UTC](https://discuss.elastic.co/t/big-data-to-elastic-with-logstash/148836 "2018-09-18T09:35:08Z")
Hi! I trying to upload Big Data in elasticsearch, but always I have java exception, if I dont have LIMIT in SQL-query. How I can to upload Big Data with jdbs plugins(I expanded alowable memory size, but it did not hel… --- ## [Is it possible to parse and store a field as an array of objects in Elasticsearch?](https://discuss.elastic.co/t/is-it-possible-to-parse-and-store-a-field-as-an-array-of-objects-in-elasticsearch/148566)
**Author:** [@anra137](https://discuss.elastic.co/u/anra137)\ **Replies:** 2\ **Last updated:** [September 18, 2018, 7:54am UTC](https://discuss.elastic.co/t/is-it-possible-to-parse-and-store-a-field-as-an-array-of-objects-in-elasticsearch/148566 "2018-09-18T07:54:03Z")
My logline would look like this 2018-03-09 16:12:20,315 INFO \[com.sample.teste.TestClass\] (https-express.sample.com-433-8) {my-name-1=my-value-1, my-name-2=my-value-2} Testing logger And I have pipeline config as filt… --- ## [Terribly long time to load data from MySQL](https://discuss.elastic.co/t/terribly-long-time-to-load-data-from-mysql/148627)
**Author:** [@Gosforth](https://discuss.elastic.co/u/Gosforth)\ **Replies:** 9\ **Last updated:** [September 18, 2018, 6:33am UTC](https://discuss.elastic.co/t/terribly-long-time-to-load-data-from-mysql/148627 "2018-09-18T06:33:32Z")
It takes two seconds to display data from view in MySQL. But it takes hour (I still wait so I cannot tell it will succeed) logstash starts to load data from this view (since running the command logstash -f config) Konfi… --- ## [Slow startup and no data transmit on recursive directory](https://discuss.elastic.co/t/slow-startup-and-no-data-transmit-on-recursive-directory/147968)
**Author:** [@ali\_tavakoli](https://discuss.elastic.co/u/ali_tavakoli)\ **Replies:** 2\ **Last updated:** [September 18, 2018, 6:07am UTC](https://discuss.elastic.co/t/slow-startup-and-no-data-transmit-on-recursive-directory/147968 "2018-09-18T06:07:01Z")
Hi guys I configured my logstash to input plugin from file and send to elasticsearch. my file directory is recursive and number of file is about 500k in recursive directories and each file is average 2 MB. logstash sta… --- ## [Existing mapping for \[field name\] must be of type object but found \[field type\]](https://discuss.elastic.co/t/existing-mapping-for-field-name-must-be-of-type-object-but-found-field-type/148411)
**Author:** [@peterch](https://discuss.elastic.co/u/peterch)\ **Replies:** 4\ **Last updated:** [September 18, 2018, 2:04am UTC](https://discuss.elastic.co/t/existing-mapping-for-field-name-must-be-of-type-object-but-found-field-type/148411 "2018-09-18T02:04:10Z")
Dear All, I split a list of data into different events. Here's the example data data source 1.178.179.217 1.179.170.7 1.93.0.224 100.16.243.115 101.187.28.8 After split to different event, i put a template to c… --- ## [If/CSV Filter not working](https://discuss.elastic.co/t/if-csv-filter-not-working/148879)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 8:47pm UTC](https://discuss.elastic.co/t/if-csv-filter-not-working/148879 "2018-09-17T20:47:14Z")
Running the below pipeline and my events are not being split into fields and the tag isn't being added. I would have to assume there is something wrong with my if statement but I'm not sure what. The tag Exchange is ad… --- ## [De-duplicating with MURMUR3 vs SHA256](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262)
**Author:** [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Replies:** 3\ **Last updated:** [September 17, 2018, 7:24pm UTC](https://discuss.elastic.co/t/de-duplicating-with-murmur3-vs-sha256/147262 "2018-09-17T19:24:39Z")
Hi, has anyone used the fingerprint plugin with MURMUR3? So far I find it has quite high collision rate. Even with just a few hundred thousands records managed to get 20 collisions. Testing with sha256 over 2million rec… --- ## [MessageSizeTooLargeException](https://discuss.elastic.co/t/messagesizetoolargeexception/146894)
**Author:** [@ccoqueiro](https://discuss.elastic.co/u/ccoqueiro)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 7:15pm UTC](https://discuss.elastic.co/t/messagesizetoolargeexception/146894 "2018-09-17T19:15:38Z")
Hi All, I'm new here and sorry if put this topic in wrong path. So, I treid to use logstash plugin version 2.3.4, in this case, the logstash connect in a topic in the kafka and sendo the output for other element, but, w… --- ## [EBADF: Bad file descriptor](https://discuss.elastic.co/t/ebadf-bad-file-descriptor/148666)
**Author:** [@kilo](https://discuss.elastic.co/u/kilo)\ **Replies:** 4\ **Last updated:** [September 17, 2018, 6:50pm UTC](https://discuss.elastic.co/t/ebadf-bad-file-descriptor/148666 "2018-09-17T18:50:01Z")
Logstash is throwing the following error: \[2018-09-14T13:53:47,371\]\[WARN \]\[logstash.inputs.websocket\] websocket input client threw exception, restarting {:exception=\>#\ **Author:** [@jamiebuxxx](https://discuss.elastic.co/u/jamiebuxxx)\ **Replies:** 0\ **Last updated:** [September 17, 2018, 6:16pm UTC](https://discuss.elastic.co/t/sending-batch-to-kafka-failed-retry-loop-error/148871 "2018-09-17T18:16:45Z") Hello, I recently updated our log-brokers to use Logstash 5.6.11. This version is currently using the logstash-output-kafka plugin version 5.1.11. We have 3 log brokers and we recently noticed connections from Kafka dro… --- ## [Managing pipeline](https://discuss.elastic.co/t/managing-pipeline/148852)
**Author:** [@riteshun](https://discuss.elastic.co/u/riteshun)\ **Replies:** 2\ **Last updated:** [September 17, 2018, 5:16pm UTC](https://discuss.elastic.co/t/managing-pipeline/148852 "2018-09-17T17:16:45Z")
Hi, We have pipelines that are managed from console. How can we see the history when changes were made to them and what all changed. Thanks --- ## [Parsing an array of coordinate values](https://discuss.elastic.co/t/parsing-an-array-of-coordinate-values/147669)
**Author:** [@waynesmallman](https://discuss.elastic.co/u/waynesmallman)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 5:04pm UTC](https://discuss.elastic.co/t/parsing-an-array-of-coordinate-values/147669 "2018-09-17T17:04:17Z")
Hi, we're attempting to parse coordinate data: "\[33.5586, -112.0955\]" but we don't know if it needs to be converted, and if so, what type should we use? So far, we've tried string, but it's as if Logstash is attempting … --- ## [Logstash to parse TSV logs](https://discuss.elastic.co/t/logstash-to-parse-tsv-logs/147517)
**Author:** [@apurva](https://discuss.elastic.co/u/apurva)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 4:59pm UTC](https://discuss.elastic.co/t/logstash-to-parse-tsv-logs/147517 "2018-09-17T16:59:36Z")
Need to parse TSV logs via logstash 6.3.2. Need to understand possibilities for same. 1 is to write a separate exe which converts tsv into csv, parse csv via logstash. Can we write any program in logstash itself to rea… --- ## [File not loaded entirely using stdin plugin](https://discuss.elastic.co/t/file-not-loaded-entirely-using-stdin-plugin/148645)
**Author:** [@\_kml](https://discuss.elastic.co/u/_kml)\ **Replies:** 2\ **Last updated:** [September 15, 2018, 9:20pm UTC](https://discuss.elastic.co/t/file-not-loaded-entirely-using-stdin-plugin/148645 "2018-09-15T21:20:22Z")
I am using stdin plugin to load a file of about 2500 events, parse it and then write it to elasticsearch, so what I have is: $ cat file | ./bin/logstash -f config/myconfig.conf but what I have is that only about 600 ev… --- ## [How to extract data from this pattern, with grok filter its not working](https://discuss.elastic.co/t/how-to-extract-data-from-this-pattern-with-grok-filter-its-not-working/148486)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 4:50pm UTC](https://discuss.elastic.co/t/how-to-extract-data-from-this-pattern-with-grok-filter-its-not-working/148486 "2018-09-17T16:50:46Z")
How to extract data from this pattern, with grok filter its not working below data is available in log file logEvent": { "message": "JDBC - Exception. Not retrying. event: "{"LastModifiedDate":"2018-09-12T18:19:15.00… --- ## [How to recognize and separate different syslogs](https://discuss.elastic.co/t/how-to-recognize-and-separate-different-syslogs/148189)
**Author:** [@annk](https://discuss.elastic.co/u/annk)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 4:45pm UTC](https://discuss.elastic.co/t/how-to-recognize-and-separate-different-syslogs/148189 "2018-09-17T16:45:18Z")
Hi, I have couple of applications sending logs on to logstash on same port using rsyslog. Is it possible to identify them on logstash syslog input using $InputFileTag or any other tags that I can configure on rsyslog c… --- ## [How to handle json-formated data in input channel in connection with multiline?](https://discuss.elastic.co/t/how-to-handle-json-formated-data-in-input-channel-in-connection-with-multiline/148464)
**Author:** [@elk\_kl](https://discuss.elastic.co/u/elk_kl)\ **Replies:** 4\ **Last updated:** [September 17, 2018, 3:43pm UTC](https://discuss.elastic.co/t/how-to-handle-json-formated-data-in-input-channel-in-connection-with-multiline/148464 "2018-09-17T15:43:27Z")
Hi all, I am currently trying to apply the multiline codec (with particular regards to Java stack traces) in the input tcp stream in logstash on events (logs) that are provided to logstash by the logspout-logstash tool … --- ## [Problem With timestamp and timezone](https://discuss.elastic.co/t/problem-with-timestamp-and-timezone/148622)
**Author:** [@Tiago\_Pinto](https://discuss.elastic.co/u/Tiago_Pinto)\ **Replies:** 3\ **Last updated:** [September 17, 2018, 2:55pm UTC](https://discuss.elastic.co/t/problem-with-timestamp-and-timezone/148622 "2018-09-17T14:55:56Z")
Hello all, I'm running into some problems when I'm trying to enrich data using input jdbc and filter jdbc\_streaming. This is my conf file input { jdbc { jdbc\_connection\_string =\> "connectionString" jdbc\_user =\> … --- ## [Filebeat don't send data to Logstash : Invalid Frame Type, received: 84](https://discuss.elastic.co/t/filebeat-dont-send-data-to-logstash-invalid-frame-type-received-84/148838)
**Author:** [@MLdz](https://discuss.elastic.co/u/MLdz)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 12:57pm UTC](https://discuss.elastic.co/t/filebeat-dont-send-data-to-logstash-invalid-frame-type-received-84/148838 "2018-09-17T12:57:25Z")
Hello, i have a problems with the connexion between filebeat and logstash, i have this error and i dont arrive t send logs from filebeat.... The version : filebeat version 6.4.0 (amd64), libbeat 6.4.0 logstash 6.2.3,… --- ## [Winlogbeat to Logstash = Config file help required](https://discuss.elastic.co/t/winlogbeat-to-logstash-config-file-help-required/148828)
**Author:** [@JKCode](https://discuss.elastic.co/u/JKCode)\ **Replies:** 0\ **Last updated:** [September 17, 2018, 12:02pm UTC](https://discuss.elastic.co/t/winlogbeat-to-logstash-config-file-help-required/148828 "2018-09-17T12:02:24Z")
Hi All, I am trying to set up Logstash to receive Winlogbeat input with Logstash config file with conditional redirect of output to specific elasticsearch indexing. Getting Configuration error; complete details below = … --- ## [Log data entered wrong index](https://discuss.elastic.co/t/log-data-entered-wrong-index/148748)
**Author:** [@wonki](https://discuss.elastic.co/u/wonki)\ **Replies:** 2\ **Last updated:** [September 17, 2018, 10:08am UTC](https://discuss.elastic.co/t/log-data-entered-wrong-index/148748 "2018-09-17T10:08:26Z")
Hi everyone! 2018-09-17 08:59:59.501 \<- this is my log time and i set my logstash elasticsearch output plugin elasticsearch{ host =\> \["127.0.0.1:9200"\] index =\> "log-%{+YYYY.MM.dd}" } So as expected, my log is inc… --- ## [Grok not breaking on matching first pattern](https://discuss.elastic.co/t/grok-not-breaking-on-matching-first-pattern/148794)
**Author:** [@Varun\_Goyal](https://discuss.elastic.co/u/Varun_Goyal)\ **Replies:** 0\ **Last updated:** [September 17, 2018, 9:01am UTC](https://discuss.elastic.co/t/grok-not-breaking-on-matching-first-pattern/148794 "2018-09-17T09:01:56Z")
I am using grok to extract first line data. For that i tried GREEDYDATA, also tried \*. but it seems it matches last entity. My input has multiline codec. Input Field: Content installed successfully on the client\\r\\n… --- ## [Kibana alert issue](https://discuss.elastic.co/t/kibana-alert-issue/148743)
**Author:** [@jeongsyhd](https://discuss.elastic.co/u/jeongsyhd)\ **Replies:** 1\ **Last updated:** [September 17, 2018, 6:37am UTC](https://discuss.elastic.co/t/kibana-alert-issue/148743 "2018-09-17T06:37:17Z")
There was an issue in kibana. Why does it happen below? Image 1. The following screen will appear when accessing the web page or when refreshing. I have not set a login / password for kibana. Image 2 Image 3 I… --- ## [Q. grok filter pattern](https://discuss.elastic.co/t/q-grok-filter-pattern/148731)
**Author:** [@jeongsyhd](https://discuss.elastic.co/u/jeongsyhd)\ **Replies:** 7\ **Last updated:** [September 16, 2018, 4:45pm UTC](https://discuss.elastic.co/t/q-grok-filter-pattern/148731 "2018-09-16T16:45:26Z")
I created a filter as shown below. There are three types of logs, so there are three patterns. log A can use the message if "action" in \[message\]. But what about log B and log C ? log A: sip = 1.1.1.1 dip = 2.2.2.2 s… --- ## [Logstash failed to create queue](https://discuss.elastic.co/t/logstash-failed-to-create-queue/146852)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 0\ **Last updated:** [August 31, 2018, 11:05am UTC](https://discuss.elastic.co/t/logstash-failed-to-create-queue/146852 "2018-08-31T11:05:36Z")
Today, I find out, my queue is full, and my logstash suddenly generate this error: \[2018-08-31T12:56:13,494\]\[ERROR\]\[logstash.pipeline \] Logstash failed to create queue {:pipeline\_id=\>"default-pipeline", "exceptio… --- ## [Use grok to skip seperators lines](https://discuss.elastic.co/t/use-grok-to-skip-seperators-lines/148672)
**Author:** [@Talal\_Elammas](https://discuss.elastic.co/u/Talal_Elammas)\ **Replies:** 1\ **Last updated:** [September 16, 2018, 5:27am UTC](https://discuss.elastic.co/t/use-grok-to-skip-seperators-lines/148672 "2018-09-16T05:27:55Z")
Hello, I have a log file structured as 2 line separators then the log data , I created my parser for the log data , but I don't know how to eliminate those two lines from being proceed through the filter, please for… --- ## [How to use custom fields from filebeat in logstash?](https://discuss.elastic.co/t/how-to-use-custom-fields-from-filebeat-in-logstash/148708)
**Author:** [@AlonKerem](https://discuss.elastic.co/u/AlonKerem)\ **Replies:** 1\ **Last updated:** [September 16, 2018, 5:24am UTC](https://discuss.elastic.co/t/how-to-use-custom-fields-from-filebeat-in-logstash/148708 "2018-09-16T05:24:42Z")
hi everyone, new to logstash (about 100 hours in), i have 2 servers that are running filebeat, one of them is called x and the other is y, in the configuration of x there is this (filebeat.yml - only releveant lines ar… --- ## [Codec =\> rubydebug = required in production environment?](https://discuss.elastic.co/t/codec-rubydebug-required-in-production-environment/148727)
**Author:** [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Replies:** 1\ **Last updated:** [September 16, 2018, 4:41am UTC](https://discuss.elastic.co/t/codec-rubydebug-required-in-production-environment/148727 "2018-09-16T04:41:14Z")
Once the configuration and stable, Do we still need this in the config ? stdout { codec =\> rubydebug } Hope this going to consume resources and space in large log environment ? --- ## [Multi type of messages from once log source to logstash](https://discuss.elastic.co/t/multi-type-of-messages-from-once-log-source-to-logstash/148693)
**Author:** [@Ninh\_Ha](https://discuss.elastic.co/u/Ninh_Ha)\ **Replies:** 1\ **Last updated:** [September 15, 2018, 8:39pm UTC](https://discuss.elastic.co/t/multi-type-of-messages-from-once-log-source-to-logstash/148693 "2018-09-15T20:39:08Z")
I have a device have multiple type of logs (difference failed for each type) =\> i want to count number of failed??? for filter of logs, Please help me! --- ## [How to replace grok pattern with new name](https://discuss.elastic.co/t/how-to-replace-grok-pattern-with-new-name/148715)
**Author:** [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Replies:** 4\ **Last updated:** [September 15, 2018, 8:09pm UTC](https://discuss.elastic.co/t/how-to-replace-grok-pattern-with-new-name/148715 "2018-09-15T20:09:08Z")
below example -- May be i have given wrong subject : but what iam looking example as below : First example is my netscaler log and second one is switch log Switch log syslog message ok. But netscalar log i get as net… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=683) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=685)