# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1108 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1109 --- ## [Import data from MySQL to Elasticsearch](https://discuss.elastic.co/t/import-data-from-mysql-to-elasticsearch/59402)
**Author:** [@amran1](https://discuss.elastic.co/u/amran1)\ **Replies:** 1\ **Last updated:** [August 31, 2016, 8:19am UTC](https://discuss.elastic.co/t/import-data-from-mysql-to-elasticsearch/59402 "2016-08-31T08:19:21Z")
Hello, How to import data from MySQL to Elasticsearch via Logstash... how to configure... thank you --- ## [How to Configure S3 input plugin?](https://discuss.elastic.co/t/how-to-configure-s3-input-plugin/59371)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 2\ **Last updated:** [August 31, 2016, 6:54am UTC](https://discuss.elastic.co/t/how-to-configure-s3-input-plugin/59371 "2016-08-31T06:54:50Z")
Hi I am trying to connect the S3 bucket to my logstash, since i have stored log files on S3. Would you pls suggest what i am missing? Below is my input filter: input { s3 { bucket =\> "purchaselogs" access\_key… --- ## [How the use the multiline in these situation?](https://discuss.elastic.co/t/how-the-use-the-multiline-in-these-situation/59388)
**Author:** [@LazyerHong](https://discuss.elastic.co/u/LazyerHong)\ **Replies:** 2\ **Last updated:** [August 31, 2016, 6:26am UTC](https://discuss.elastic.co/t/how-the-use-the-multiline-in-these-situation/59388 "2016-08-31T06:26:35Z")
There is logs: 2016-08-30 04:10:02,104\[INFO\] HttpRequest\[191.1.3.81:33983:null\] 2016-08-30 04:10:02,107\[INFO\] 用户\[10253\]，执行\[monitor\]前状态为：1100000000 \[com.cairh.xpe.context.user.interceptor.FlowEngineInterceptor.preHandle(F… --- ## [How to use the multiline in for these logs](https://discuss.elastic.co/t/how-to-use-the-multiline-in-for-these-logs/59391)
**Author:** [@LazyerHong](https://discuss.elastic.co/u/LazyerHong)\ **Replies:** 1\ **Last updated:** [August 31, 2016, 6:15am UTC](https://discuss.elastic.co/t/how-to-use-the-multiline-in-for-these-logs/59391 "2016-08-31T06:15:08Z")
For example i have logs here: 1111 2222 3333 1111 2222 3333 now i want to combine the log with the same number just like 11111111 22222222 33333333 how can i do with the multiline???It is very urgent for me now. Th… --- ## [Accessing specific fields from tree structure](https://discuss.elastic.co/t/accessing-specific-fields-from-tree-structure/59297)
**Author:** [@Primeval](https://discuss.elastic.co/u/Primeval)\ **Replies:** 2\ **Last updated:** [August 31, 2016, 5:47am UTC](https://discuss.elastic.co/t/accessing-specific-fields-from-tree-structure/59297 "2016-08-31T05:47:38Z")
I have dictionary in yaml that looks like this: - A001: - {B001: {C001: 10s, C002: 15s}} - {B002: {C003: 20s, C004: 25s}} i want to access specific value, for example when i have A001,B001,C001 in my logs i wan… --- ## [Logstash redis input thread](https://discuss.elastic.co/t/logstash-redis-input-thread/59278)
**Author:** [@zpp](https://discuss.elastic.co/u/zpp)\ **Replies:** 2\ **Last updated:** [August 31, 2016, 1:03am UTC](https://discuss.elastic.co/t/logstash-redis-input-thread/59278 "2016-08-31T01:03:05Z")
Referring to this article which talks about how to increase throughput by adjusting the redis input thread and batch size parameters, will this multi-threading (i.e. if I configure thread to \>1) affect the functionality… --- ## [Logstash S3 output gzip empty files](https://discuss.elastic.co/t/logstash-s3-output-gzip-empty-files/59373)
**Author:** [@lrem006](https://discuss.elastic.co/u/lrem006)\ **Replies:** 0\ **Last updated:** [August 30, 2016, 11:41pm UTC](https://discuss.elastic.co/t/logstash-s3-output-gzip-empty-files/59373 "2016-08-30T23:41:06Z")
I was so excited to see that file compression is available on the release on github! https://github.com/logstash-plugins/logstash-output-s3 I tried it and noticed that I was getting empty files with a size of 20 byte… --- ## [Logstash to ES over HTTPS](https://discuss.elastic.co/t/logstash-to-es-over-https/59298)
**Author:** [@Ravi\_Shanker\_Reddy](https://discuss.elastic.co/u/Ravi_Shanker_Reddy)\ **Replies:** 1\ **Last updated:** [August 30, 2016, 11:04pm UTC](https://discuss.elastic.co/t/logstash-to-es-over-https/59298 "2016-08-30T23:04:26Z")
In the ES docs I saw that the logstash connects to ES over HTTP. https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html This output only speaks the HTTP protocol. HTTP is the preferred prot… --- ## [String to Date, besides default system @timestamp](https://discuss.elastic.co/t/string-to-date-besides-default-system-timestamp/59269)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 2\ **Last updated:** [August 30, 2016, 10:38pm UTC](https://discuss.elastic.co/t/string-to-date-besides-default-system-timestamp/59269 "2016-08-30T22:38:51Z")
Hi I want to convert a String type field to Date type field, while keeping the system auto-generated @timestamp. The example log is below. Any ideas pls? Mar 1 00:02:14 eqx-astockweb1 adobestock\[59746\]: {"app\_id":"as"… --- ## [S3 output plugin not releasing file handles fast enough](https://discuss.elastic.co/t/s3-output-plugin-not-releasing-file-handles-fast-enough/59364)
**Author:** [@JoelS](https://discuss.elastic.co/u/JoelS)\ **Replies:** 0\ **Last updated:** [August 30, 2016, 9:57pm UTC](https://discuss.elastic.co/t/s3-output-plugin-not-releasing-file-handles-fast-enough/59364 "2016-08-30T21:57:23Z")
Log activity recently picked up on my system, and now it is holding on to several gigs of space even after it has uploaded the logfile to S3 and deleted it from the system. I have my configuration set up to rotate the … --- ## [Logstash to process multiple events sent from Filebeat](https://discuss.elastic.co/t/logstash-to-process-multiple-events-sent-from-filebeat/59142)
**Author:** [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\ **Replies:** 7\ **Last updated:** [August 30, 2016, 9:47pm UTC](https://discuss.elastic.co/t/logstash-to-process-multiple-events-sent-from-filebeat/59142 "2016-08-30T21:47:16Z")
Hi I have to send different formatted logs which are stored under same path. Using Filebeat as a log shipper i was able to send individual log files to logstash . By specifying multiple prospectors and document\_type in… --- ## [How to configure Redis Cluster with Logstash Cluster?](https://discuss.elastic.co/t/how-to-configure-redis-cluster-with-logstash-cluster/59182)
**Author:** [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Replies:** 3\ **Last updated:** [August 30, 2016, 8:13pm UTC](https://discuss.elastic.co/t/how-to-configure-redis-cluster-with-logstash-cluster/59182 "2016-08-30T20:13:24Z")
Hi, I am trying to deploy a ELK stack(as POC). In this I want a cluster of Redis as an input to Logstash cluster. How can I do this? What would be the configuration required? --- ## [Parsing nested key/value pairs with kv](https://discuss.elastic.co/t/parsing-nested-key-value-pairs-with-kv/33536)
**Author:** [@gogators](https://discuss.elastic.co/u/gogators)\ **Replies:** 2\ **Last updated:** [August 30, 2016, 7:59pm UTC](https://discuss.elastic.co/t/parsing-nested-key-value-pairs-with-kv/33536 "2016-08-30T19:59:53Z")
I'm trying to parse a log file that is composed of key/value pairs separated by '='. The problem I have is that some of the keys need to be interpreted as nested objects. For example a log line might look like this: 11… --- ## [Enable rolling logstash 2.3.4](https://discuss.elastic.co/t/enable-rolling-logstash-2-3-4/59355)
**Author:** [@prabhakar349](https://discuss.elastic.co/u/prabhakar349)\ **Replies:** 1\ **Last updated:** [August 30, 2016, 7:46pm UTC](https://discuss.elastic.co/t/enable-rolling-logstash-2-3-4/59355 "2016-08-30T19:46:59Z")
Hi All, I want to enable rolling log file for logstash process itself. I looked into documentation but it only have option to log only in one file which never rotates. Thanks, Prabhakar --- ## [Logs files multi line filter (ASCII)](https://discuss.elastic.co/t/logs-files-multi-line-filter-ascii/58970)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 11\ **Last updated:** [August 30, 2016, 7:46pm UTC](https://discuss.elastic.co/t/logs-files-multi-line-filter-ascii/58970 "2016-08-30T19:46:10Z")
HI, I would like to find out if logstash can process multiple lines that are below each other? Here is an example of a log: { 1=352621443221 2=3525945678 3=20140225132715 4=345 7=0 … --- ## [Invalid configuration "undefined method terminal"](https://discuss.elastic.co/t/invalid-configuration-undefined-method-terminal/58852)
**Author:** [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Replies:** 7\ **Last updated:** [August 30, 2016, 7:01pm UTC](https://discuss.elastic.co/t/invalid-configuration-undefined-method-terminal/58852 "2016-08-30T19:01:14Z")
I've been debugging my logstash config, and end up running into this: root@indexerNode01:/opt/logstash/bin# ./logstash -t -f /etc/logstash/conf.d/filter.conf The given configuration is invalid. Reason: undefined metho… --- ## [HAPROXY (HAPROXYHTTP) truncates last field if over 1024, breaking grok pattern](https://discuss.elastic.co/t/haproxy-haproxyhttp-truncates-last-field-if-over-1024-breaking-grok-pattern/57855)
**Author:** [@Michael\_Tracey](https://discuss.elastic.co/u/Michael_Tracey)\ **Replies:** 4\ **Last updated:** [August 30, 2016, 6:39pm UTC](https://discuss.elastic.co/t/haproxy-haproxyhttp-truncates-last-field-if-over-1024-breaking-grok-pattern/57855 "2016-08-30T18:39:36Z")
According to haproxy, the standard httplog truncates the final field if it's over 1024 characters long (from the manual): "http\_request" is the complete HTTP request line, including the method, request and HTTP ve… --- ## [Logstash server NOT seeing files shipped by rsyslog from remote client?](https://discuss.elastic.co/t/logstash-server-not-seeing-files-shipped-by-rsyslog-from-remote-client/59227)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 2\ **Last updated:** [August 30, 2016, 6:10pm UTC](https://discuss.elastic.co/t/logstash-server-not-seeing-files-shipped-by-rsyslog-from-remote-client/59227 "2016-08-30T18:10:12Z")
Logstash is not seeing files sent over from a remote client (VM running rsyslog) on my ELK server. I've been following the great article How to ship logs with Rsyslog and Logstash and am at the point where I'm shipping… --- ## [Is setting \`document\_type\` on filebeat a bad idea? Should Logstash be grok'ing and setting it?](https://discuss.elastic.co/t/is-setting-document-type-on-filebeat-a-bad-idea-should-logstash-be-groking-and-setting-it/58714)
**Author:** [@predominant](https://discuss.elastic.co/u/predominant)\ **Replies:** 2\ **Last updated:** [August 30, 2016, 4:57pm UTC](https://discuss.elastic.co/t/is-setting-document-type-on-filebeat-a-bad-idea-should-logstash-be-groking-and-setting-it/58714 "2016-08-30T16:57:31Z")
I'm finalising a deployment for a tenant based system whereby there are various systems pushing logs through logstash. Having filebeat set document\_type based on prospectors seems like a bad idea. What if people don't … --- ## [Integrating Logstash with websphere running in AIX](https://discuss.elastic.co/t/integrating-logstash-with-websphere-running-in-aix/58972)
**Author:** [@Hari1](https://discuss.elastic.co/u/Hari1)\ **Replies:** 4\ **Last updated:** [August 30, 2016, 4:06pm UTC](https://discuss.elastic.co/t/integrating-logstash-with-websphere-running-in-aix/58972 "2016-08-30T16:06:21Z")
I have a websphere application server running in a AIX machine, and all the log file are configured to fall under (usual) path /opt/..... /Appserver/logs/SystemOut.log.... Now I have successfully installed ELK in one of … --- ## [Latest version of logstash-input-dynamodb is breaking logstash](https://discuss.elastic.co/t/latest-version-of-logstash-input-dynamodb-is-breaking-logstash/56845)
**Author:** [@Andre\_Leite](https://discuss.elastic.co/u/Andre_Leite)\ **Replies:** 1\ **Last updated:** [August 30, 2016, 3:46pm UTC](https://discuss.elastic.co/t/latest-version-of-logstash-input-dynamodb-is-breaking-logstash/56845 "2016-08-30T15:46:00Z")
The latest version for the logstash-input-dynamodb plugin is currently breaking my setup. Whenever I try to start the logstash service I get a warning and I am asked to run the script with the configtest parameter which… --- ## [Rsyslog+ELK Integration](https://discuss.elastic.co/t/rsyslog-elk-integration/59125)
**Author:** [@Bhuvanesh](https://discuss.elastic.co/u/Bhuvanesh)\ **Replies:** 3\ **Last updated:** [August 30, 2016, 3:12pm UTC](https://discuss.elastic.co/t/rsyslog-elk-integration/59125 "2016-08-30T15:12:19Z")
Hi All, I am new to ELK stack. I am now trying to implement a centralized log server for one of the project in my organization. This implementation expectations are, Ship logs (system logs, webserver logs, firewall … --- ## [GrokDebugger work well but not in logstash](https://discuss.elastic.co/t/grokdebugger-work-well-but-not-in-logstash/59327)
**Author:** [@gitbub1992](https://discuss.elastic.co/u/gitbub1992)\ **Replies:** 0\ **Last updated:** [August 30, 2016, 3:01pm UTC](https://discuss.elastic.co/t/grokdebugger-work-well-but-not-in-logstash/59327 "2016-08-30T15:01:59Z")
Grok pattern test well in GrokDebugger,but got nothing in logstash after process input file with multiline. Input looks like this: \[2016-08-30 15:09:30,906\]\[/data/release/cdn/cgi/apps/cdn/RequestAjax.class.php:176:\_\_… --- ## [How to restrict fields from csv file using logstash?](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643)
**Author:** [@shankar.T](https://discuss.elastic.co/u/shankar.T)\ **Replies:** 12\ **Last updated:** [August 30, 2016, 11:31am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643 "2016-08-30T11:31:47Z")
For example in csv file i have 100 fields. But i want to load only 10 specific fileds which i needed. How can i achieve this using logstash? Thanks in advance.. Thanks & Regards, Shankarananth.T --- ## [How to change the value of environment variable?](https://discuss.elastic.co/t/how-to-change-the-value-of-environment-variable/59276)
**Author:** [@YYtea](https://discuss.elastic.co/u/YYtea)\ **Replies:** 3\ **Last updated:** [August 30, 2016, 11:31am UTC](https://discuss.elastic.co/t/how-to-change-the-value-of-environment-variable/59276 "2016-08-30T11:31:05Z")
input { stdin{ type =\> "${log\_type:unknown}" } } tried these and does not work ruby{ code =\>"event.set('\[@metadata\]\[log\_type\]','Atype')" } ruby{ code =\>"$log\_type = 'Atype'" } using logstash 5.0.0-alpha4 … --- ## [How to bind Cluster IP of redis to Logstash cluster?](https://discuss.elastic.co/t/how-to-bind-cluster-ip-of-redis-to-logstash-cluster/59229)
**Author:** [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Replies:** 1\ **Last updated:** [August 30, 2016, 11:13am UTC](https://discuss.elastic.co/t/how-to-bind-cluster-ip-of-redis-to-logstash-cluster/59229 "2016-08-30T11:13:41Z")
I am trying to deploy a ELK stack(as POC). In this I want a cluster of Redis as an input to Logstash cluster. How can I do this? What would be the configuration required? --- ## [I see no new entries uploaded with logstash in kibana - 74b40a](https://discuss.elastic.co/t/i-see-no-new-entries-uploaded-with-logstash-in-kibana-74b40a/59062)
**Author:** [@wawewiwow](https://discuss.elastic.co/u/wawewiwow)\ **Replies:** 3\ **Last updated:** [August 30, 2016, 10:21am UTC](https://discuss.elastic.co/t/i-see-no-new-entries-uploaded-with-logstash-in-kibana-74b40a/59062 "2016-08-30T10:21:10Z")
I see only three entries uploaded with logstash today in Kibana. But I see that number of entries loaded are pretty large in kopf. What am I doing wrong? --- ## [Unable to output JSON http messages with new lines](https://discuss.elastic.co/t/unable-to-output-json-http-messages-with-new-lines/55291)
**Author:** [@raffaele](https://discuss.elastic.co/u/raffaele)\ **Replies:** 3\ **Last updated:** [August 30, 2016, 6:05am UTC](https://discuss.elastic.co/t/unable-to-output-json-http-messages-with-new-lines/55291 "2016-08-30T06:05:07Z")
Hi there I am struggling getting Logstash to insert new lines in JSON http output messages. Configuring Logstash to send out http messages (output-http plugin) works however, putting a new line into the mapping text wi… --- ## [Need help with logstash config(version in first row)](https://discuss.elastic.co/t/need-help-with-logstash-config-version-in-first-row/59206)
**Author:** [@Alexey\_Plotnikov](https://discuss.elastic.co/u/Alexey_Plotnikov)\ **Replies:** 5\ **Last updated:** [August 30, 2016, 5:50am UTC](https://discuss.elastic.co/t/need-help-with-logstash-config-version-in-first-row/59206 "2016-08-30T05:50:17Z")
Hello, i don't know how to parse this log file: VERSION: 1.0.2 00:00:01.159\[http-thread-pool-8080(5)\] - ERROR - SOMETHING #1 00:00:22.372\[http-thread-pool-8080(1)\] - ERROR - SOMETHING #2 00:00:23.837\[http… --- ## [Move the location of newly added add\_field out of nested?](https://discuss.elastic.co/t/move-the-location-of-newly-added-add-field-out-of-nested/59143)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 11\ **Last updated:** [August 30, 2016, 5:49am UTC](https://discuss.elastic.co/t/move-the-location-of-newly-added-add-field-out-of-nested/59143 "2016-08-30T05:49:06Z")
Hi I can access the nested field by \[field\]\[sub\_field\], and add this as a new field. However, this new field is still nested, then it is no use to me. How can i change its position out of net pls? Below is an example. I… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1107) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1109)