# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1062 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1063 --- ## [Logstash is not creating the Index form the template index\_not\_found\_exception](https://discuss.elastic.co/t/logstash-is-not-creating-the-index-form-the-template-index-not-found-exception/69614)
**Author:** [@ganapathydselva](https://discuss.elastic.co/u/ganapathydselva)\ **Replies:** 0\ **Last updated:** [December 20, 2016, 8:13pm UTC](https://discuss.elastic.co/t/logstash-is-not-creating-the-index-form-the-template-index-not-found-exception/69614 "2016-12-20T20:13:57Z")
HI , Logstash is not creating the Index form the default template or the temple name provided in Elasticsearch . i am trying to process my logs with custom template svlogs but Inex is not getting created on the go base… --- ## [APT Package: Queue directory not writeable](https://discuss.elastic.co/t/apt-package-queue-directory-not-writeable/69392)
**Author:** [@padyx](https://discuss.elastic.co/u/padyx)\ **Replies:** 4\ **Last updated:** [December 20, 2016, 5:53pm UTC](https://discuss.elastic.co/t/apt-package-queue-directory-not-writeable/69392 "2016-12-20T17:53:47Z")
Has anyone else experienced the issue when installing Logstash 5.x as an APT-Package, that it won't start: \[2016-12-19T07:06:27,418\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>#\ **Author:** [@Matt\_Simcox](https://discuss.elastic.co/u/Matt_Simcox)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 5:39pm UTC](https://discuss.elastic.co/t/logstash-log-records-fail-to-load-in-large-log-file-but-work-when-run-separately/69384 "2016-12-20T17:39:11Z") Filebeat is feeding data into logstash and passing this onto ES. I'm monitoring the load of around 5 million records and notice that I'm getting a lot of \_grokparsefailure. when I drill into the logstash output I can s… --- ## [Use another plugin in a custom filter plugin?](https://discuss.elastic.co/t/use-another-plugin-in-a-custom-filter-plugin/69601)
**Author:** [@er0ck](https://discuss.elastic.co/u/er0ck)\ **Replies:** 0\ **Last updated:** [December 20, 2016, 5:36pm UTC](https://discuss.elastic.co/t/use-another-plugin-in-a-custom-filter-plugin/69601 "2016-12-20T17:36:16Z")
Hi All, what is the best way to use other filter plugins in a custom plugin? right now we have a bunch of ruby snippets in our configs that i'd like to leverage the plugin framework for testing, deployment, etc. t… --- ## [File creates empty sincedb file and no data is processed](https://discuss.elastic.co/t/file-creates-empty-sincedb-file-and-no-data-is-processed/69368)
**Author:** [@chromechris](https://discuss.elastic.co/u/chromechris)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 5:25pm UTC](https://discuss.elastic.co/t/file-creates-empty-sincedb-file-and-no-data-is-processed/69368 "2016-12-20T17:25:28Z")
I'm sure there is a logical reason for this. I have a file that is parsed most of the times but sometimes it is not. I have deleted the sincedb file pattern for this file multiple times, tried setting the "sincedb\_path" … --- ## [How to parse this kind of logs](https://discuss.elastic.co/t/how-to-parse-this-kind-of-logs/69422)
**Author:** [@hkatyal](https://discuss.elastic.co/u/hkatyal)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 5:15pm UTC](https://discuss.elastic.co/t/how-to-parse-this-kind-of-logs/69422 "2016-12-20T17:15:44Z")
===================================== 2016-12-19 07:01:24 7f119e9fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds BACKGROUND THREAD srv\_master… --- ## [Failed to install .deb logstash file on ubuntu 14.04](https://discuss.elastic.co/t/failed-to-install-deb-logstash-file-on-ubuntu-14-04/69438)
**Author:** [@Soham\_Navadiya](https://discuss.elastic.co/u/Soham_Navadiya)\ **Replies:** 1\ **Last updated:** [December 20, 2016, 5:03pm UTC](https://discuss.elastic.co/t/failed-to-install-deb-logstash-file-on-ubuntu-14-04/69438 "2016-12-20T17:03:03Z")
I am trying to setup ELK on my local machine. I successfully installed elastic search and kibana. But during installation of logstash I am getting error. Below is the terminal logs. I tried a lot but It is not working. A… --- ## [Why logstash is the best tool to send logs to Elasticsearch](https://discuss.elastic.co/t/why-logstash-is-the-best-tool-to-send-logs-to-elasticsearch/69458)
**Author:** [@rajraja](https://discuss.elastic.co/u/rajraja)\ **Replies:** 1\ **Last updated:** [December 20, 2016, 4:46pm UTC](https://discuss.elastic.co/t/why-logstash-is-the-best-tool-to-send-logs-to-elasticsearch/69458 "2016-12-20T16:46:52Z")
Hi Team, It is a kind question. I am just curious to know why logstash is the best tool to send logs to elasticsearch. We are running our all applications in AWS so, we are planning to use AWS Cloudwarch logs to collec… --- ## [Problem with logstash handling the mysql-slow.log](https://discuss.elastic.co/t/problem-with-logstash-handling-the-mysql-slow-log/69531)
**Author:** [@houfan](https://discuss.elastic.co/u/houfan)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 4:21pm UTC](https://discuss.elastic.co/t/problem-with-logstash-handling-the-mysql-slow-log/69531 "2016-12-20T16:21:26Z")
this is the filebeat.yml: filebeat: prospectors: - paths: - /usr/local/nginx/logs/mysql-slow.log input\_type: log document\_type: mysql-slow multiline: pattern: ^# User@Host negate: true match: … --- ## [Logstah date help](https://discuss.elastic.co/t/logstah-date-help/69171)
**Author:** [@pierre](https://discuss.elastic.co/u/pierre)\ **Replies:** 6\ **Last updated:** [December 20, 2016, 2:52pm UTC](https://discuss.elastic.co/t/logstah-date-help/69171 "2016-12-20T14:52:40Z")
i have this : 01-11-2015;17:41:01;641 filter { grok { break\_on\_match =\> "false" match =\> { "message" =\> '%{DATA}%{DATE\_EU:Date};%{TIME:Date};%{NUMBER:Nombre}%{DATA}'} } } Here my date = string and i want date = date s… --- ## [How to set the \[id\] field to value of \[path\] field (input-file)](https://discuss.elastic.co/t/how-to-set-the-id-field-to-value-of-path-field-input-file/69547)
**Author:** [@Tao\_Zhang](https://discuss.elastic.co/u/Tao_Zhang)\ **Replies:** 1\ **Last updated:** [December 20, 2016, 2:30pm UTC](https://discuss.elastic.co/t/how-to-set-the-id-field-to-value-of-path-field-input-file/69547 "2016-12-20T14:30:02Z")
My GOAL: I want set id to managementID like below when input a file: Ex. input-file-path: \\servername\*\\managementID\*\\input-file.txt The managementID is a unique value and I want update the file by this ID. Maybe … --- ## [Post data to logstash using http input](https://discuss.elastic.co/t/post-data-to-logstash-using-http-input/69166)
**Author:** [@adhikz](https://discuss.elastic.co/u/adhikz)\ **Replies:** 10\ **Last updated:** [December 20, 2016, 1:05pm UTC](https://discuss.elastic.co/t/post-data-to-logstash-using-http-input/69166 "2016-12-20T13:05:55Z")
I am current trying to do a JavaScript post to Logstash by using a tcp input. JavaScript Post xhr = new XMLHttpRequest(); var url = "http://localhost:5043"; xhr.open("POST", url, true); xhr.setRequestHeader("Cont… --- ## [Changing Logstash Heap Size](https://discuss.elastic.co/t/changing-logstash-heap-size/69542)
**Author:** [@wenyao](https://discuss.elastic.co/u/wenyao)\ **Replies:** 4\ **Last updated:** [December 20, 2016, 11:41am UTC](https://discuss.elastic.co/t/changing-logstash-heap-size/69542 "2016-12-20T11:41:09Z")
Is it possible to change the max and min heap size for logstash v5.0.1? If possible, how? --- ## [Unable to create Kafka consumer from given configuration](https://discuss.elastic.co/t/unable-to-create-kafka-consumer-from-given-configuration/68609)
**Author:** [@Haomeng](https://discuss.elastic.co/u/Haomeng)\ **Replies:** 4\ **Last updated:** [December 20, 2016, 11:36am UTC](https://discuss.elastic.co/t/unable-to-create-kafka-consumer-from-given-configuration/68609 "2016-12-20T11:36:31Z")
I run with logstash5.0 kafka plugin, got below message, and not sure why is it a ArgumentError, and which argument is wrong, any ideas?? \[2016-12-09T16:32:43,420\]\[DEBUG\]\[org.apache.kafka.clients.consumer.KafkaConsumer\]… --- ## [Is it possible to have a custom document\_id in Logstash?](https://discuss.elastic.co/t/is-it-possible-to-have-a-custom-document-id-in-logstash/69543)
**Author:** [@Kulasangar\_Gowrisang](https://discuss.elastic.co/u/Kulasangar_Gowrisang)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 11:36am UTC](https://discuss.elastic.co/t/is-it-possible-to-have-a-custom-document-id-in-logstash/69543 "2016-12-20T11:36:18Z")
I'm having two jdbc filters (but the same connection parameters, apart from the MySQL query) in my input part where i'm trying to get data from the database and push it to the index. As I said the only difference in both… --- ## [Aggregate log information](https://discuss.elastic.co/t/aggregate-log-information/69423)
**Author:** [@tinti](https://discuss.elastic.co/u/tinti)\ **Replies:** 1\ **Last updated:** [December 20, 2016, 11:26am UTC](https://discuss.elastic.co/t/aggregate-log-information/69423 "2016-12-20T11:26:23Z")
I want to enhance my logging. Imagine the following log lines: conn\_id=1234 accept connection from 192.168.0.1 conn\_id=1234 authentication successful user1 conn\_id=1234 search request xyz conn\_id=1234 delete request xy… --- ## [Logstash and elasticsearch performing](https://discuss.elastic.co/t/logstash-and-elasticsearch-performing/69279)
**Author:** [@pierre](https://discuss.elastic.co/u/pierre)\ **Replies:** 11\ **Last updated:** [December 20, 2016, 10:37am UTC](https://discuss.elastic.co/t/logstash-and-elasticsearch-performing/69279 "2016-12-20T10:37:17Z")
hey , my configuration 40 G Ram, 2 CPU 4 core , elk and nginx in the same server il give 15g -Xmx for logstash and elactic input { file { start\_position =\> "beginning" path =\> "/var/log/StatVM/\*" sincedb\_path =\> … --- ## [Adding a pattern in grok file in Logstash Grok File - Creating Grok Pattern](https://discuss.elastic.co/t/adding-a-pattern-in-grok-file-in-logstash-grok-file-creating-grok-pattern/69162)
**Author:** [@taurs](https://discuss.elastic.co/u/taurs)\ **Replies:** 3\ **Last updated:** [December 20, 2016, 10:09am UTC](https://discuss.elastic.co/t/adding-a-pattern-in-grok-file-in-logstash-grok-file-creating-grok-pattern/69162 "2016-12-20T10:09:52Z")
Hi, Here is my log pattern for which I need to write the grok for: Log Pattern : 2016-12-05 17:31:10:675 WARNI Received signal: SIGTERM \[OSignalHandler\] Another log pattern : 2016-12-05 16:52:12:805 SEVER Unabl… --- ## [Logstash 5.1.1 gets hanged while reading csv](https://discuss.elastic.co/t/logstash-5-1-1-gets-hanged-while-reading-csv/69172)
**Author:** [@Hassan\_Nashrah](https://discuss.elastic.co/u/Hassan_Nashrah)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 9:36am UTC](https://discuss.elastic.co/t/logstash-5-1-1-gets-hanged-while-reading-csv/69172 "2016-12-20T09:36:11Z")
when i run logstash. it starts read the csv send data to elasticsearch as well. but it hangs in the middle. logs show no error input { file { path =\> "/opt/logstash-5.1.1/bin/01.12.2016.csv" … --- ## [Config files unable to acess env variable value in Ubuntu](https://discuss.elastic.co/t/config-files-unable-to-acess-env-variable-value-in-ubuntu/69427)
**Author:** [@ezequielo](https://discuss.elastic.co/u/ezequielo)\ **Replies:** 5\ **Last updated:** [December 20, 2016, 9:17am UTC](https://discuss.elastic.co/t/config-files-unable-to-acess-env-variable-value-in-ubuntu/69427 "2016-12-20T09:17:36Z")
I am trying to use environment variables so I don't need to change my config files for every environment. So far, in my local Windows environment I was able to do it without problems. But it seems like in Linux/ubuntu th… --- ## [Fingerprint plugin crashes](https://discuss.elastic.co/t/fingerprint-plugin-crashes/69468)
**Author:** [@IuriiSergiichuk](https://discuss.elastic.co/u/IuriiSergiichuk)\ **Replies:** 4\ **Last updated:** [December 20, 2016, 8:16am UTC](https://discuss.elastic.co/t/fingerprint-plugin-crashes/69468 "2016-12-20T08:16:48Z")
Hello there. I'm using ELK 5.1.1 stack and right now was trying to configure fingerprint plugin to generate hash-code for URL links. Here's part of configuration: input { http{ type =\> "frontend\_eve… --- ## [Logstash unable to create new index](https://discuss.elastic.co/t/logstash-unable-to-create-new-index/69396)
**Author:** [@xsallowed](https://discuss.elastic.co/u/xsallowed)\ **Replies:** 5\ **Last updated:** [December 20, 2016, 7:32am UTC](https://discuss.elastic.co/t/logstash-unable-to-create-new-index/69396 "2016-12-20T07:32:32Z")
Hi Following is my configuration file for logstash and I am trying to create a new index by specifying non standard index name in the output section of this logstash file. I am unable to the data parsed if I am giving t… --- ## [Logstash mutate filter can change single field into multi\_fields?](https://discuss.elastic.co/t/logstash-mutate-filter-can-change-single-field-into-multi-fields/69526)
**Author:** [@tanimoto](https://discuss.elastic.co/u/tanimoto)\ **Replies:** 2\ **Last updated:** [December 20, 2016, 7:18am UTC](https://discuss.elastic.co/t/logstash-mutate-filter-can-change-single-field-into-multi-fields/69526 "2016-12-20T07:18:31Z")
My logstash conf is like this. input { jdbc { jdbc\_driver\_library =\> "/usr/local/pgsql/share/java/postgresql-9.4.1212.jar" jdbc\_driver\_class =\> "org.postgresql.Driver" jdbc\_connection\_string =\> "jdbc:pos… --- ## [Why is there no output in terminal?](https://discuss.elastic.co/t/why-is-there-no-output-in-terminal/69251)
**Author:** [@houfan](https://discuss.elastic.co/u/houfan)\ **Replies:** 4\ **Last updated:** [December 20, 2016, 7:04am UTC](https://discuss.elastic.co/t/why-is-there-no-output-in-terminal/69251 "2016-12-20T07:04:22Z")
version: logstash-2.3.4 nginx-1.8.1 here is my logstash.conf :slight\_smile: : input { \*\* file{\*\* \*\* path =\> \["/home/houfan/nginx/errorlog/ac\_api\_error.log"\]\*\* \*\* start\_position =\> "beginning"\*\* … --- ## [Need help in configuring live logs from server to logstash](https://discuss.elastic.co/t/need-help-in-configuring-live-logs-from-server-to-logstash/69466)
**Author:** [@rashmy](https://discuss.elastic.co/u/rashmy)\ **Replies:** 3\ **Last updated:** [December 20, 2016, 6:51am UTC](https://discuss.elastic.co/t/need-help-in-configuring-live-logs-from-server-to-logstash/69466 "2016-12-20T06:51:33Z")
Hi, I want to access live logs from my server and configure this path in logstash so that I can search exceptions in the logs with filters in logstash and view the result in kibana. Can anyone help me in this. Thanks --- ## [Logstash failover s3](https://discuss.elastic.co/t/logstash-failover-s3/69037)
**Author:** [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Replies:** 3\ **Last updated:** [December 20, 2016, 6:49am UTC](https://discuss.elastic.co/t/logstash-failover-s3/69037 "2016-12-20T06:49:22Z")
i was planning to pull elb logs from s3 using logstash. i want to use multiple logstash for failover control. is there anything i can use multiple logstash to pull logs from s3 bucket with loadbalance or something else c… --- ## [S3 output can't find files it generates](https://discuss.elastic.co/t/s3-output-cant-find-files-it-generates/68546)
**Author:** [@vad](https://discuss.elastic.co/u/vad)\ **Replies:** 2\ **Last updated:** [December 19, 2016, 7:00pm UTC](https://discuss.elastic.co/t/s3-output-cant-find-files-it-generates/68546 "2016-12-19T19:00:50Z")
Hi, I'm using Logstash 5.0.2 (and .0.1 before) and I need to add an s3 output but I can't because randomly one of the file logstash generates cannot be found: \[2016-12-07T14:37:39,809\]\[ERROR\]\[logstash.outputs.s3 \]… --- ## [Logstash crashing](https://discuss.elastic.co/t/logstash-crashing/69408)
**Author:** [@Karl\_Trasschaert](https://discuss.elastic.co/u/Karl_Trasschaert)\ **Replies:** 9\ **Last updated:** [December 19, 2016, 5:57pm UTC](https://discuss.elastic.co/t/logstash-crashing/69408 "2016-12-19T17:57:22Z")
Hi, I have regulary my logstash server crashing. Sometimes after a few restart it works again but alwais crash after a fex hours. As i'm not a king in linux/logstash can you help me to find out what's wrong ? Here is… --- ## [Logstash IMAP input unable to parse HTML email](https://discuss.elastic.co/t/logstash-imap-input-unable-to-parse-html-email/68951)
**Author:** [@Geezer](https://discuss.elastic.co/u/Geezer)\ **Replies:** 2\ **Last updated:** [December 19, 2016, 3:46pm UTC](https://discuss.elastic.co/t/logstash-imap-input-unable-to-parse-html-email/68951 "2016-12-19T15:46:15Z")
Hello, I'm trying to use the logstash-input-imap plugin to parse email. I can read plain text emails OK but with an HTML email I get a page of HTML code as the message, rather than the text. I have set the content\_ty… --- ## [How to remove special character from string in logstash](https://discuss.elastic.co/t/how-to-remove-special-character-from-string-in-logstash/68273)
**Author:** [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Replies:** 8\ **Last updated:** [December 19, 2016, 1:35pm UTC](https://discuss.elastic.co/t/how-to-remove-special-character-from-string-in-logstash/68273 "2016-12-19T13:35:49Z")
I'm getting some fields in the kibana like "attributes.host\_mem\_file\_buffer 1.3747773440e+09|g". I want to first replace with "|g" to "" and then convert to float value. I tried below one but it's not replacing … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1061) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1063)