# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=533 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 534 --- ## [LOGSTASH ES INPUT](https://discuss.elastic.co/t/logstash-es-input/184542)
**Author:** [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Replies:** 2\ **Last updated:** [June 7, 2019, 5:48am UTC](https://discuss.elastic.co/t/logstash-es-input/184542 "2019-06-07T05:48:21Z")
Hi there, I'm trying to use elasticsearch input in Logstash. My ES node is secured via TLS. I've configured ca\_file setting in Logstash with my CAchain but I obtain http client did not trust this server's certificate … --- ## [Stop Logstash automatically if Elasticsearch is not running](https://discuss.elastic.co/t/stop-logstash-automatically-if-elasticsearch-is-not-running/184499)
**Author:** [@Velmurugan](https://discuss.elastic.co/u/Velmurugan)\ **Replies:** 3\ **Last updated:** [June 7, 2019, 4:09am UTC](https://discuss.elastic.co/t/stop-logstash-automatically-if-elasticsearch-is-not-running/184499 "2019-06-07T04:09:47Z")
I am running logstash on server 1 and I have configured Elasticsearch and kibana on server 2 resulting that server 1's logstash will send data/logs to server 2's Elasticsearch. Recently I have encountered one pro… --- ## [Logstash throws an error when parsing a huge log file](https://discuss.elastic.co/t/logstash-throws-an-error-when-parsing-a-huge-log-file/184681)
**Author:** [@ikarthikb](https://discuss.elastic.co/u/ikarthikb)\ **Replies:** 0\ **Last updated:** [June 7, 2019, 3:06am UTC](https://discuss.elastic.co/t/logstash-throws-an-error-when-parsing-a-huge-log-file/184681 "2019-06-07T03:06:39Z")
I have a logstash configuration thats reading files from a directory. The input file consists of a number of xml blocks. The filter looks for xml blocks and then tries to get a value of a particular element within the x… --- ## [Troubles with apache rewrite log grok](https://discuss.elastic.co/t/troubles-with-apache-rewrite-log-grok/184237)
**Author:** [@adamatx](https://discuss.elastic.co/u/adamatx)\ **Replies:** 3\ **Last updated:** [June 6, 2019, 9:55pm UTC](https://discuss.elastic.co/t/troubles-with-apache-rewrite-log-grok/184237 "2019-06-06T21:55:43Z")
Hi! First time posting, I haven't had much luck finding apache rewrite log info anywhere online and I've been struggling with this so I figured I'd check withe the community. I'm new to logstash (6.3.2) and this is my … --- ## [Error Uploading a CSV file to Logstash](https://discuss.elastic.co/t/error-uploading-a-csv-file-to-logstash/184638)
**Author:** [@Rohn07](https://discuss.elastic.co/u/Rohn07)\ **Replies:** 2\ **Last updated:** [June 6, 2019, 7:20pm UTC](https://discuss.elastic.co/t/error-uploading-a-csv-file-to-logstash/184638 "2019-06-06T19:20:33Z")
ELK\_carsdata.conf input{ file{ path =\> "D:/ELK\_Data/carsdata/cars.csv" start\_position =\> "beginning" sincedb\_path =\> "NUL" } } filter{ csv{ separator =\> "," columns =\> \["mpg","cylinders"," cubicinches","hp","w… --- ## [How to read avro records from kafka using logstash](https://discuss.elastic.co/t/how-to-read-avro-records-from-kafka-using-logstash/184652)
**Author:** [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 7:18pm UTC](https://discuss.elastic.co/t/how-to-read-avro-records-from-kafka-using-logstash/184652 "2019-06-06T19:18:07Z")
I am trying to read avro records from a kafka topic using logstash. below is my config: input{ kafka { bootstrap\_servers =\>"test123" topics =\> \["abc123"\] group\_id =\> "abc123-cg1" auto\_offset\_reset =\> "latest" codec =\>… --- ## [Logstash Aggregation Plugin](https://discuss.elastic.co/t/logstash-aggregation-plugin/184632)
**Author:** [@Elk\_huh](https://discuss.elastic.co/u/Elk_huh)\ **Replies:** 5\ **Last updated:** [June 6, 2019, 7:04pm UTC](https://discuss.elastic.co/t/logstash-aggregation-plugin/184632 "2019-06-06T19:04:24Z")
We are getting multiple events come in ( because of multiple botnet calls, but it is the same SRC and DST) How do you use the aggregation plugin to combine it all into 1 . Timestamps are slightly different Time src … --- ## [Logstash parser](https://discuss.elastic.co/t/logstash-parser/184642)
**Author:** [@vinu89](https://discuss.elastic.co/u/vinu89)\ **Replies:** 2\ **Last updated:** [June 6, 2019, 6:54pm UTC](https://discuss.elastic.co/t/logstash-parser/184642 "2019-06-06T18:54:42Z")
Hi can someone help me to write the parser for this below logs Message 335881139 enqueued on incoming connection (ICID 0) from xyz@xyz.com to xyz@xyz.com Message 335881139 scanned by Malware Protection engine.MALICIOUS… --- ## [Old template being reinstalled by logstash](https://discuss.elastic.co/t/old-template-being-reinstalled-by-logstash/184530)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 1\ **Last updated:** [June 6, 2019, 6:49pm UTC](https://discuss.elastic.co/t/old-template-being-reinstalled-by-logstash/184530 "2019-06-06T18:49:14Z")
I have upgraded from 6.0.0 to 6.7.2. We initially experienced a problem where the type field was pushing 2 types out which we resolved by adding a new template in Elasticsearch. I have checked the logstash logs where I … --- ## [Logstash to Azure IoT hub via http/https](https://discuss.elastic.co/t/logstash-to-azure-iot-hub-via-http-https/184648)
**Author:** [@kishorerv93](https://discuss.elastic.co/u/kishorerv93)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 6:43pm UTC](https://discuss.elastic.co/t/logstash-to-azure-iot-hub-via-http-https/184648 "2019-06-06T18:43:56Z")
Any one using IoT hub as an output in logstash. I know that there is a plugin logstash-output-iothub plugin to leverage logstash to ship to IoT hub. output { iothub { connection\_string =\> "{set your iot hub connec… --- ## [Trying to gsub with regex and seeing character encoding error](https://discuss.elastic.co/t/trying-to-gsub-with-regex-and-seeing-character-encoding-error/184647)
**Author:** [@az123](https://discuss.elastic.co/u/az123)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 6:38pm UTC](https://discuss.elastic.co/t/trying-to-gsub-with-regex-and-seeing-character-encoding-error/184647 "2019-06-06T18:38:04Z")
I am trying to parse json coming over a tcp port. I noticed json parse errors and it looks like its due to some non-json strings in the input stream. I then crafted a regexp that matches these spurious messages and tried… --- ## [Error connecting MySQL with Logstash](https://discuss.elastic.co/t/error-connecting-mysql-with-logstash/184637)
**Author:** [@Rohn07](https://discuss.elastic.co/u/Rohn07)\ **Replies:** 3\ **Last updated:** [June 6, 2019, 6:04pm UTC](https://discuss.elastic.co/t/error-connecting-mysql-with-logstash/184637 "2019-06-06T18:04:43Z")
mysql.conf input{ jdbc { jdbc\_driver\_library =\> "C:\\ELK\\mysql-connector-java-5.1.36-bin.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/test" jdbc\_user =\> "r… --- ## [Logstash fingerprint 7.1](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598)
**Author:** [@pgervais](https://discuss.elastic.co/u/pgervais)\ **Replies:** 5\ **Last updated:** [June 6, 2019, 4:48pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-7-1/184598 "2019-06-06T16:48:31Z")
I want to ensure that the document id are based on the input source. For this i'm using the fingerprint filter in logstash. In previous version i've used it as follows fingerprint { source =\> "message" target =\> "\[@m… --- ## [PingFederate Server Logs Help](https://discuss.elastic.co/t/pingfederate-server-logs-help/184629)
**Author:** [@sysadrift](https://discuss.elastic.co/u/sysadrift)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 4:07pm UTC](https://discuss.elastic.co/t/pingfederate-server-logs-help/184629 "2019-06-06T16:07:58Z")
I'm trying to get a grok pattern working for PingFederate server logs, specifically the instances that have multi-line XML. The logs are coming in via FileBeats which is configured correctly for multiline files. Below … --- ## [Different Grok expressions for the same log file](https://discuss.elastic.co/t/different-grok-expressions-for-the-same-log-file/184622)
**Author:** [@habi](https://discuss.elastic.co/u/habi)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 3:29pm UTC](https://discuss.elastic.co/t/different-grok-expressions-for-the-same-log-file/184622 "2019-06-06T15:29:33Z")
Hey all, i have a question with regards to the grok filter, so i have different types log formats in my nginx-access.log and i want to parse them out seperately, in my kibana, i did try adding the optional fields but th… --- ## [Final mapping would have more than 1 type: \[\_doc, doc\]](https://discuss.elastic.co/t/final-mapping-would-have-more-than-1-type-doc-doc/184580)
**Author:** [@Yashwant\_Shettigar](https://discuss.elastic.co/u/Yashwant_Shettigar)\ **Replies:** 3\ **Last updated:** [June 6, 2019, 3:14pm UTC](https://discuss.elastic.co/t/final-mapping-would-have-more-than-1-type-doc-doc/184580 "2019-06-06T15:14:26Z")
I am getting below error in my logstash logs: \[2019-06-06T01:06:27,503\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"winlogbeat-2… --- ## [Filter Metric Flush\_Interval](https://discuss.elastic.co/t/filter-metric-flush-interval/184618)
**Author:** [@BrunoSdeMenezes](https://discuss.elastic.co/u/BrunoSdeMenezes)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 3:12pm UTC](https://discuss.elastic.co/t/filter-metric-flush-interval/184618 "2019-06-06T15:12:41Z")
Hello, I'm doing some metrics in a logstash script to generate a new index with some calculations. Everything works fine for now, but when I change the flush\_interval to some value high, it doesn't work, there is no out… --- ## [How to write while Loops inside plugins or filters?](https://discuss.elastic.co/t/how-to-write-while-loops-inside-plugins-or-filters/184426)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 10\ **Last updated:** [June 6, 2019, 3:03pm UTC](https://discuss.elastic.co/t/how-to-write-while-loops-inside-plugins-or-filters/184426 "2019-06-06T15:03:03Z")
Hi, Is there any way to write loops inside filters or outputs plugins? For a certain message {"time"=100,"name"="test"} I want to implement something like below: c=0, while c\<=10 time=time+c c=c+1 if c=10 reset to … --- ## [How to put one message to several indices by condition?](https://discuss.elastic.co/t/how-to-put-one-message-to-several-indices-by-condition/184391)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 6\ **Last updated:** [June 6, 2019, 12:30pm UTC](https://discuss.elastic.co/t/how-to-put-one-message-to-several-indices-by-condition/184391 "2019-06-06T12:30:07Z")
Hi there. I have a bunch of log messages that I want to parse by several ways. All messages are guaranteed to fit one common grok filter so they are going to index 1. But i want to check this messages agains other gro… --- ## [SQS Input Pipeline Blocking Pipeline Management Updates and Terminate Job](https://discuss.elastic.co/t/sqs-input-pipeline-blocking-pipeline-management-updates-and-terminate-job/182295)
**Author:** [@homero\_g](https://discuss.elastic.co/u/homero_g)\ **Replies:** 1\ **Last updated:** [June 6, 2019, 12:26pm UTC](https://discuss.elastic.co/t/sqs-input-pipeline-blocking-pipeline-management-updates-and-terminate-job/182295 "2019-06-06T12:26:29Z")
I have a very simple SQS input to ES output pipeline that is blocking the logstash job. Just starting logstash with no messages in the SQS and trying to stop the job with Ctrl + C I get the following error \[2019-05-22T1… --- ## [Not able to find indices, Learning about elastic stack](https://discuss.elastic.co/t/not-able-to-find-indices-learning-about-elastic-stack/184553)
**Author:** [@dbranco](https://discuss.elastic.co/u/dbranco)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 10:15am UTC](https://discuss.elastic.co/t/not-able-to-find-indices-learning-about-elastic-stack/184553 "2019-06-06T10:15:00Z")
Hello everyone; I am new in this about the elastic stack. I have been reading about it and learning since 3 days ago, but I still have missing pieces. I tried to orchestrate a simple sample using docker/docker-compose … --- ## [Logstash slows down overtime](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274)
**Author:** [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Replies:** 14\ **Last updated:** [June 6, 2019, 10:21am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274 "2019-06-06T10:21:22Z")
input { file { path =\> "/home/elk/logstash-6.5.4/input\_data/log-1000k.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { kv { field\_split =\> " " value\_split =\> "=" } ruby { init =\> … --- ## [Replace Whitespaces In Key's](https://discuss.elastic.co/t/replace-whitespaces-in-keys/184518)
**Author:** [@tomx1](https://discuss.elastic.co/u/tomx1)\ **Replies:** 4\ **Last updated:** [June 6, 2019, 10:17am UTC](https://discuss.elastic.co/t/replace-whitespaces-in-keys/184518 "2019-06-06T10:17:04Z")
As we are now collecting access logs of our Checkpoint Firewalls, we are facing the issue of horrible inconsistent key values (so the name of the keys itself :wink: ). They are formated in different ways but I was able t… --- ## [Load Testing of Logstash](https://discuss.elastic.co/t/load-testing-of-logstash/183944)
**Author:** [@viratag26](https://discuss.elastic.co/u/viratag26)\ **Replies:** 3\ **Last updated:** [June 6, 2019, 9:46am UTC](https://discuss.elastic.co/t/load-testing-of-logstash/183944 "2019-06-06T09:46:09Z")
HI Everyone I want to test how many clients(similar) can my currently setup Logstash server handle. I need to increase the number of servers on which filebeat agent will be installed which will be sending data at a con… --- ## [Logstash message truncation prevention](https://discuss.elastic.co/t/logstash-message-truncation-prevention/184533)
**Author:** [@SISA\_Nirmal\_kumar\_B](https://discuss.elastic.co/u/SISA_Nirmal_kumar_B)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 9:14am UTC](https://discuss.elastic.co/t/logstash-message-truncation-prevention/184533 "2019-06-06T09:14:22Z")
Is there any way to prevent logstash from truncating large nested json array. --- ## [Monitoring of logstash in kibana not working](https://discuss.elastic.co/t/monitoring-of-logstash-in-kibana-not-working/184100)
**Author:** [@yossib](https://discuss.elastic.co/u/yossib)\ **Replies:** 4\ **Last updated:** [June 6, 2019, 8:52am UTC](https://discuss.elastic.co/t/monitoring-of-logstash-in-kibana-not-working/184100 "2019-06-06T08:52:53Z")
3 node cluster, https configured configuration in logstash.yml: xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.hosts: \["https://esnode3.iucc.ac.il:9200", "https://esnode2.iucc.ac.il:9200", "https://esno… --- ## [Mysql Binlog File to Logstash](https://discuss.elastic.co/t/mysql-binlog-file-to-logstash/184529)
**Author:** [@AtulxD123](https://discuss.elastic.co/u/AtulxD123)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 8:50am UTC](https://discuss.elastic.co/t/mysql-binlog-file-to-logstash/184529 "2019-06-06T08:50:49Z")
I wish to pipeline Binlog file data to logstsah, is there any way to config logstash to do the same? --- ## [Create a calculated field or empty if value missing](https://discuss.elastic.co/t/create-a-calculated-field-or-empty-if-value-missing/181908)
**Author:** [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\ **Replies:** 7\ **Last updated:** [June 6, 2019, 8:38am UTC](https://discuss.elastic.co/t/create-a-calculated-field-or-empty-if-value-missing/181908 "2019-06-06T08:38:26Z")
Hi here, I'm struggling trying to find the best way to create a field been calculated by doing the multiplication of two others. So first, I need to test that both fields are not null and then I can do my multiplication… --- ## [Logstash json parse failure when json contains a "message" field](https://discuss.elastic.co/t/logstash-json-parse-failure-when-json-contains-a-message-field/184407)
**Author:** [@poitschkka01](https://discuss.elastic.co/u/poitschkka01)\ **Replies:** 5\ **Last updated:** [June 6, 2019, 7:50am UTC](https://discuss.elastic.co/t/logstash-json-parse-failure-when-json-contains-a-message-field/184407 "2019-06-06T07:50:50Z")
Hello, I'm using the json filter to parse the json formatted logfile. The json includes a message field. The (formatted) record looks like this: { "transactionId": "1559728771468", "hostName": "AUE2RHEESSVMSSP010000… --- ## [Configuring the new logstash version - issues on the output Elasticsearch template for the mapping of my logs](https://discuss.elastic.co/t/configuring-the-new-logstash-version-issues-on-the-output-elasticsearch-template-for-the-mapping-of-my-logs/184358)
**Author:** [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\ **Replies:** 9\ **Last updated:** [June 6, 2019, 7:40am UTC](https://discuss.elastic.co/t/configuring-the-new-logstash-version-issues-on-the-output-elasticsearch-template-for-the-mapping-of-my-logs/184358 "2019-06-06T07:40:17Z")
Hi Guys I'm trying to configure the new version logstash (7.x) Everything is working as prefect, logstash is running and I can see my logs on kibana :slight\_smile: . BUT the mapping/parsing elasticsearch isn't working :… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=532) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=534)