# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=475 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 476 --- ## [Logstash slow input for TCP CEF Codec](https://discuss.elastic.co/t/logstash-slow-input-for-tcp-cef-codec/199808)
**Author:** [@Moses\_Lim\_Him\_Kuong](https://discuss.elastic.co/u/Moses_Lim_Him_Kuong)\ **Replies:** 3\ **Last updated:** [September 19, 2019, 2:35am UTC](https://discuss.elastic.co/t/logstash-slow-input-for-tcp-cef-codec/199808 "2019-09-19T02:35:05Z")
I'm setting up an instance of logstash that will be receiving CEF events from tcp port. I'm expecting it to be able to get up to 10k EPS but its only getting to around 2k on the input. What should i do? i'm on a 8core … --- ## [Logstash keeps restarting after clean install // Ubuntu 18.04 + Java 11](https://discuss.elastic.co/t/logstash-keeps-restarting-after-clean-install-ubuntu-18-04-java-11/200094)
**Author:** [@alilland](https://discuss.elastic.co/u/alilland)\ **Replies:** 2\ **Last updated:** [September 19, 2019, 12:21am UTC](https://discuss.elastic.co/t/logstash-keeps-restarting-after-clean-install-ubuntu-18-04-java-11/200094 "2019-09-19T00:21:45Z")
Steps to reproduce error: prerequisites: a clean install of Ubuntu 18.04 (I am running on Ec2) Step 1) $ sudo apt-get update Step 2) $ java -version Command 'java' not found, but can be installed with: sudo apt ins… --- ## [Logstash-output-google\_pubsub plugin performance Issues](https://discuss.elastic.co/t/logstash-output-google-pubsub-plugin-performance-issues/200079)
**Author:** [@ivantsepp](https://discuss.elastic.co/u/ivantsepp)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 10:16pm UTC](https://discuss.elastic.co/t/logstash-output-google-pubsub-plugin-performance-issues/200079 "2019-09-18T22:16:23Z")
Hey all! I experienced some performance issues with the logstash-output-google\_pubsub plugin. I believe I figured out the issue and created a PR at https://github.com/logstash-plugins/logstash-output-google\_pubsub/pull/… --- ## [What is the best method to ship the logs from the clients to logstash without any beat?](https://discuss.elastic.co/t/what-is-the-best-method-to-ship-the-logs-from-the-clients-to-logstash-without-any-beat/200090)
**Author:** [@rajkumar.m](https://discuss.elastic.co/u/rajkumar.m)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 9:45pm UTC](https://discuss.elastic.co/t/what-is-the-best-method-to-ship-the-logs-from-the-clients-to-logstash-without-any-beat/200090 "2019-09-18T21:45:16Z")
What is the best method to ship the logs from the clients to logstash without any beat? --- ## [Java Heap Space error OutOfMemoryError Logstash](https://discuss.elastic.co/t/java-heap-space-error-outofmemoryerror-logstash/198899)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 6\ **Last updated:** [September 18, 2019, 6:53pm UTC](https://discuss.elastic.co/t/java-heap-space-error-outofmemoryerror-logstash/198899 "2019-09-18T18:53:32Z")
I am getting an error Logstash - java.lang.OutOfMemoryError: Java heap space. However, I have changed the heap memory in the jvm.options file from 1g to 2g to 8g to 10g. I am still getting the same errors though. How can… --- ## [Eventstore Grok or any other compatible filter](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768)
**Author:** [@Ghassan\_Zein](https://discuss.elastic.co/u/Ghassan_Zein)\ **Replies:** 3\ **Last updated:** [September 18, 2019, 6:34pm UTC](https://discuss.elastic.co/t/eventstore-grok-or-any-other-compatible-filter/199768 "2019-09-18T18:34:26Z")
I am stuck trying to find a compatible eventstore parser that can filter out error logs coming from an eventstore server through filebeat, just like APACHELOG, SYSLOG and others have their own. If there's no specific fi… --- ## [Extract date and time from message field into a dedicated field](https://discuss.elastic.co/t/extract-date-and-time-from-message-field-into-a-dedicated-field/199908)
**Author:** [@Vipin\_Gaur](https://discuss.elastic.co/u/Vipin_Gaur)\ **Replies:** 3\ **Last updated:** [September 18, 2019, 4:51pm UTC](https://discuss.elastic.co/t/extract-date-and-time-from-message-field-into-a-dedicated-field/199908 "2019-09-18T16:51:52Z")
how to extract date and time from message field into a dedicated field. this is my .conf file input { beats { port =\> 5044 } } filter { if \[log\_type\] == "json" { mutate { rename =\> { "message" =\> "raw\_data… --- ## [Error after enabling SSL on logstash 7.3.0](https://discuss.elastic.co/t/error-after-enabling-ssl-on-logstash-7-3-0/200044)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 3:17pm UTC](https://discuss.elastic.co/t/error-after-enabling-ssl-on-logstash-7-3-0/200044 "2019-09-18T15:17:50Z")
We are currently configuring SSL on logstash but after adding the SSL config to the elasticsearch output an error is being generated which I cannot determine what is generating the error. We have a security policy that … --- ## [How schedule works in logstash when moving data from one cluster to the other cluster of elasticsearch](https://discuss.elastic.co/t/how-schedule-works-in-logstash-when-moving-data-from-one-cluster-to-the-other-cluster-of-elasticsearch/200043)
**Author:** [@JinnaBalu](https://discuss.elastic.co/u/JinnaBalu)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 3:13pm UTC](https://discuss.elastic.co/t/how-schedule-works-in-logstash-when-moving-data-from-one-cluster-to-the-other-cluster-of-elasticsearch/200043 "2019-09-18T15:13:04Z")
I am trying to move customer index from one cluster to ther other cluster removing some fields. Will schedule backups data incrementally. Data is updated to the input from the application We want logstash to move data … --- ## [\[ruby\] filter configuration is invalid](https://discuss.elastic.co/t/ruby-filter-configuration-is-invalid/199958)
**Author:** [@hdma](https://discuss.elastic.co/u/hdma)\ **Replies:** 2\ **Last updated:** [September 18, 2019, 3:09pm UTC](https://discuss.elastic.co/t/ruby-filter-configuration-is-invalid/199958 "2019-09-18T15:09:18Z")
Hello I tried to create a ruby's filter to permit logging messaage only when I have mounted a partition. This is the conf file: filter { ruby { code =\> 'if File.readlines("/proc/mounts").any?{ |line| line.split(' ')\[… --- ## [Migrating from "file input" to "beats input" ==\> filter/matching problem](https://discuss.elastic.co/t/migrating-from-file-input-to-beats-input-filter-matching-problem/200041)
**Author:** [@zebu14](https://discuss.elastic.co/u/zebu14)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 3:06pm UTC](https://discuss.elastic.co/t/migrating-from-file-input-to-beats-input-filter-matching-problem/200041 "2019-09-18T15:06:04Z")
Hello, I need to send my squid (modified) access.log to an Elasticsearch cluster. I've asked some log files from our system team and I used them as an input into my cluster (File =\> logstash =\> Elastic) in order to pre… --- ## [ILM Policy overriden by Logstash](https://discuss.elastic.co/t/ilm-policy-overriden-by-logstash/200031)
**Author:** [@bharath.krishn2](https://discuss.elastic.co/u/bharath.krishn2)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 2:28pm UTC](https://discuss.elastic.co/t/ilm-policy-overriden-by-logstash/200031 "2019-09-18T14:28:45Z")
I have a requirement for pull the data from two separate sources. I'm using 'file' and 'jdbc' as input plugin in the logstash configuration. In the output, I've segragated elasticsearch indices based on the types. Belo… --- ## [How to set logstash startup time?](https://discuss.elastic.co/t/how-to-set-logstash-startup-time/199579)
**Author:** [@hannkeat](https://discuss.elastic.co/u/hannkeat)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 2:23pm UTC](https://discuss.elastic.co/t/how-to-set-logstash-startup-time/199579 "2019-09-18T14:23:49Z")
Hi Team, I have tried many method, however it still fail to change startup time for logstash. Please guide me . .thanks in advance --- ## [Logstash s3 input plugin](https://discuss.elastic.co/t/logstash-s3-input-plugin/200016)
**Author:** [@Jagadeesh\_Babu](https://discuss.elastic.co/u/Jagadeesh_Babu)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 1:28pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin/200016 "2019-09-18T13:28:28Z")
Hi, I tried to create a folder in S3 and using the path of the S3 as input for in logstash file. I'm getting an error to remove the forward slash in the input. input { s3 { bucket =\> "o2-folder-bucket/… --- ## [If statement in logstash output](https://discuss.elastic.co/t/if-statement-in-logstash-output/199885)
**Author:** [@Brian\_DiCola](https://discuss.elastic.co/u/Brian_DiCola)\ **Replies:** 2\ **Last updated:** [September 18, 2019, 1:03pm UTC](https://discuss.elastic.co/t/if-statement-in-logstash-output/199885 "2019-09-18T13:03:19Z")
Here is my filebeats config processors: add\_tags: tags: \[web, dev\] target: "environment" #=========================== Filebeat inputs ============================= filebeat.inputs: type: log enabled: true pat… --- ## [How to update if document\_id is not unique](https://discuss.elastic.co/t/how-to-update-if-document-id-is-not-unique/200010)
**Author:** [@Maria\_Efimova](https://discuss.elastic.co/u/Maria_Efimova)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 1:00pm UTC](https://discuss.elastic.co/t/how-to-update-if-document-id-is-not-unique/200010 "2019-09-18T13:00:08Z")
My output is elasticsearch. All documents with a field A="some\_value" has to be updated. The field A is not unique. The unique \_id=AB where A is the field I am compare to and B is another field. How it can be done in lo… --- ## [Parse Single Line XML In Logstash](https://discuss.elastic.co/t/parse-single-line-xml-in-logstash/199884)
**Author:** [@babaturk](https://discuss.elastic.co/u/babaturk)\ **Replies:** 2\ **Last updated:** [September 18, 2019, 12:58pm UTC](https://discuss.elastic.co/t/parse-single-line-xml-in-logstash/199884 "2019-09-18T12:58:41Z")
I am trying to output from Logstash to Elasticsearch the "result" (pass) in my xml file that consists of this single line: \\\\ He… --- ## [Grok debugger in dev tools parses correctly, fails running logstash](https://discuss.elastic.co/t/grok-debugger-in-dev-tools-parses-correctly-fails-running-logstash/199087)
**Author:** [@starrynight](https://discuss.elastic.co/u/starrynight)\ **Replies:** 5\ **Last updated:** [September 18, 2019, 12:53pm UTC](https://discuss.elastic.co/t/grok-debugger-in-dev-tools-parses-correctly-fails-running-logstash/199087 "2019-09-18T12:53:09Z")
I was attempting to parse an atlassian bitbucket log here -- with the log example being this: 10.10.10.2,10.11.1.2 | https | i@5EB8GFx642x581467x0 | - | 2019-08-27 10:42:28,325 | "GET /security\_logout HTTP/1.1" | "" "Za… --- ## [Logstash & KIbana timestamp indexing](https://discuss.elastic.co/t/logstash-kibana-timestamp-indexing/199929)
**Author:** [@ALLAHshukur\_Ahmadzad](https://discuss.elastic.co/u/ALLAHshukur_Ahmadzad)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 12:44pm UTC](https://discuss.elastic.co/t/logstash-kibana-timestamp-indexing/199929 "2019-09-18T12:44:39Z")
Hi dear community.I am having problems with the index naming of logs that belongs to the one day but indexed as previous.Now the logs are like following: Log: 09-09-2019 00:00:08 - log..... Logstash output: output { … --- ## [Unable to parse nested json using logstash json filter](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888)
**Author:** [@vj\_patnana](https://discuss.elastic.co/u/vj_patnana)\ **Replies:** 5\ **Last updated:** [September 18, 2019, 12:24pm UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888 "2019-09-18T12:24:30Z")
Hello All, I am trying to parse logs which are in json format. I am trying to parse the message json field in the logs. I have tried using json plugin filter but it is not working as expected. Here is the log - { … --- ## [Change dynamically log file without re adding previous lines in elastic search and kibana](https://discuss.elastic.co/t/change-dynamically-log-file-without-re-adding-previous-lines-in-elastic-search-and-kibana/198696)
**Author:** [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 9:49am UTC](https://discuss.elastic.co/t/change-dynamically-log-file-without-re-adding-previous-lines-in-elastic-search-and-kibana/198696 "2019-09-18T09:49:19Z")
Hello, I have a log file which is generated by a script and exported on ELK machine, ingested by logstash. Every 1 or 2 months, I regenerate the file and the news logs are added to the initial file, and if there're new … --- ## [Logstash csv data output error](https://discuss.elastic.co/t/logstash-csv-data-output-error/199925)
**Author:** [@pgw5540](https://discuss.elastic.co/u/pgw5540)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 7:12am UTC](https://discuss.elastic.co/t/logstash-csv-data-output-error/199925 "2019-09-18T07:12:54Z")
Output csv data to elasticserach through logstash. However, sometimes certain problems occur. For example, the data of column B is outputed in column A. Therefore, the following error occurs because data of another fi… --- ## [Enable SNI in logstash tcp output module](https://discuss.elastic.co/t/enable-sni-in-logstash-tcp-output-module/199919)
**Author:** [@funduraghu](https://discuss.elastic.co/u/funduraghu)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 6:59am UTC](https://discuss.elastic.co/t/enable-sni-in-logstash-tcp-output-module/199919 "2019-09-18T06:59:01Z")
I use ssl over tcp to secure my communication with remote server. My remote server serves multiple domains and needs clients to send SNI so that it can serve correct certificates. I observed that logstash tcp output modu… --- ## [Could not index event to Elasticsearch. {:status=\>400](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-status-400/199916)
**Author:** [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 6:29am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-status-400/199916 "2019-09-18T06:29:49Z")
Hi There, i am using ELK-6.5.1 version and trying to create geo\_point indexes for kibana visualization, I have updated below template in kibana dev-tools and getting below error can anyone help me on this ? PUT \_templa… --- ## [Mysql aggregation does not find task\_id](https://discuss.elastic.co/t/mysql-aggregation-does-not-find-task-id/199800)
**Author:** [@NewToThis](https://discuss.elastic.co/u/NewToThis)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 6:29am UTC](https://discuss.elastic.co/t/mysql-aggregation-does-not-find-task-id/199800 "2019-09-18T06:29:24Z")
Hi there, Being scratching my head for a while now on the following problem: I am doing an aggregration on a specific task\_id which for some reason works when the input is from a csv log file but not when using a mysql… --- ## [How to parse out logs based on a hostname](https://discuss.elastic.co/t/how-to-parse-out-logs-based-on-a-hostname/199841)
**Author:** [@habi](https://discuss.elastic.co/u/habi)\ **Replies:** 1\ **Last updated:** [September 18, 2019, 5:44am UTC](https://discuss.elastic.co/t/how-to-parse-out-logs-based-on-a-hostname/199841 "2019-09-18T05:44:10Z")
hi i am trying to parse out nginx logs based on a specific hostname and applygrep to those logs and send the output to a seperate file and i was thinking of using bash scripting for this. I would like to know how can i … --- ## [ILM Support for beat-logstash-elastic](https://discuss.elastic.co/t/ilm-support-for-beat-logstash-elastic/199875)
**Author:** [@Andres\_Man](https://discuss.elastic.co/u/Andres_Man)\ **Replies:** 0\ **Last updated:** [September 17, 2019, 7:22pm UTC](https://discuss.elastic.co/t/ilm-support-for-beat-logstash-elastic/199875 "2019-09-17T19:22:24Z")
Is there a way to configure ILM in the beat -\> LS-\> ES configuration? I can configure ILM with little trouble directly from beat to ES, but there is no documentation on the beat-LS-ES configuration of ILM. Is it even pos… --- ## [Microsoft sqlserver and logstash](https://discuss.elastic.co/t/microsoft-sqlserver-and-logstash/199864)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [September 17, 2019, 5:58pm UTC](https://discuss.elastic.co/t/microsoft-sqlserver-and-logstash/199864 "2019-09-17T17:58:15Z")
I have try following config and it works with java 8 and sqljdbc4.2 jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_driver\_library =\> "/root/sqljdbc\_4.2/enu/jre8/sqljdbc42.jar" what I want to… --- ## [Logstash : convert to date in filter csv](https://discuss.elastic.co/t/logstash-convert-to-date-in-filter-csv/199825)
**Author:** [@RickT](https://discuss.elastic.co/u/RickT)\ **Replies:** 0\ **Last updated:** [September 17, 2019, 1:08pm UTC](https://discuss.elastic.co/t/logstash-convert-to-date-in-filter-csv/199825 "2019-09-17T13:08:25Z")
Suite du sujet Using convert =\> date in csv filter : hi, have you got an answer about using "convert =\> date in "csv" filter ? It does'nt work in my case too. And i am also curious !... --- ## [Logstash ILM problem](https://discuss.elastic.co/t/logstash-ilm-problem/199830)
**Author:** [@bharath.krishn2](https://discuss.elastic.co/u/bharath.krishn2)\ **Replies:** 3\ **Last updated:** [September 17, 2019, 4:59pm UTC](https://discuss.elastic.co/t/logstash-ilm-problem/199830 "2019-09-17T16:59:30Z")
I've created a life cycle policy and added it to an index-template. In the policy I've enabled the hot phase which will rollover the index if max\_docs is 50 or max\_age is 1h. Now the problem is that the new indices are… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=474) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=476)