# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=600 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 601 --- ## [Filebeat and logstash not communicating](https://discuss.elastic.co/t/filebeat-and-logstash-not-communicating/168639)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 10\ **Last updated:** [February 15, 2019, 9:04pm UTC](https://discuss.elastic.co/t/filebeat-and-logstash-not-communicating/168639 "2019-02-15T21:04:07Z")
I have simplest configuration I have follwoing on system where log is being shipped from cat filebeat.yml |grep -v '#' |sed '/^$/d' filebeat.inputs: type: log enabled: true paths: /tmp/sensors\*.log scan\_frequenc… --- ## [How can I convert a given array structure?](https://discuss.elastic.co/t/how-can-i-convert-a-given-array-structure/168635)
**Author:** [@Skeeve](https://discuss.elastic.co/u/Skeeve)\ **Replies:** 2\ **Last updated:** [February 15, 2019, 9:03pm UTC](https://discuss.elastic.co/t/how-can-i-convert-a-given-array-structure/168635 "2019-02-15T21:03:09Z")
Hi, I'm still in the beginning of understanding how to use logstash. I have this structure in my event's JSON. "testDetails":\[ { "placeholder":"NO\_SECURE\_FLAG\_SET", "values":{…some more stuff…} … --- ## [Want to run ruby filter external script from command line](https://discuss.elastic.co/t/want-to-run-ruby-filter-external-script-from-command-line/168303)
**Author:** [@brandondash](https://discuss.elastic.co/u/brandondash)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 8:17pm UTC](https://discuss.elastic.co/t/want-to-run-ruby-filter-external-script-from-command-line/168303 "2019-02-15T20:17:31Z")
I use the LogStash::Event object in my script. It all goes great until I try to run it outside of a running Logstash instance. I get a huge set of errors while trying to load the libraries and I can't seem to untangle t… --- ## [How to express today's date in sql in logstash](https://discuss.elastic.co/t/how-to-express-todays-date-in-sql-in-logstash/168319)
**Author:** [@scorpioy1](https://discuss.elastic.co/u/scorpioy1)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 8:09pm UTC](https://discuss.elastic.co/t/how-to-express-todays-date-in-sql-in-logstash/168319 "2019-02-15T20:09:27Z")
How should I express today's date in sql in logstash? I tried this and it doesn't work. It throws error saying cannot convert VARCHAR '%{YYYY.DD.dd}' to DATE. Please help. input { jdbc { .... statement =\> "select \*… --- ## [Logstash upsert usage](https://discuss.elastic.co/t/logstash-upsert-usage/168244)
**Author:** [@manikandanid](https://discuss.elastic.co/u/manikandanid)\ **Replies:** 3\ **Last updated:** [February 15, 2019, 8:03pm UTC](https://discuss.elastic.co/t/logstash-upsert-usage/168244 "2019-02-15T20:03:09Z")
I am merging two docs based on a ID which i frame it out from the document i receive. The two different doc which i want merge share the same ID and i merge it in logstash using upsert. The document are merging but at th… --- ## [Sent logs to Logstash using Filebeat](https://discuss.elastic.co/t/sent-logs-to-logstash-using-filebeat/168420)
**Author:** [@erdo](https://discuss.elastic.co/u/erdo)\ **Replies:** 16\ **Last updated:** [February 15, 2019, 7:43pm UTC](https://discuss.elastic.co/t/sent-logs-to-logstash-using-filebeat/168420 "2019-02-15T19:43:45Z")
I am trying to send log files using Filebeat to logstash. Filebeat and logstash are on different servers so Filebeat ip is like this 100.100.100.100 and logstash is like 200.200.200.200 My filebeat conf: output.logstas… --- ## [Start Logstash as a service or a full binary path](https://discuss.elastic.co/t/start-logstash-as-a-service-or-a-full-binary-path/168628)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 0\ **Last updated:** [February 15, 2019, 6:13pm UTC](https://discuss.elastic.co/t/start-logstash-as-a-service-or-a-full-binary-path/168628 "2019-02-15T18:13:34Z")
Why i'm getting the different data while starting my Logstash as systemctl stat logstash.service and /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ --config.reload.automatic &. I i start the Service as serv… --- ## [CSV Import - Keys sporadically wrong](https://discuss.elastic.co/t/csv-import-keys-sporadically-wrong/168589)
**Author:** [@tbra](https://discuss.elastic.co/u/tbra)\ **Replies:** 4\ **Last updated:** [February 15, 2019, 5:00pm UTC](https://discuss.elastic.co/t/csv-import-keys-sporadically-wrong/168589 "2019-02-15T17:00:48Z")
Hello, I have this simple csv test file: D1,A1,A2,A3 15.2.2019,Firstname1,Lastname1,1200 16.2.2019,Firstname2,Lastname2,2200 which I import with this config file: input { file { path =\> "/Users/tbra/appl/elasti… --- ## [Run multiple logstash replicas in Docker](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595)
**Author:** [@chmodrs](https://discuss.elastic.co/u/chmodrs)\ **Replies:** 7\ **Last updated:** [February 15, 2019, 3:31pm UTC](https://discuss.elastic.co/t/run-multiple-logstash-replicas-in-docker/168595 "2019-02-15T15:31:05Z")
Hi, I have an ELK cluster running in Docker Swarm mode. The cluster meed my needs because i have two replicas of elastic, but i have a problem with logstash running multiple instances. When i send amount of logs to log… --- ## [Logstash is not starting with Error execute Action Create/Pipeline\_main](https://discuss.elastic.co/t/logstash-is-not-starting-with-error-execute-action-create-pipeline-main/168598)
**Author:** [@Nils\_Kaper](https://discuss.elastic.co/u/Nils_Kaper)\ **Replies:** 4\ **Last updated:** [February 15, 2019, 3:06pm UTC](https://discuss.elastic.co/t/logstash-is-not-starting-with-error-execute-action-create-pipeline-main/168598 "2019-02-15T15:06:05Z")
Hey, I am constantly trying to get logstash to run. But unfortunately it always shows the error message which is mentioned in the title. Here is my logstash.conf: input { beats { port =\> 5044 } } filter{ date{ … --- ## [Logstash - Failed to execute action](https://discuss.elastic.co/t/logstash-failed-to-execute-action/168576)
**Author:** [@Osias](https://discuss.elastic.co/u/Osias)\ **Replies:** 2\ **Last updated:** [February 15, 2019, 3:03pm UTC](https://discuss.elastic.co/t/logstash-failed-to-execute-action/168576 "2019-02-15T15:03:02Z")
Please, I need help with this error. I could not find a solution to this. \<\[2019-02-15T08:25:09,520\]\[ERROR\]\[logstash.pipeline \] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"\ **Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 1:38pm UTC](https://discuss.elastic.co/t/codec-error/168511 "2019-02-15T13:38:52Z") How do I codec Portuguese strings? I am getting the following warning in logstash: 019-02-15T06:12:27,095\]\[WARN \]\[logstash.codecs.plain \] Received an event that has a different character encoding than you configured… --- ## [Extract field from message field with in logs](https://discuss.elastic.co/t/extract-field-from-message-field-with-in-logs/167908)
**Author:** [@ray\_zuniga](https://discuss.elastic.co/u/ray_zuniga)\ **Replies:** 7\ **Last updated:** [February 15, 2019, 1:20pm UTC](https://discuss.elastic.co/t/extract-field-from-message-field-with-in-logs/167908 "2019-02-15T13:20:32Z")
im working with an elk stack that I did not deploy. I have an "index" for dns logs. it gives me a few default fields. The message field is given, but I noticed that the "client" and "query" with in it are things I would… --- ## [Http\_poller schedule =\> # SETTING MISSING](https://discuss.elastic.co/t/http-poller-schedule-setting-missing/168570)
**Author:** [@Dang\_Van\_Chuyen](https://discuss.elastic.co/u/Dang_Van_Chuyen)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 1:19pm UTC](https://discuss.elastic.co/t/http-poller-schedule-setting-missing/168570 "2019-02-15T13:19:45Z")
Just upgraded my logstash from 6.4 to 6.6 and http\_poller plugin to latest version then I get this problem \[LogStash::Runner\] http\_poller - Missing a required setting for the http\_poller input plugin: input { http\_… --- ## [Decode JWT toekn into new filed](https://discuss.elastic.co/t/decode-jwt-toekn-into-new-filed/168593)
**Author:** [@yossiv](https://discuss.elastic.co/u/yossiv)\ **Replies:** 0\ **Last updated:** [February 15, 2019, 1:15pm UTC](https://discuss.elastic.co/t/decode-jwt-toekn-into-new-filed/168593 "2019-02-15T13:15:12Z")
Hi Guys. i have a logstash system, latest version. and i have events that containe JWT token . i wonder if there is any way to decode JWT token to readble format ? thanks --- ## [Logstash ruby filter conditions and referrence](https://discuss.elastic.co/t/logstash-ruby-filter-conditions-and-referrence/168160)
**Author:** [@fedwe](https://discuss.elastic.co/u/fedwe)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 8:44am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-conditions-and-referrence/168160 "2019-02-15T08:44:10Z")
hello i am still new in ruby coding and i want to ask how can i apply these conditions in ruby and my case is that i receive different numbers from different countries like +962123456 or 00962123456 or 962123456.i have … --- ## [How can I reduce Ram memory usage of logstash?](https://discuss.elastic.co/t/how-can-i-reduce-ram-memory-usage-of-logstash/168544)
**Author:** [@programmerist](https://discuss.elastic.co/u/programmerist)\ **Replies:** 0\ **Last updated:** [February 15, 2019, 8:15am UTC](https://discuss.elastic.co/t/how-can-i-reduce-ram-memory-usage-of-logstash/168544 "2019-02-15T08:15:06Z")
I am looking for an answer for my logstash ram memory problem because it is nearly 100%. I made lots of search for it but they didn't work out for me. Below code is my logstash.cong file. I think it needs small touches. … --- ## [How to have multiple ids in elapsed filter?](https://discuss.elastic.co/t/how-to-have-multiple-ids-in-elapsed-filter/168524)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 8:08am UTC](https://discuss.elastic.co/t/how-to-have-multiple-ids-in-elapsed-filter/168524 "2019-02-15T08:08:52Z")
hi there! i use elapsed filter to have elapsed time of round txn and dropout count. now i can only add one unique\_id\_field, actually it's not enough for my customization is there anyway to add multiple ids in elapsed … --- ## [Logstash reject client connection](https://discuss.elastic.co/t/logstash-reject-client-connection/168538)
**Author:** [@walkexk](https://discuss.elastic.co/u/walkexk)\ **Replies:** 1\ **Last updated:** [February 15, 2019, 7:51am UTC](https://discuss.elastic.co/t/logstash-reject-client-connection/168538 "2019-02-15T07:51:54Z")
pcap file is in https://www.makemusic-asia.com/tcp5004.pcap tracelog is in https://www.makemusic-asia.com/logstash\_trace.log there is two WARN log and one ERROR log, all of them is after i stop logstash. --- ## [JDBC Fetch size VS JDBC Page Size](https://discuss.elastic.co/t/jdbc-fetch-size-vs-jdbc-page-size/168530)
**Author:** [@kelvink](https://discuss.elastic.co/u/kelvink)\ **Replies:** 0\ **Last updated:** [February 15, 2019, 6:40am UTC](https://discuss.elastic.co/t/jdbc-fetch-size-vs-jdbc-page-size/168530 "2019-02-15T06:40:26Z")
Hi there, We're trying to understand what is the difference between the jdbc\_page\_size parameter and jdbc\_fetch\_size, for logstash input jdbc plugin . What I understood from testing is that jdbc\_page\_size determines the… --- ## [How secure Logstash keystore to keep Database Connection credentials](https://discuss.elastic.co/t/how-secure-logstash-keystore-to-keep-database-connection-credentials/166462)
**Author:** [@Sumesh\_p\_s](https://discuss.elastic.co/u/Sumesh_p_s)\ **Replies:** 5\ **Last updated:** [February 15, 2019, 6:34am UTC](https://discuss.elastic.co/t/how-secure-logstash-keystore-to-keep-database-connection-credentials/166462 "2019-02-15T06:34:18Z")
Hi All, We have deployed ELK platinum and uses Logstash keystore to fetch DB logs and statistics. What are the exploit vector that some one can read the DB passwords from Keystore ? or is it safe to assume passwords are… --- ## [Logstash read csv column problem](https://discuss.elastic.co/t/logstash-read-csv-column-problem/168335)
**Author:** [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Replies:** 2\ **Last updated:** [February 15, 2019, 5:13am UTC](https://discuss.elastic.co/t/logstash-read-csv-column-problem/168335 "2019-02-15T05:13:35Z")
I added autodetect\_column\_names to the csv block on the filter to automatically detect the head value, but the index on kibana appears column2, column3 ... , what is the cause? config nput { file { … --- ## [Logstash aggregate filter plugin](https://discuss.elastic.co/t/logstash-aggregate-filter-plugin/168022)
**Author:** [@ozmhsh](https://discuss.elastic.co/u/ozmhsh)\ **Replies:** 3\ **Last updated:** [February 14, 2019, 11:16pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-plugin/168022 "2019-02-14T23:16:43Z")
Hi Everyone, I have a log file to parse as below: INFO - ID1 - 2019/01/01 INFO - ID2 Other logs 2019/01/02 INFO - ID3 - 2019/01/03 How can I use Aggregate Filter plugin to produce below? ID1 2019/01/01 ID2 2019/0… --- ## [Logstash high CPU % on startup ~ 190%](https://discuss.elastic.co/t/logstash-high-cpu-on-startup-190/168490)
**Author:** [@stash2logs](https://discuss.elastic.co/u/stash2logs)\ **Replies:** 3\ **Last updated:** [February 14, 2019, 10:21pm UTC](https://discuss.elastic.co/t/logstash-high-cpu-on-startup-190/168490 "2019-02-14T22:21:53Z")
I installed a fresh instance of Logstash with basic configurations for testing (no filters) just an input and an output. As soon as I start Logstash the CPU percentage shoots up to ~200%. Is this the normal behavior for … --- ## [How to Collect logs from different docker container and how to segregate them using index in logstash](https://discuss.elastic.co/t/how-to-collect-logs-from-different-docker-container-and-how-to-segregate-them-using-index-in-logstash/168412)
**Author:** [@amiya-santra](https://discuss.elastic.co/u/amiya-santra)\ **Replies:** 0\ **Last updated:** [February 14, 2019, 12:23pm UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-different-docker-container-and-how-to-segregate-them-using-index-in-logstash/168412 "2019-02-14T12:23:23Z")
how to Collect logs from different docker container and how to segregate them using index in logstash i am new to elk stack, i want to segregate different docker container logs in different index . how to configure log… --- ## [Define an index for weblogs being sent from Filebeat \> Logstash \> Elasticsearch](https://discuss.elastic.co/t/define-an-index-for-weblogs-being-sent-from-filebeat-logstash-elasticsearch/168352)
**Author:** [@bradfordaemorton](https://discuss.elastic.co/u/bradfordaemorton)\ **Replies:** 3\ **Last updated:** [February 14, 2019, 9:14pm UTC](https://discuss.elastic.co/t/define-an-index-for-weblogs-being-sent-from-filebeat-logstash-elasticsearch/168352 "2019-02-14T21:14:20Z")
How and where do I define an index for the weblogs being sent from Filebeat. Currently they are arriving as logstash-\[thedate\] which is not ideal. --- ## [Logstash enrichment](https://discuss.elastic.co/t/logstash-enrichment/166765)
**Author:** [@pavankup](https://discuss.elastic.co/u/pavankup)\ **Replies:** 5\ **Last updated:** [February 14, 2019, 8:33pm UTC](https://discuss.elastic.co/t/logstash-enrichment/166765 "2019-02-14T20:33:21Z")
Hi everyone, I have date on the first line of my log . The subsequent lines only contain time. I would need to prefix date to each of the subsequent lines . Please suggest an enrichment plugin that fits my requirement.… --- ## [Cant parse multiline log](https://discuss.elastic.co/t/cant-parse-multiline-log/168472)
**Author:** [@Mrcow](https://discuss.elastic.co/u/Mrcow)\ **Replies:** 4\ **Last updated:** [February 14, 2019, 7:57pm UTC](https://discuss.elastic.co/t/cant-parse-multiline-log/168472 "2019-02-14T19:57:56Z")
i have a one kind of multiline log like this and i dont know how to parse it, all the lines after the first always is the same but can be any number of lines 2019/02/14 11:54:33 BGP: 172.16.10.254 rcvd UPDATE w/ attr: … --- ## [Logstash makes new columns and gives different outputs on the same data](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411)
**Author:** [@ablaas](https://discuss.elastic.co/u/ablaas)\ **Replies:** 2\ **Last updated:** [February 14, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411 "2019-02-14T16:58:57Z")
Hi, I am using logstash with the CSV plugin to put data in ES. Problem is that logstash is generating extra columns while I don't see any extra fields in the CSV file. Moreover, when I run logstash over the same csv i… --- ## [Grok: how to handle not found patterns](https://discuss.elastic.co/t/grok-how-to-handle-not-found-patterns/168323)
**Author:** [@shimpi](https://discuss.elastic.co/u/shimpi)\ **Replies:** 1\ **Last updated:** [February 14, 2019, 2:49pm UTC](https://discuss.elastic.co/t/grok-how-to-handle-not-found-patterns/168323 "2019-02-14T14:49:24Z")
Hello Members, I'm receiving this string as input event: 2019-02-13 19:20:07,659 INFO 00ACDD5C000BAFFDC4D1353F49568680 \[main\] TEST (some\_text) and I'm using pretty simple pattern: grok { # grab the date, log le… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=599) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=601)