# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=255 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 256 --- ## [Aggregate filter, error when same task\_id in two separate aggregate filters](https://discuss.elastic.co/t/aggregate-filter-error-when-same-task-id-in-two-separate-aggregate-filters/264151)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 3\ **Last updated:** [February 12, 2021, 9:10pm UTC](https://discuss.elastic.co/t/aggregate-filter-error-when-same-task-id-in-two-separate-aggregate-filters/264151 "2021-02-12T21:10:06Z")
Hi, I'm running a Logstash 7.9.2 Docker container inside a Kubernetes cluster and I'm seeing the following error message in Logstash when I attempt to use the same task\_id in two separate aggregate filters: \[ERROR\] 20… --- ## [Logstash aggregate syslog entries](https://discuss.elastic.co/t/logstash-aggregate-syslog-entries/263783)
**Author:** [@Pablo\_Marques](https://discuss.elastic.co/u/Pablo_Marques)\ **Replies:** 12\ **Last updated:** [February 12, 2021, 8:08pm UTC](https://discuss.elastic.co/t/logstash-aggregate-syslog-entries/263783 "2021-02-12T20:08:25Z")
Hello I am getting syslogs from Clearpass servers and using logstash to ingest them into elasticsearch. Some of these messages are received on multiple syslog packets, but they are really the same (big) message. I am … --- ## [MalformedCSVError: Illegal quoting in line 1](https://discuss.elastic.co/t/malformedcsverror-illegal-quoting-in-line-1/264048)
**Author:** [@checknew](https://discuss.elastic.co/u/checknew)\ **Replies:** 5\ **Last updated:** [February 12, 2021, 8:07pm UTC](https://discuss.elastic.co/t/malformedcsverror-illegal-quoting-in-line-1/264048 "2021-02-12T20:07:27Z")
Hi all, I am seeing WARNING messages when csv is pushed to logstash, I want to eliminate the WARNING messages, any help appreciated: Sample csv: 20185656,2021-02-01 17:52:47,2021-02-01 18:23:15,"Hi Test2, this is ch… --- ## [How do I find which event is generating this error](https://discuss.elastic.co/t/how-do-i-find-which-event-is-generating-this-error/264166)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 0\ **Last updated:** [February 12, 2021, 6:07pm UTC](https://discuss.elastic.co/t/how-do-i-find-which-event-is-generating-this-error/264166 "2021-02-12T18:07:29Z")
I have thousands of event coming from metricbeat to logstash. then I do some parsing. but every on/off I getting following errors. is there a easy way to find out which event is it? An exceptionCaught() event was fire… --- ## [Logstash JDBC file permission errors, cannot send data to elastic](https://discuss.elastic.co/t/logstash-jdbc-file-permission-errors-cannot-send-data-to-elastic/263810)
**Author:** [@wlbsxnlp22](https://discuss.elastic.co/u/wlbsxnlp22)\ **Replies:** 5\ **Last updated:** [February 12, 2021, 3:43pm UTC](https://discuss.elastic.co/t/logstash-jdbc-file-permission-errors-cannot-send-data-to-elastic/263810 "2021-02-12T15:43:28Z")
Hi, I keep getting errors when running JDBC, it seems to complain about being unable to read to files on disc but I have checked the permissions and there is write access. The problem seems to become worse when having a… --- ## [XML load with Logstash into Elasticsearch](https://discuss.elastic.co/t/xml-load-with-logstash-into-elasticsearch/263767)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 4\ **Last updated:** [February 12, 2021, 3:28pm UTC](https://discuss.elastic.co/t/xml-load-with-logstash-into-elasticsearch/263767 "2021-02-12T15:28:06Z")
Hi , I have tried to create a load process with Logstash from an XML file to Elasticsearch. i use ELK 7.9.0 on windows. this is my config file : input { file { path =\> "C:/Talend/workspace/data/giata/geography/geog… --- ## [Persistent Queueing Performance](https://discuss.elastic.co/t/persistent-queueing-performance/264149)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [February 12, 2021, 3:15pm UTC](https://discuss.elastic.co/t/persistent-queueing-performance/264149 "2021-02-12T15:15:39Z")
Hi, As I understand it, persistent queue operations are single threaded. And that tallies with what I've seen during performance tests. Can I ask, are there plans to make persistent queues multi-threaded? If so, how far… --- ## [Unable to run logstash 7.10 on custom java path](https://discuss.elastic.co/t/unable-to-run-logstash-7-10-on-custom-java-path/264130)
**Author:** [@Suddhasil\_Sarkar](https://discuss.elastic.co/u/Suddhasil_Sarkar)\ **Replies:** 0\ **Last updated:** [February 12, 2021, 12:59pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-10-on-custom-java-path/264130 "2021-02-12T12:59:46Z")
Hi ELK Gurus, Need help- I was trying to run my logstash with customized JVM which is not happening. I set JAVA\_HOME pointing to my custom path "/app/platform/java/" but logstash is running with default java 11 - "/usr/… --- ## [Split an array of arrays](https://discuss.elastic.co/t/split-an-array-of-arrays/263724)
**Author:** [@opellulo](https://discuss.elastic.co/u/opellulo)\ **Replies:** 4\ **Last updated:** [February 12, 2021, 10:52am UTC](https://discuss.elastic.co/t/split-an-array-of-arrays/263724 "2021-02-12T10:52:00Z")
Hi all, I'm currently working on parsing a really nasty xml stream: events come in single file form, can get up to 5000 lines and, most important, includes arrays of arrays which crucially hold the data I need to parse … --- ## [Logstash input via website user file upload](https://discuss.elastic.co/t/logstash-input-via-website-user-file-upload/263994)
**Author:** [@Daniel\_Jankech](https://discuss.elastic.co/u/Daniel_Jankech)\ **Replies:** 2\ **Last updated:** [February 12, 2021, 10:16am UTC](https://discuss.elastic.co/t/logstash-input-via-website-user-file-upload/263994 "2021-02-12T10:16:54Z")
Hi, Im very new to the whole ELK stack and Im still learning. For my current project Im required to process some MongoDB log files uploaded by user through our website and I have trouble figuring out how I should configu… --- ## [Parsing USG Pro Firewall logs using GROK](https://discuss.elastic.co/t/parsing-usg-pro-firewall-logs-using-grok/264078)
**Author:** [@jaysbeekay](https://discuss.elastic.co/u/jaysbeekay)\ **Replies:** 2\ **Last updated:** [February 12, 2021, 4:36am UTC](https://discuss.elastic.co/t/parsing-usg-pro-firewall-logs-using-grok/264078 "2021-02-12T04:36:31Z")
I have managed to stand up an ELK stack on an Ubuntu host to bring in my NGINX logs, which is all working well. I have now tried to expand this to bring in firewall logs from my USG Pro Firewall and am running into some… --- ## [Logstash kv field\_split regex](https://discuss.elastic.co/t/logstash-kv-field-split-regex/264058)
**Author:** [@newmember](https://discuss.elastic.co/u/newmember)\ **Replies:** 2\ **Last updated:** [February 12, 2021, 1:10am UTC](https://discuss.elastic.co/t/logstash-kv-field-split-regex/264058 "2021-02-12T01:10:26Z")
I am using logstash 2.4.0 for business reasons. Does the field\_split split on each of these characters: (,\\r,\\n,|,\\n,),\\t OR using regex does the split recognize the OR option in the brackets? (\\r\\n|\\n)\\t kv { … --- ## [Syslog cef + logstash + security device logs parsing issues](https://discuss.elastic.co/t/syslog-cef-logstash-security-device-logs-parsing-issues/264070)
**Author:** [@Xor44](https://discuss.elastic.co/u/Xor44)\ **Replies:** 0\ **Last updated:** [February 12, 2021, 12:39am UTC](https://discuss.elastic.co/t/syslog-cef-logstash-security-device-logs-parsing-issues/264070 "2021-02-12T00:39:47Z")
Hello , I'm beginning with logstash, I need a help to understand why CEF logs sent by security equipment are not parsed correctly. I am using the filter below but some fields are not named correctly. here is an sample … --- ## [Azure Activity logs issue](https://discuss.elastic.co/t/azure-activity-logs-issue/264061)
**Author:** [@renadoz](https://discuss.elastic.co/u/renadoz)\ **Replies:** 0\ **Last updated:** [February 11, 2021, 10:35pm UTC](https://discuss.elastic.co/t/azure-activity-logs-issue/264061 "2021-02-11T22:35:40Z")
Dear Community, I am having an issue with getting azure activity logs in Kibana in a proper format. It is like azure activity logs are not fully reflected on kibana side. Although it is the same log, several fields ar… --- ## [Grok for time with UUUU](https://discuss.elastic.co/t/grok-for-time-with-uuuu/264040)
**Author:** [@newmember](https://discuss.elastic.co/u/newmember)\ **Replies:** 2\ **Last updated:** [February 11, 2021, 9:57pm UTC](https://discuss.elastic.co/t/grok-for-time-with-uuuu/264040 "2021-02-11T21:57:28Z")
I am reading this in the grok patterns pages but I dont see a syntax for the time format. # datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it) Is there a grok pattern for this time format? 10:02:19.9830 --- ## [Long and float fields showing up as text fields in Kibana](https://discuss.elastic.co/t/long-and-float-fields-showing-up-as-text-fields-in-kibana/263921)
**Author:** [@asnyameeteen](https://discuss.elastic.co/u/asnyameeteen)\ **Replies:** 0\ **Last updated:** [February 10, 2021, 6:37pm UTC](https://discuss.elastic.co/t/long-and-float-fields-showing-up-as-text-fields-in-kibana/263921 "2021-02-10T18:37:49Z")
Running Kibana version 5.5.2. My current setup is Logstash is taking the logs from Docker containers, runs grok filters before sending the logs to elasticsearch. The specific logs that I need to show up as long, float a… --- ## [Translate Filter plugin for a real CSV](https://discuss.elastic.co/t/translate-filter-plugin-for-a-real-csv/263940)
**Author:** [@jchaves506](https://discuss.elastic.co/u/jchaves506)\ **Replies:** 3\ **Last updated:** [February 11, 2021, 8:02pm UTC](https://discuss.elastic.co/t/translate-filter-plugin-for-a-real-csv/263940 "2021-02-11T20:02:10Z")
Hello I have a big CSV file(10 columns, 40Mb) that I want to use as a dictionary using the translate filter. I'm already doing that with a little csv files (2 columns, 4kb) and that is working fine. I'm doing it like th… --- ## [Mysql schema logs push to logstash](https://discuss.elastic.co/t/mysql-schema-logs-push-to-logstash/264016)
**Author:** [@psganeshk](https://discuss.elastic.co/u/psganeshk)\ **Replies:** 2\ **Last updated:** [February 11, 2021, 7:59pm UTC](https://discuss.elastic.co/t/mysql-schema-logs-push-to-logstash/264016 "2021-02-11T19:59:19Z")
Logstash config file '' input { jdbc { clean\_run =\> true jdbc\_validate\_connection =\> true jdbc\_driver\_library =\> 'C:\\Program Files (x86)\\MySQL\\Connector J 8 0\\mysql-connector-java-8.0.22.jar' jdbc\_driver\_class =\> … --- ## [URGENT.. PLEASE HELP : Not able to execute a procedure from logstash jdbc\_filter](https://discuss.elastic.co/t/urgent-please-help-not-able-to-execute-a-procedure-from-logstash-jdbc-filter/263858)
**Author:** [@Melvin\_Shaju](https://discuss.elastic.co/u/Melvin_Shaju)\ **Replies:** 1\ **Last updated:** [February 11, 2021, 7:56pm UTC](https://discuss.elastic.co/t/urgent-please-help-not-able-to-execute-a-procedure-from-logstash-jdbc-filter/263858 "2021-02-11T19:56:42Z")
I'm getting this following error when trying to execute a mysql procedure call from logstash. This procedure is getting successfully executed through mysql command line. The error is jdbcstreaming - Exception when exec… --- ## [Logstash config for parsing ASA logs](https://discuss.elastic.co/t/logstash-config-for-parsing-asa-logs/262563)
**Author:** [@psycadelicgecko](https://discuss.elastic.co/u/psycadelicgecko)\ **Replies:** 2\ **Last updated:** [February 11, 2021, 5:00pm UTC](https://discuss.elastic.co/t/logstash-config-for-parsing-asa-logs/262563 "2021-02-11T17:00:27Z")
So I am brand new to ELK/Elastic so I'm sure I'm doing at least a few things wrong. I was able to find a couple examples of Logstash config files for parsing ASA logs. They are similar to what I've seen on this forum. S… --- ## [Grok by beats agent](https://discuss.elastic.co/t/grok-by-beats-agent/264008)
**Author:** [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Replies:** 2\ **Last updated:** [February 11, 2021, 2:12pm UTC](https://discuss.elastic.co/t/grok-by-beats-agent/264008 "2021-02-11T14:12:27Z")
Hello :smiley: I have several beats entries on my logstash, but the logs I process are different, so I would like to do a grok by beat. example log lines : 2017/07/27 18:02:37 VCS ERROR V-16-1-54031 Resource XXXXXXXXX… --- ## [Appending Random number in csv file name](https://discuss.elastic.co/t/appending-random-number-in-csv-file-name/263575)
**Author:** [@ritusingh](https://discuss.elastic.co/u/ritusingh)\ **Replies:** 7\ **Last updated:** [February 11, 2021, 8:19am UTC](https://discuss.elastic.co/t/appending-random-number-in-csv-file-name/263575 "2021-02-11T08:19:59Z")
Hi, I want to append random number after the file name. output { stdout { codec =\> rubydebug } csv { # elastic field name fields =\> \["@timestamp","requestid","ngnix.responsebytes","ngnix.cpu.usage"\] #… --- ## [Logstash Data Persistency](https://discuss.elastic.co/t/logstash-data-persistency/263948)
**Author:** [@leventyalcin](https://discuss.elastic.co/u/leventyalcin)\ **Replies:** 0\ **Last updated:** [February 11, 2021, 4:21am UTC](https://discuss.elastic.co/t/logstash-data-persistency/263948 "2021-02-11T04:21:52Z")
Hi, As you might know, in-flight queue is hold in memory by Logstash. I'm trying to enable data persistency with a persistent queue. However, restarting logstash or the instance wipes out the data. When I saw that I t… --- ## [Remove garbage from message end](https://discuss.elastic.co/t/remove-garbage-from-message-end/263916)
**Author:** [@John\_Smith](https://discuss.elastic.co/u/John_Smith)\ **Replies:** 3\ **Last updated:** [February 10, 2021, 11:19pm UTC](https://discuss.elastic.co/t/remove-garbage-from-message-end/263916 "2021-02-10T23:19:13Z")
Hello, I've following message send by filebeat: "message" =\> "{"country-code":"GB","payload":"{\\"access\_token\\":\\"\*\*\*\*\*\*\*\*\*\*\\",\\"token\_type\\":\\"Bearer\\",}","status":"ok","status-code":200,"error":"","content-type":"app… --- ## [Is it possible to split a logstash file output into a random number of output files for throughput reasons?](https://discuss.elastic.co/t/is-it-possible-to-split-a-logstash-file-output-into-a-random-number-of-output-files-for-throughput-reasons/263909)
**Author:** [@jethropickering](https://discuss.elastic.co/u/jethropickering)\ **Replies:** 1\ **Last updated:** [February 10, 2021, 6:22pm UTC](https://discuss.elastic.co/t/is-it-possible-to-split-a-logstash-file-output-into-a-random-number-of-output-files-for-throughput-reasons/263909 "2021-02-10T18:22:24Z")
Hi all. We are writing to file at about 5000-20000 eps at times. logstash is able to do this fine. The reason we need to write to file and not an endpoint is because of a custom solution(that can't be changed) that read… --- ## [Cron scheduler for jdbc plugin](https://discuss.elastic.co/t/cron-scheduler-for-jdbc-plugin/263878)
**Author:** [@Bryce\_Fernandes](https://discuss.elastic.co/u/Bryce_Fernandes)\ **Replies:** 10\ **Last updated:** [February 10, 2021, 3:14pm UTC](https://discuss.elastic.co/t/cron-scheduler-for-jdbc-plugin/263878 "2021-02-10T15:14:41Z")
Hi, I am using jdbc plugin and schedule for fetching data from database. Currently cron is running fine for schedule =\> "\* \* \* \* \*". My requirement is cron ingestion should start at given time (10 pm) and end at given … --- ## [Mutate add\_field and rename don't work](https://discuss.elastic.co/t/mutate-add-field-and-rename-dont-work/263868)
**Author:** [@Ernesto\_Guerra](https://discuss.elastic.co/u/Ernesto_Guerra)\ **Replies:** 2\ **Last updated:** [February 10, 2021, 2:43pm UTC](https://discuss.elastic.co/t/mutate-add-field-and-rename-dont-work/263868 "2021-02-10T14:43:31Z")
Hi, I'm trying to use mutate filter to change the name of a field. I've tried rename but it doesn't do anything, and tried creating a new field and passing the contents of the old one and didn't work either. My first t… --- ## [Logstash filter](https://discuss.elastic.co/t/logstash-filter/263718)
**Author:** [@Sujithra\_S](https://discuss.elastic.co/u/Sujithra_S)\ **Replies:** 3\ **Last updated:** [February 10, 2021, 2:35pm UTC](https://discuss.elastic.co/t/logstash-filter/263718 "2021-02-10T14:35:01Z")
Hi, I am fetching the data from Azure - event hubs , I need to split the data which is in message field and i have tried split filter and grok filter it is showing the error which is in tags. Please help me to sort out… --- ## [Multiple configuration files](https://discuss.elastic.co/t/multiple-configuration-files/263880)
**Author:** [@Sunflower](https://discuss.elastic.co/u/Sunflower)\ **Replies:** 0\ **Last updated:** [February 10, 2021, 1:45pm UTC](https://discuss.elastic.co/t/multiple-configuration-files/263880 "2021-02-10T13:45:03Z")
Hi all, I have one conf file with multiple inputs and outputs(based on types) and I'd like to make it easy to read and to manage so I thought to create multiple configuration files, one for each type. Is there anything… --- ## [Fetch the data from multiple indexes and create one report](https://discuss.elastic.co/t/fetch-the-data-from-multiple-indexes-and-create-one-report/263867)
**Author:** [@mrunalini](https://discuss.elastic.co/u/mrunalini)\ **Replies:** 0\ **Last updated:** [February 10, 2021, 12:18pm UTC](https://discuss.elastic.co/t/fetch-the-data-from-multiple-indexes-and-create-one-report/263867 "2021-02-10T12:18:48Z")
Hi Team, I have 3 different index with one transaction\_id as common in all, i am generation report with fields from all 3 indices at every midnight for transactions of last 3 days. I have different fields in all indice… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=254) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=256)