# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=597 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 598 --- ## [Use grok to filter mysql slow-queries](https://discuss.elastic.co/t/use-grok-to-filter-mysql-slow-queries/169248)
**Author:** [@dank](https://discuss.elastic.co/u/dank)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 4:17pm UTC](https://discuss.elastic.co/t/use-grok-to-filter-mysql-slow-queries/169248 "2019-02-20T16:17:50Z")
Hello all, I'm having a hard time to filter my MySQL slow-queries using logstash (version 6.6.1). I'm using the following grok filter : input { file { path =\> \["/tmp/testFile.txt"\] type =\> "mysql" codec =\> multili… --- ## [Can logstash replace local syslog server?](https://discuss.elastic.co/t/can-logstash-replace-local-syslog-server/169121)
**Author:** [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 3:56pm UTC](https://discuss.elastic.co/t/can-logstash-replace-local-syslog-server/169121 "2019-02-20T15:56:18Z")
Im wondering if its possible to run a logstash service listening on port 514 and forward external syslog data to elastic through it. My issue is that I have SANs and switches that won't let me specify a non-standard (IE … --- ## [Sql\_last\_value and Tracking files are no longer updated](https://discuss.elastic.co/t/sql-last-value-and-tracking-files-are-no-longer-updated/169208)
**Author:** [@Roger\_Lavoie](https://discuss.elastic.co/u/Roger_Lavoie)\ **Replies:** 2\ **Last updated:** [February 20, 2019, 3:30pm UTC](https://discuss.elastic.co/t/sql-last-value-and-tracking-files-are-no-longer-updated/169208 "2019-02-20T15:30:06Z")
Hello all, I am using Logstash 6.5.4 and started pulling data from my SQL Server, the first 50,000 records went in fine by batches of 10,000 at a time, but after a couple of hours.. everything stopped updating, the track… --- ## [How to index multiple CSV files to visulize in Kibana?](https://discuss.elastic.co/t/how-to-index-multiple-csv-files-to-visulize-in-kibana/169016)
**Author:** [@akshayprabhu80](https://discuss.elastic.co/u/akshayprabhu80)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 3:05pm UTC](https://discuss.elastic.co/t/how-to-index-multiple-csv-files-to-visulize-in-kibana/169016 "2019-02-20T15:05:58Z")
Hi, I want to create chart in Kibana using two CSV files. Can somebody help me how to do it? I tried my level best. But not finding any solution. Thanks in advance. Regards, Akshay --- ## [Logstash ruby filter condition error](https://discuss.elastic.co/t/logstash-ruby-filter-condition-error/169005)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 3:04pm UTC](https://discuss.elastic.co/t/logstash-ruby-filter-condition-error/169005 "2019-02-20T15:04:35Z")
whenever I added an if condition the code doesn't work on the null offerlist field here is my filter code =\> ' if event.get("offerList") == "NULL" t = event.get("UCounter") t.each\_index { |i| … --- ## [Logsasth SQS input plugin threads](https://discuss.elastic.co/t/logsasth-sqs-input-plugin-threads/169151)
**Author:** [@ep4sh](https://discuss.elastic.co/u/ep4sh)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 2:59pm UTC](https://discuss.elastic.co/t/logsasth-sqs-input-plugin-threads/169151 "2019-02-20T14:59:10Z")
Good day, could you tell me how to properly set the parameter of Logstash input SQS plugin? Coming from of CPU cores? --- ## [Parsing Array in to Elasticsearch](https://discuss.elastic.co/t/parsing-array-in-to-elasticsearch/169210)
**Author:** [@Ash\_Powell](https://discuss.elastic.co/u/Ash_Powell)\ **Replies:** 2\ **Last updated:** [February 20, 2019, 2:41pm UTC](https://discuss.elastic.co/t/parsing-array-in-to-elasticsearch/169210 "2019-02-20T14:41:19Z")
We're looking to parse an array into Elasticsearch but I'm struggling to write some Ruby code to convert it into the way that I want. The raw data looks like this "marketingPreferences" =\> { "u… --- ## [Logstash 5.2 | Wanted to call jdbc statement(query) only when some external service/event hit to logstash like JMS](https://discuss.elastic.co/t/logstash-5-2-wanted-to-call-jdbc-statement-query-only-when-some-external-service-event-hit-to-logstash-like-jms/169236)
**Author:** [@Satish\_Darade](https://discuss.elastic.co/u/Satish_Darade)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 2:31pm UTC](https://discuss.elastic.co/t/logstash-5-2-wanted-to-call-jdbc-statement-query-only-when-some-external-service-event-hit-to-logstash-like-jms/169236 "2019-02-20T14:31:52Z")
I am stuck in logstash input JDBC plugin. I am new to elasticsearch and I wanted to load data from database to elaststcsearch index using logstash. 1st time I have successfully loaded database data into to elasticsearch… --- ## [Capture specific word from message](https://discuss.elastic.co/t/capture-specific-word-from-message/169180)
**Author:** [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Replies:** 2\ **Last updated:** [February 20, 2019, 2:22pm UTC](https://discuss.elastic.co/t/capture-specific-word-from-message/169180 "2019-02-20T14:22:11Z")
Hi, I need to create a field with the username from the message. How can i capture the username from a message and put it in a separate field? Ex: Feb 20 15:23:33 localhost sshd\[24958\]: Failed password for invalid use… --- ## [Grok filter not being triggered in Logstash server](https://discuss.elastic.co/t/grok-filter-not-being-triggered-in-logstash-server/168608)
**Author:** [@Carlos\_Bodini](https://discuss.elastic.co/u/Carlos_Bodini)\ **Replies:** 16\ **Last updated:** [February 20, 2019, 2:18pm UTC](https://discuss.elastic.co/t/grok-filter-not-being-triggered-in-logstash-server/168608 "2019-02-20T14:18:52Z")
hey guys! I have a grok filter in my Logstash server for my IIS logs... but it seams the logs are not being filter by it, and there's no errors in the logstash's logs, what would happen if the problem was in the filter … --- ## [How to convert yyyyMMddHHmmssSSSZ string into timestamp](https://discuss.elastic.co/t/how-to-convert-yyyymmddhhmmsssssz-string-into-timestamp/168596)
**Author:** [@priya08](https://discuss.elastic.co/u/priya08)\ **Replies:** 9\ **Last updated:** [February 20, 2019, 1:24pm UTC](https://discuss.elastic.co/t/how-to-convert-yyyymmddhhmmsssssz-string-into-timestamp/168596 "2019-02-20T13:24:19Z")
Hi , I want to convert date string yyyyMMddHHmmssSSSZ to timezone , I have tried with date { match =\> \["datetime", "YYYYMMddHHmmssSSS" \] target =\> "datetime" } but i am getting \_dateparsefailure in tags. How to co… --- ## [Logstash -\> Elastic search login Error](https://discuss.elastic.co/t/logstash-elastic-search-login-error/169219)
**Author:** [@macdadi112](https://discuss.elastic.co/u/macdadi112)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 1:22pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-login-error/169219 "2019-02-20T13:22:43Z")
I am trying to log into elasticsearch using the logstash user. The deployment is done via kubernetes and we have logstash pod running a pod. I log into the logstash and try the following: Curl -k -u : https://elastics… --- ## [Drop first x line from my csv log file](https://discuss.elastic.co/t/drop-first-x-line-from-my-csv-log-file/169214)
**Author:** [@cankaya07](https://discuss.elastic.co/u/cankaya07)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 1:10pm UTC](https://discuss.elastic.co/t/drop-first-x-line-from-my-csv-log-file/169214 "2019-02-20T13:10:04Z")
Hi, i have a csv log file. and i want to parse and send to ES that file via logstash. But my csv file has 3 lines of comment line. Here the shape of my log #Version: 15.01.1034.026 #Log-type: Message Tracking Log #Date… --- ## [\[logstash.filters.jdbc.lookup\] Parameter field not found in event {:lookup\_id=\>](https://discuss.elastic.co/t/logstash-filters-jdbc-lookup-parameter-field-not-found-in-event-lookup-id/169031)
**Author:** [@josep.clavero](https://discuss.elastic.co/u/josep.clavero)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 9:42am UTC](https://discuss.elastic.co/t/logstash-filters-jdbc-lookup-parameter-field-not-found-in-event-lookup-id/169031 "2019-02-20T09:42:24Z")
Hi, I want to enrich some log data with a table in oracle. \[2019-02-19T14:06:29,775\]\[WARN \]\[logstash.filters.jdbc.lookup\] Parameter field not found in event {:lookup\_id=\>"local\_integration\_descr", :invalid\_parameters=\>\[… --- ## [Can logstash convert date format?](https://discuss.elastic.co/t/can-logstash-convert-date-format/169166)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 8:57am UTC](https://discuss.elastic.co/t/can-logstash-convert-date-format/169166 "2019-02-20T08:57:23Z")
hi all, can logstash convert input shamsi date to the Gregorian format?. many thanks. --- ## [Logstash service keeps pausing](https://discuss.elastic.co/t/logstash-service-keeps-pausing/169165)
**Author:** [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 8:56am UTC](https://discuss.elastic.co/t/logstash-service-keeps-pausing/169165 "2019-02-20T08:56:35Z")
I installed Logstash on Windows Server 2012R2 using the steps referenced here : http://robwillis.info/2016/05/installing-elasticsearch-logstash-and-kibana-elk-on-windows-server-2012-r2/ The service was installed succesf… --- ## [Logstash mutate can't convert 2 fields at once](https://discuss.elastic.co/t/logstash-mutate-cant-convert-2-fields-at-once/168947)
**Author:** [@ksemaev](https://discuss.elastic.co/u/ksemaev)\ **Replies:** 3\ **Last updated:** [February 20, 2019, 8:52am UTC](https://discuss.elastic.co/t/logstash-mutate-cant-convert-2-fields-at-once/168947 "2019-02-20T08:52:20Z")
When I have both a.b and c.d in one log entry, and try to mutate-convert them, only first conversion occures, and the second is not, so it causes logstash to throw failed to parse field warning. Current config is: filt… --- ## [Using Elastic stack with Kafka for reading system metrics](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032)
**Author:** [@astrovj](https://discuss.elastic.co/u/astrovj)\ **Replies:** 2\ **Last updated:** [February 20, 2019, 8:43am UTC](https://discuss.elastic.co/t/using-elastic-stack-with-kafka-for-reading-system-metrics/169032 "2019-02-20T08:43:19Z")
Hello, Below is the flow of my Elastic stack (setup1) metricbeat (system module) -\> kafka cluster -\> logstash -\> elasticsearch -\> kibana In kibana I get the metrics in form of a single message like below message:{"@t… --- ## [Reading value from file and using it dynamically in JDBC select query where clause](https://discuss.elastic.co/t/reading-value-from-file-and-using-it-dynamically-in-jdbc-select-query-where-clause/169134)
**Author:** [@ApekshaB](https://discuss.elastic.co/u/ApekshaB)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 5:55am UTC](https://discuss.elastic.co/t/reading-value-from-file-and-using-it-dynamically-in-jdbc-select-query-where-clause/169134 "2019-02-20T05:55:09Z")
I want to use 'file' filter, read a value from that file, and in same conf file, i need to use that value in jdbc query to Database to fetch data rows depending on the value retrieved from the file. Is is possible to do… --- ## [How to get multiple layers of json data one by one](https://discuss.elastic.co/t/how-to-get-multiple-layers-of-json-data-one-by-one/169141)
**Author:** [@okaaaa](https://discuss.elastic.co/u/okaaaa)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 6:42am UTC](https://discuss.elastic.co/t/how-to-get-multiple-layers-of-json-data-one-by-one/169141 "2019-02-20T06:42:41Z")
I am from China, and my English is not very good.Now, I need some help, I hope I can get your help.:grin: Indicate that I am a newbie.:joy::joy::joy: I recently used logstash to read the json log file, then output it t… --- ## [Netflow -\> ELK, poor performance](https://discuss.elastic.co/t/netflow-elk-poor-performance/169133)
**Author:** [@otoce](https://discuss.elastic.co/u/otoce)\ **Replies:** 0\ **Last updated:** [February 20, 2019, 5:38am UTC](https://discuss.elastic.co/t/netflow-elk-poor-performance/169133 "2019-02-20T05:38:15Z")
Hi, our setup (pilot): 10 neflow exporters (cca 10k events/sec) to logstash with netflow modul to elastic. In the beginning (after restart logstash) everything is ok but after 10 min input is droping to 1k events/sec (k… --- ## [Logstash zabbix output plugin error said error=\>#\ **Author:** [@xemina](https://discuss.elastic.co/u/xemina)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 3:26am UTC](https://discuss.elastic.co/t/logstash-zabbix-output-plugin-error-said-error-zabbix-protocol-data-length-is-too-short-data/169124 "2019-02-20T03:26:04Z") Hi , I got this error if I write output with zabbix plugin filter { if "oeelog\_wmq" in \[tags\] { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:wmqtime}" } } date { match =\> \["wmqtime", "yyyy-MM-dd HH:mm:ss… --- ## [Mapper\_parsing\_exception for Income Range field?](https://discuss.elastic.co/t/mapper-parsing-exception-for-income-range-field/169109)
**Author:** [@parshvms](https://discuss.elastic.co/u/parshvms)\ **Replies:** 1\ **Last updated:** [February 20, 2019, 2:48am UTC](https://discuss.elastic.co/t/mapper-parsing-exception-for-income-range-field/169109 "2019-02-20T02:48:33Z")
Hello everyone, I am new to the community and trying to figure out a solution to one of the problems. I have a field named "Income Bracket" that has values like "$50,000 - $100,000", "$50,000 and under", "$100,000 - $5… --- ## [Trying to use GROK to filter data to different index](https://discuss.elastic.co/t/trying-to-use-grok-to-filter-data-to-different-index/169099)
**Author:** [@snovak](https://discuss.elastic.co/u/snovak)\ **Replies:** 3\ **Last updated:** [February 19, 2019, 11:44pm UTC](https://discuss.elastic.co/t/trying-to-use-grok-to-filter-data-to-different-index/169099 "2019-02-19T23:44:01Z")
We have been using GROK to filter and parse message date from the Unbound DNS server files coming in via FileBeat for some time. Now we are starting to send some other data with FileBeat and want to separate out the Unb… --- ## [Creating Data to vizualize in kibana with http\_poller and filtering in logstash](https://discuss.elastic.co/t/creating-data-to-vizualize-in-kibana-with-http-poller-and-filtering-in-logstash/168707)
**Author:** [@SuperTomcat1](https://discuss.elastic.co/u/SuperTomcat1)\ **Replies:** 6\ **Last updated:** [February 19, 2019, 11:40pm UTC](https://discuss.elastic.co/t/creating-data-to-vizualize-in-kibana-with-http-poller-and-filtering-in-logstash/168707 "2019-02-19T23:40:11Z")
I am pulling metrics with http\_poller. its got all sort of fun. so i tried to sake of ease to copy a field out of an array and put it in its own field. I want to graph it based on timestamps. But elastic is creating it a… --- ## [Little amount of logs appears in Kibana](https://discuss.elastic.co/t/little-amount-of-logs-appears-in-kibana/169015)
**Author:** [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)\ **Replies:** 3\ **Last updated:** [February 19, 2019, 8:30pm UTC](https://discuss.elastic.co/t/little-amount-of-logs-appears-in-kibana/169015 "2019-02-19T20:30:35Z")
Hi there! I have a question please. I'm using filbeat ,logstash and kibana installed both on K8S, running and working. It is important for me , to see the nginx-ingress logs from K8S. I can see the logs in kibana, bu… --- ## [Aggregation EPS and Average Elapsed Time](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018)
**Author:** [@rtb](https://discuss.elastic.co/u/rtb)\ **Replies:** 8\ **Last updated:** [February 19, 2019, 8:27pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018 "2019-02-19T20:27:41Z")
I need to create an aggregation to provide a count of EPS and Average elapsed time for an operation every second. I have the following but with my test data, I don't get any output. Below is my filter and I am running… --- ## [How to filter data with Logstash before storing parsed data in Elasticsearch](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071)
**Author:** [@Han\_Xu](https://discuss.elastic.co/u/Han_Xu)\ **Replies:** 2\ **Last updated:** [February 19, 2019, 8:02pm UTC](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071 "2019-02-19T20:02:58Z")
I understand that Logstash is for aggregating and processing logs. I have NGIX logs and had Logstash config setup as: filter { grok { match =\> \[ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}"\] over… --- ## [Trying to add ruby filter to remove leading zero](https://discuss.elastic.co/t/trying-to-add-ruby-filter-to-remove-leading-zero/168901)
**Author:** [@paano](https://discuss.elastic.co/u/paano)\ **Replies:** 5\ **Last updated:** [February 19, 2019, 5:36pm UTC](https://discuss.elastic.co/t/trying-to-add-ruby-filter-to-remove-leading-zero/168901 "2019-02-19T17:36:01Z")
I have the following the json message. { "eventData" : { "commandContext" : { "eventUserId" : "system", "eventOrigin" : "java", "eventQueueMgr" : "TEST1", "eventAccountingToken" : 000000000000000000000000000000… --- ## [Docs are stored on same index ¡¡](https://discuss.elastic.co/t/docs-are-stored-on-same-index/169060)
**Author:** [@Pablo\_Repolles](https://discuss.elastic.co/u/Pablo_Repolles)\ **Replies:** 3\ **Last updated:** [February 19, 2019, 4:30pm UTC](https://discuss.elastic.co/t/docs-are-stored-on-same-index/169060 "2019-02-19T16:30:53Z")
The trouble is that de docs are stored into both index at the same time. Instead to use each own index, one per diferent input. input { udp { host =\> "192.168.254.60" port =\> 5001 type =\> mikrotik } udp { host =\> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=596) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=598)