# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=384 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 385 --- ## [Possible to search for the specific string in filter grook then concatenate to a pattern. TIA!](https://discuss.elastic.co/t/possible-to-search-for-the-specific-string-in-filter-grook-then-concatenate-to-a-pattern-tia/225214)
**Author:** [@smebrolster](https://discuss.elastic.co/u/smebrolster)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 2:19pm UTC](https://discuss.elastic.co/t/possible-to-search-for-the-specific-string-in-filter-grook-then-concatenate-to-a-pattern-tia/225214 "2020-03-26T14:19:50Z")
--- ## [Increasing Logstash Throughput When the Codec is the Bottleneck](https://discuss.elastic.co/t/increasing-logstash-throughput-when-the-codec-is-the-bottleneck/224158)
**Author:** [@seth.yes](https://discuss.elastic.co/u/seth.yes)\ **Replies:** 11\ **Last updated:** [March 26, 2020, 2:08pm UTC](https://discuss.elastic.co/t/increasing-logstash-throughput-when-the-codec-is-the-bottleneck/224158 "2020-03-26T14:08:21Z")
Setup I have multiple machines running a containerized Logstash 7.6. These machines have 32 core and 256GB RAM. I am using these nodes to process network flow data based on Elastiflow. These nodes are listening for data … --- ## [Logstash taking so much time to index MySQL query](https://discuss.elastic.co/t/logstash-taking-so-much-time-to-index-mysql-query/225196)
**Author:** [@manideep](https://discuss.elastic.co/u/manideep)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 1:23pm UTC](https://discuss.elastic.co/t/logstash-taking-so-much-time-to-index-mysql-query/225196 "2020-03-26T13:23:46Z")
Hi ALL, I have a simple sql query with 3 joins. i am trying to index those results in ES via Logstash. i created a view and try to index it. I was following up until one hour and I left whole night. but then it indexed… --- ## [While sending iis logs from filebeat to logstash few feilds do not get populated](https://discuss.elastic.co/t/while-sending-iis-logs-from-filebeat-to-logstash-few-feilds-do-not-get-populated/225173)
**Author:** [@Sagar\_Joshi](https://discuss.elastic.co/u/Sagar_Joshi)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 11:11am UTC](https://discuss.elastic.co/t/while-sending-iis-logs-from-filebeat-to-logstash-few-feilds-do-not-get-populated/225173 "2020-03-26T11:11:55Z")
While sending IIS logs from filebeat --\> Logstash --\> ElasticSearch --\> Kibana , we lose couple of fields like (user\_agent and geo location ) even after applying GROK pattern inside logstash FILTER. Hence couple of visu… --- ## [Using date filter with field from autodetected column in csv plugin](https://discuss.elastic.co/t/using-date-filter-with-field-from-autodetected-column-in-csv-plugin/223965)
**Author:** [@rafael07](https://discuss.elastic.co/u/rafael07)\ **Replies:** 6\ **Last updated:** [March 26, 2020, 11:02am UTC](https://discuss.elastic.co/t/using-date-filter-with-field-from-autodetected-column-in-csv-plugin/223965 "2020-03-26T11:02:56Z")
Hello. I am trying to parse a date from a column, which name is autodetected using the csv plugin. The problem is that the date plugin is doing nothing with this field. If instead of using "autodetect\_column\_names" I wr… --- ## [Logstash Pipelines doesn't work when logs come from Filebeat](https://discuss.elastic.co/t/logstash-pipelines-doesnt-work-when-logs-come-from-filebeat/225162)
**Author:** [@oula\_oula](https://discuss.elastic.co/u/oula_oula)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 10:17am UTC](https://discuss.elastic.co/t/logstash-pipelines-doesnt-work-when-logs-come-from-filebeat/225162 "2020-03-26T10:17:17Z")
Hi, I've built a pipeline.conf in Logstash that works when the input is a path file (so without using filebeat shipping). But when I just modify the input, so by replacing path input with beats port 5044 (Filebeat), a… --- ## [How to forwards logs from logstash to qradar over SSL/TLS?](https://discuss.elastic.co/t/how-to-forwards-logs-from-logstash-to-qradar-over-ssl-tls/225157)
**Author:** [@Goryoun](https://discuss.elastic.co/u/Goryoun)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 10:05am UTC](https://discuss.elastic.co/t/how-to-forwards-logs-from-logstash-to-qradar-over-ssl-tls/225157 "2020-03-26T10:05:01Z")
We are planning to use logstash as a shipper to send logs to qradar. We are planning to get logstash logs as syslog to Qradar. How we can transfer the data securely? There is cipher plugin available in logstash to send … --- ## [How to slow down large amount of data coming from filebeat?](https://discuss.elastic.co/t/how-to-slow-down-large-amount-of-data-coming-from-filebeat/224970)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 5\ **Last updated:** [March 26, 2020, 9:37am UTC](https://discuss.elastic.co/t/how-to-slow-down-large-amount-of-data-coming-from-filebeat/224970 "2020-03-26T09:37:25Z")
Hi Team, How will you slow down large amount of data streaming from filebeat to logstash so that it can be processed accurately in filter section. Thanks and Regards, Sagar Mandal --- ## [Logstash - filter message](https://discuss.elastic.co/t/logstash-filter-message/225129)
**Author:** [@rockyu](https://discuss.elastic.co/u/rockyu)\ **Replies:** 2\ **Last updated:** [March 26, 2020, 9:04am UTC](https://discuss.elastic.co/t/logstash-filter-message/225129 "2020-03-26T09:04:17Z")
If my logs is 2020-03-26-12:12:36 aaa, I want to filter out the date like 2020-03-26-12:12:36 ,How can I do it ? My config is grok { match =\> {"message =\> (?\(\[0-9|-\]\\S+\[0-9|:\]\\S+)"} add\_f… --- ## [Handling different kind of logs](https://discuss.elastic.co/t/handling-different-kind-of-logs/221158)
**Author:** [@Guru\_Prasanth](https://discuss.elastic.co/u/Guru_Prasanth)\ **Replies:** 0\ **Last updated:** [February 27, 2020, 7:38am UTC](https://discuss.elastic.co/t/handling-different-kind-of-logs/221158 "2020-02-27T07:38:55Z")
Hi all, I have a requirement where i need to handle various types of logs. One option is to use filter for each kind pof logs and process them. Is there any other better options as APACHECOMMONLOGS and APACHECOMBINEDL… --- ## [Masking password in logstash config file](https://discuss.elastic.co/t/masking-password-in-logstash-config-file/225122)
**Author:** [@Thamini\_NellaiKumar](https://discuss.elastic.co/u/Thamini_NellaiKumar)\ **Replies:** 0\ **Last updated:** [March 26, 2020, 6:30am UTC](https://discuss.elastic.co/t/masking-password-in-logstash-config-file/225122 "2020-03-26T06:30:57Z")
Hi, I want to mask my password in logstash config file. I came to know that we can do this by secure key store. Is this feature available in free version of kibana? Currently am using version 7.5.1 --- ## [Yum.log file](https://discuss.elastic.co/t/yum-log-file/224249)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 2\ **Last updated:** [March 25, 2020, 9:46pm UTC](https://discuss.elastic.co/t/yum-log-file/224249 "2020-03-25T21:46:41Z")
Hi , in our company we need to work with /var/log/yum.log file , can I do that with filebeat or I have to use file input ? Thanks . --- ## [Gid numbers and kibana parsing](https://discuss.elastic.co/t/gid-numbers-and-kibana-parsing/225049)
**Author:** [@rwr1](https://discuss.elastic.co/u/rwr1)\ **Replies:** 0\ **Last updated:** [March 25, 2020, 7:12pm UTC](https://discuss.elastic.co/t/gid-numbers-and-kibana-parsing/225049 "2020-03-25T19:12:29Z")
Hey! I am using logstash with suricata and kibana. I changed the suricata gid numbers, which according to the docs, have no technical implications. However, it changed the way I see all of my alerts in Kibana. I'm going … --- ## [Looking for some guidance on Sourcefire Parsing](https://discuss.elastic.co/t/looking-for-some-guidance-on-sourcefire-parsing/224887)
**Author:** [@SaskRaven](https://discuss.elastic.co/u/SaskRaven)\ **Replies:** 6\ **Last updated:** [March 25, 2020, 6:43pm UTC](https://discuss.elastic.co/t/looking-for-some-guidance-on-sourcefire-parsing/224887 "2020-03-25T18:43:21Z")
Hey Logstash Pros, We have found an interesting issue when ingesting our Sourcefire logs into Logstash. When we ingest the syslog message, Kibana adds a field "Error\_Processing\_Payload" which replicates the 'Message' f… --- ## [Help Getting a List of Logstash Clients](https://discuss.elastic.co/t/help-getting-a-list-of-logstash-clients/225018)
**Author:** [@bberesna](https://discuss.elastic.co/u/bberesna)\ **Replies:** 0\ **Last updated:** [March 25, 2020, 4:13pm UTC](https://discuss.elastic.co/t/help-getting-a-list-of-logstash-clients/225018 "2020-03-25T16:13:46Z")
I'm looking for a way to get a list of all my logstash clients without using the kibana gui Can anyone help me? --- ## [Split json](https://discuss.elastic.co/t/split-json/224952)
**Author:** [@JPelastic](https://discuss.elastic.co/u/JPelastic)\ **Replies:** 1\ **Last updated:** [March 25, 2020, 3:39pm UTC](https://discuss.elastic.co/t/split-json/224952 "2020-03-25T15:39:37Z")
Problem is basic: I just don't understand how split/mutate works. I got nested json data from an elastic source that I want to flatten into a new elastic target (for context). Data: .... "artikel" =\> { … --- ## [Logstash Index Data based on Certificate of Beat](https://discuss.elastic.co/t/logstash-index-data-based-on-certificate-of-beat/225000)
**Author:** [@cbxxen](https://discuss.elastic.co/u/cbxxen)\ **Replies:** 0\ **Last updated:** [March 25, 2020, 2:20pm UTC](https://discuss.elastic.co/t/logstash-index-data-based-on-certificate-of-beat/225000 "2020-03-25T14:20:31Z")
We plan to store Log Files of multiple Systems to our Elasticsearch. To reduce the Operational overhead I would like to store the Data in different Indicies according to the Certificate which the Beats use. Has someone d… --- ## [Lon Lat as Location - need help to parse the string](https://discuss.elastic.co/t/lon-lat-as-location-need-help-to-parse-the-string/224979)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 0\ **Last updated:** [March 25, 2020, 11:54am UTC](https://discuss.elastic.co/t/lon-lat-as-location-need-help-to-parse-the-string/224979 "2020-03-25T11:54:54Z")
Hi there, i got a location field (lon, lat) like this: "(6.95779078709084,51.1122094336657)" I dont know how to write the mutate filter in logstash for this string Beforehand I had lon lat as single fields and using … --- ## [Conditional in output filter fails on Linux](https://discuss.elastic.co/t/conditional-in-output-filter-fails-on-linux/224880)
**Author:** [@moderable](https://discuss.elastic.co/u/moderable)\ **Replies:** 8\ **Last updated:** [March 25, 2020, 11:08am UTC](https://discuss.elastic.co/t/conditional-in-output-filter-fails-on-linux/224880 "2020-03-25T11:08:26Z")
I have the output filter below. It helps me choose between my development Elasticsearch server and the corporate one. If I set \[@metadata\]\[local\_es\] to false it will send the events to the corporate server otherwise it s… --- ## [Logstash Kafka client logs many INVALID\_FETCH\_SESSION\_EPOCH messages](https://discuss.elastic.co/t/logstash-kafka-client-logs-many-invalid-fetch-session-epoch-messages/224822)
**Author:** [@Martin\_K](https://discuss.elastic.co/u/Martin_K)\ **Replies:** 1\ **Last updated:** [March 25, 2020, 9:33am UTC](https://discuss.elastic.co/t/logstash-kafka-client-logs-many-invalid-fetch-session-epoch-messages/224822 "2020-03-25T09:33:41Z")
In our production environment (elastic 7.3) we have a lot of logs from Logstash with the following messages: \[Consumer clientId=metrics\_prod-0, groupId=internal\_in\_metrics\_prod\] Node 3 was unable to process the fetch re… --- ## [Problem de parsing logs?](https://discuss.elastic.co/t/problem-de-parsing-logs/224771)
**Author:** [@Xmaster\_Buzz](https://discuss.elastic.co/u/Xmaster_Buzz)\ **Replies:** 1\ **Last updated:** [March 25, 2020, 9:26am UTC](https://discuss.elastic.co/t/problem-de-parsing-logs/224771 "2020-03-25T09:26:18Z")
Hello, I am pushing the data from kafka to elasticsearch using logstash, I am having the error below: How can I fix the problem, did any body know where the problem is coming from? --- ## [Logstash "Successfully started Logstash API endpoin" Error](https://discuss.elastic.co/t/logstash-successfully-started-logstash-api-endpoin-error/224376)
**Author:** [@Kumar\_Subham](https://discuss.elastic.co/u/Kumar_Subham)\ **Replies:** 2\ **Last updated:** [March 24, 2020, 9:35pm UTC](https://discuss.elastic.co/t/logstash-successfully-started-logstash-api-endpoin-error/224376 "2020-03-24T21:35:12Z")
I am a beginner in logstash i am not sure how to fix this error. "Successfully started Logstash API endpoint {:port=\>9600}" logstash.conf input { stdin{} } output { elasticsearch { hosts =\> \["l… --- ## [Cannot run Logstash on windows](https://discuss.elastic.co/t/cannot-run-logstash-on-windows/224425)
**Author:** [@cgrosu](https://discuss.elastic.co/u/cgrosu)\ **Replies:** 1\ **Last updated:** [March 24, 2020, 8:30pm UTC](https://discuss.elastic.co/t/cannot-run-logstash-on-windows/224425 "2020-03-24T20:30:33Z")
I followed the steps to install and run logstash from https://www.elastic.co/guide/en/logstash/current/installing-logstash.html https://www.elastic.co/guide/en/logstash/current/running-logstash-windows.html Unfortunate… --- ## [Convert fields to ECS](https://discuss.elastic.co/t/convert-fields-to-ecs/224888)
**Author:** [@david-vazquez](https://discuss.elastic.co/u/david-vazquez)\ **Replies:** 7\ **Last updated:** [March 24, 2020, 7:57pm UTC](https://discuss.elastic.co/t/convert-fields-to-ecs/224888 "2020-03-24T19:57:01Z")
Hello, I´m trying to ingest log file using ECS (Elastic Common Schema). For doing it, I have to rename source fields following the ECS convention. I made a filter which is not working: filter { json { source =\> "messag… --- ## [Detect if event.get("message").scan fails](https://discuss.elastic.co/t/detect-if-event-get-message-scan-fails/224818)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 7\ **Last updated:** [March 24, 2020, 6:53pm UTC](https://discuss.elastic.co/t/detect-if-event-get-message-scan-fails/224818 "2020-03-24T18:53:41Z")
Hello All, i would like to detect if "pattternmatch4" fails and do another process. How can i detect if it fails? ruby { code =\> 'event.set("patternmatch3", event.get("message").scan(/text.… --- ## [Parse](https://discuss.elastic.co/t/parse/224900)
**Author:** [@Claudio\_Ract\_Costa](https://discuss.elastic.co/u/Claudio_Ract_Costa)\ **Replies:** 1\ **Last updated:** [March 24, 2020, 6:30pm UTC](https://discuss.elastic.co/t/parse/224900 "2020-03-24T18:30:12Z")
SPCAS01-SpCas911-Gx-Test;1759780163;288613496;5e790ef9-37802|862|552(8783624|72455748963158|35487963214587|0|2|1|125874698\_191127017\_2020-03-23T23:53:58|191127017|0|0|4539489|466971741|51885749|0|26849|0|0|0|0|0|0|0|0|2… --- ## [Logstash output IF conditions not working with JMS input](https://discuss.elastic.co/t/logstash-output-if-conditions-not-working-with-jms-input/224906)
**Author:** [@dhruvp](https://discuss.elastic.co/u/dhruvp)\ **Replies:** 0\ **Last updated:** [March 24, 2020, 6:18pm UTC](https://discuss.elastic.co/t/logstash-output-if-conditions-not-working-with-jms-input/224906 "2020-03-24T18:18:13Z")
Hi I am new to ELK Stack and I have working solutions for logstash for one index output but now i have multiple microservices sending logs to same JMS JNDI Name using Solace queue but I want to output them in separate in… --- ## [Logstash shut down in spite of my pipelines.yml is configured](https://discuss.elastic.co/t/logstash-shut-down-in-spite-of-my-pipelines-yml-is-configured/224894)
**Author:** [@oula\_oula](https://discuss.elastic.co/u/oula_oula)\ **Replies:** 0\ **Last updated:** [March 24, 2020, 5:44pm UTC](https://discuss.elastic.co/t/logstash-shut-down-in-spite-of-my-pipelines-yml-is-configured/224894 "2020-03-24T17:44:03Z")
Hello, I have test my pipeline by running logstash with "-f path/to/my\_pipeline.conf" and everything worked very well. So now I would like to run logstash using my pipeline without the "-f", so I configured my pipelines… --- ## [Logstash aggregate filter never index same data from CSV file](https://discuss.elastic.co/t/logstash-aggregate-filter-never-index-same-data-from-csv-file/224883)
**Author:** [@leobr29](https://discuss.elastic.co/u/leobr29)\ **Replies:** 1\ **Last updated:** [March 24, 2020, 5:04pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-never-index-same-data-from-csv-file/224883 "2020-03-24T17:04:43Z")
Hi everyone, I come to you because I think that I I don't understand something with aggregate filter or more generally with logstash. I'm trying to read a CSV file and use aggregate filter, trying to reproduce somethi… --- ## [Grok filter for logstash/filebeats](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481)
**Author:** [@gsuboc](https://discuss.elastic.co/u/gsuboc)\ **Replies:** 42\ **Last updated:** [March 24, 2020, 4:59pm UTC](https://discuss.elastic.co/t/grok-filter-for-logstash-filebeats/216481 "2020-03-24T16:59:29Z")
I'm trying to extract my logs field using grok but it doesn't work. Here's my grok filter: filter { # parse the CSV structure generated from the log file into fields grok { match =\> {"message" =\> "%{GREEDYDATA:Job\_Name… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=383) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=385)