# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=790 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 791 --- ## [Logstash CPU usage issue after some time running](https://discuss.elastic.co/t/logstash-cpu-usage-issue-after-some-time-running/124823)
**Author:** [@lael](https://discuss.elastic.co/u/lael)\ **Replies:** 1\ **Last updated:** [March 30, 2018, 9:21am UTC](https://discuss.elastic.co/t/logstash-cpu-usage-issue-after-some-time-running/124823 "2018-03-30T09:21:36Z")
After some time of operation, our logstash instances (same configuration) are experiencing CPU and load average issues as shown below. When the issue occurs we have to restart the impacted logstash instance for it … --- ## [Logstash received data but dont push forward to next hoop, data come into Logstash queue](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056)
**Author:** [@tatdat](https://discuss.elastic.co/u/tatdat)\ **Replies:** 7\ **Last updated:** [March 30, 2018, 9:32am UTC](https://discuss.elastic.co/t/logstash-received-data-but-dont-push-forward-to-next-hoop-data-come-into-logstash-queue/126056 "2018-03-30T09:32:14Z")
Im using Elastic stack 6.2.2, Centos 7. LS-Collector have 8GB RAM, 8vPCU. Number event proccess : 4k e/s My data flow : Filebeat -\> LS-Collector -\> RabbitMQ -\> LS-Proccessor -\> ES In LS-Collector, i use file output … --- ## [Failed to convert the data type from string to float](https://discuss.elastic.co/t/failed-to-convert-the-data-type-from-string-to-float/126175)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 3\ **Last updated:** [March 30, 2018, 5:40am UTC](https://discuss.elastic.co/t/failed-to-convert-the-data-type-from-string-to-float/126175 "2018-03-30T05:40:15Z")
Hi experts, I have met an data type converting issue. My requirement is parse a xml file ------ done collect some node information which i need ------ done add a field to be used to retrieve it ------ done convert da… --- ## [How to transfer the array to csv format](https://discuss.elastic.co/t/how-to-transfer-the-array-to-csv-format/125685)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 6\ **Last updated:** [March 30, 2018, 2:18am UTC](https://discuss.elastic.co/t/how-to-transfer-the-array-to-csv-format/125685 "2018-03-30T02:18:21Z")
Hi experts, I have parsed the XML file and get some useful NODEs' information, they are saved as a array format. I want to draw a trend chart by using Kibana, but I can't draw it because all of data are saved as array f… --- ## [Logstash Multiline Parse Error](https://discuss.elastic.co/t/logstash-multiline-parse-error/126051)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 2\ **Last updated:** [March 30, 2018, 1:26am UTC](https://discuss.elastic.co/t/logstash-multiline-parse-error/126051 "2018-03-30T01:26:17Z")
Hello, My log file has multiline patterns(it's a CSV file having around 15 fields) example log: IM25756756,ADDRESS SERVICE (HPI),01/05/2018 15:41:12,67,mlm,202227,Closed,c1t09878.itcs.hpicorp.net,W-INCFLS-HPIT-LINUX,W… --- ## [The logstash plugin how can install plugin with .gem file? without internet](https://discuss.elastic.co/t/the-logstash-plugin-how-can-install-plugin-with-gem-file-without-internet/126168)
**Author:** [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Replies:** 1\ **Last updated:** [March 30, 2018, 12:36am UTC](https://discuss.elastic.co/t/the-logstash-plugin-how-can-install-plugin-with-gem-file-without-internet/126168 "2018-03-30T00:36:57Z")
the logstash plugin how can install plugin with .gem file? with offline. the .gem file is local system file. thanks. --- ## [Error in processing events from DLQ](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149)
**Author:** [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Replies:** 2\ **Last updated:** [March 29, 2018, 9:18pm UTC](https://discuss.elastic.co/t/error-in-processing-events-from-dlq/126149 "2018-03-29T21:18:20Z")
All, I am trying to expose the raw events in a index dedicated to DLQ. These are the fields I would like to expose: index\_name (where it was unable to perform originally) error\_reason(exception) raw\_event(\_source) … --- ## [Logstash filter cipher - static\_iv instead of random iv does NOT WORK](https://discuss.elastic.co/t/logstash-filter-cipher-static-iv-instead-of-random-iv-does-not-work/126116)
**Author:** [@Orsius](https://discuss.elastic.co/u/Orsius)\ **Replies:** 3\ **Last updated:** [March 29, 2018, 8:50pm UTC](https://discuss.elastic.co/t/logstash-filter-cipher-static-iv-instead-of-random-iv-does-not-work/126116 "2018-03-29T20:50:06Z")
Dear community member, I use logstash-filter-cipher to encrypt my "message" field; the confituration works pretty well the "message" is well crypt BUT the encryption is base on "iv\_random" therefore the same "string" is… --- ## [Groking duration](https://discuss.elastic.co/t/groking-duration/126146)
**Author:** [@Milan\_Kaliraj](https://discuss.elastic.co/u/Milan_Kaliraj)\ **Replies:** 3\ **Last updated:** [March 29, 2018, 8:38pm UTC](https://discuss.elastic.co/t/groking-duration/126146 "2018-03-29T20:38:41Z")
HI , How can we grok duration field for following piece of message: "Duration: 0h:00m:16s, Bytes xmt: 76895, Bytes rcv: 81897, Reason: User Requested" --- ## [Convert @timestamp date format](https://discuss.elastic.co/t/convert-timestamp-date-format/125966)
**Author:** [@Bhairavi](https://discuss.elastic.co/u/Bhairavi)\ **Replies:** 8\ **Last updated:** [March 29, 2018, 7:40pm UTC](https://discuss.elastic.co/t/convert-timestamp-date-format/125966 "2018-03-29T19:40:36Z")
Hi! This is the sample logs which I am trying to store in elasticsearch index. Mar 26 08:48:21 ip-192-168-0-94 sshd\[18576\]: Received disconnect from 115.238.245.2: 11: \[preauth\] I want to convert this date pattern to … --- ## [Grok parser not working as expected with GREEDYDATA](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849)
**Author:** [@sachinvy](https://discuss.elastic.co/u/sachinvy)\ **Replies:** 12\ **Last updated:** [March 29, 2018, 7:31pm UTC](https://discuss.elastic.co/t/grok-parser-not-working-as-expected-with-greedydata/125849 "2018-03-29T19:31:20Z")
Here is an example: Data in file: 03-27-18 09:32:10,563 \[1452\] DEBUG AdvancedProperties - No value in: HKEY\_LOCAL\_MACHINE\\SOFTWARE\\GlobalSCAPE Inc.\\EFT Server 7.0\\ReplaceBackslashWithSlashInPathsForFTP, using default: … --- ## [Splitting IP addresses](https://discuss.elastic.co/t/splitting-ip-addresses/126117)
**Author:** [@ecamero2](https://discuss.elastic.co/u/ecamero2)\ **Replies:** 2\ **Last updated:** [March 29, 2018, 6:14pm UTC](https://discuss.elastic.co/t/splitting-ip-addresses/126117 "2018-03-29T18:14:07Z")
Good Morning, I have a need to remove part of a field if it exists. I have a field that contains either one or two ip addresses: ie: "1.2.3.4" OR "1.2.3.4,5.6.7.8" I only want to retain the first address (before the… --- ## [Logstash Plugin for Cloudwatch logs](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 5\ **Last updated:** [March 29, 2018, 5:33pm UTC](https://discuss.elastic.co/t/logstash-plugin-for-cloudwatch-logs/125677 "2018-03-29T17:33:03Z")
Hi, I am trying to integrate AWS Lambda logs onto ELK Stack. My AWS Lambda is writing logs into Cloudwatch logs. I am trying to get this data into logstash using logstash-input-cloudwatch plugin. But, having difficulti… --- ## [Converting to geo\_point](https://discuss.elastic.co/t/converting-to-geo-point/125417)
**Author:** [@Pororo](https://discuss.elastic.co/u/Pororo)\ **Replies:** 3\ **Last updated:** [March 29, 2018, 5:13pm UTC](https://discuss.elastic.co/t/converting-to-geo-point/125417 "2018-03-29T17:13:55Z")
Hi, For now, I have my own fields called "lat" and "lon" in the json file. I was trying to make ES recognize it as the geo\_point data so I did some kinds of editing in Logstash. filter { mutate{ add\_field =\> {"loca… --- ## [Geo point mapping](https://discuss.elastic.co/t/geo-point-mapping/125105)
**Author:** [@Pororo](https://discuss.elastic.co/u/Pororo)\ **Replies:** 7\ **Last updated:** [March 29, 2018, 5:10pm UTC](https://discuss.elastic.co/t/geo-point-mapping/125105 "2018-03-29T17:10:40Z")
Hi, my dataset basically consists of the geo-information, but the format is not as perfect as I can directly ingest them in and let ES recognize it as the geo point type. {"geo\_information": "united states/michigan/hol… --- ## [Delete document in ElasticSearch Output Plugin using Script](https://discuss.elastic.co/t/delete-document-in-elasticsearch-output-plugin-using-script/126128)
**Author:** [@msugas19](https://discuss.elastic.co/u/msugas19)\ **Replies:** 1\ **Last updated:** [March 29, 2018, 5:07pm UTC](https://discuss.elastic.co/t/delete-document-in-elasticsearch-output-plugin-using-script/126128 "2018-03-29T17:07:02Z")
I am trying to delete documents using a scripted query in Elastic Search through Logstash, however whenever I do I get the following error: \[2018-03-28T22:06:17,681\]\[ERROR\]\[logstash.outputs.elasticsearch\] Encountered a … --- ## [Multiple Pipeline - Single or multiple events?](https://discuss.elastic.co/t/multiple-pipeline-single-or-multiple-events/125078)
**Author:** [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Replies:** 6\ **Last updated:** [March 29, 2018, 2:54pm UTC](https://discuss.elastic.co/t/multiple-pipeline-single-or-multiple-events/125078 "2018-03-29T14:54:25Z")
I could configure multiple pipelines using logstash setup. But have a generic query on practice. Source from redis cache, but cache key is different for each type of event. Pipelines filters are different for each type… --- ## [Pipelines or "if else" conditions for output](https://discuss.elastic.co/t/pipelines-or-if-else-conditions-for-output/124372)
**Author:** [@phil.zac](https://discuss.elastic.co/u/phil.zac)\ **Replies:** 12\ **Last updated:** [March 29, 2018, 2:41pm UTC](https://discuss.elastic.co/t/pipelines-or-if-else-conditions-for-output/124372 "2018-03-29T14:41:14Z")
Hi, I am using elasticsearch-5.6.1 kibana-5.6.1 logstash-5.6.1 metricbeat-5.6.1 packetbeat-5.6.1 x-pack-5.6.1 for Windows Server 2012 R2 My issue - I cant figure out how to make multiple output for logstash.co… --- ## [Title case conversion in logstash filter-mutate](https://discuss.elastic.co/t/title-case-conversion-in-logstash-filter-mutate/119443)
**Author:** [@sujeetKumar](https://discuss.elastic.co/u/sujeetKumar)\ **Replies:** 6\ **Last updated:** [March 29, 2018, 12:31pm UTC](https://discuss.elastic.co/t/title-case-conversion-in-logstash-filter-mutate/119443 "2018-03-29T12:31:52Z")
Env- Elastic 5.5 Requirement: For Kibana Region map, Country Name is required in Title case. So, I can do this in logstash. I have tried below statement: filter { mutate { gsub =\> \[ # replace all first character o… --- ## [Http\_poller is not working for me, please help](https://discuss.elastic.co/t/http-poller-is-not-working-for-me-please-help/124050)
**Author:** [@David\_Gidony](https://discuss.elastic.co/u/David_Gidony)\ **Replies:** 19\ **Last updated:** [March 29, 2018, 12:16pm UTC](https://discuss.elastic.co/t/http-poller-is-not-working-for-me-please-help/124050 "2018-03-29T12:16:18Z")
Hi, i'm trying to fetch data from solarwinds via the rest-api, when i use the url in my browser, everything works fine. when i run this config file, i get this weird error, im not even sure how to read it. this is my … --- ## [Logstash Reindexing task behaves irratic](https://discuss.elastic.co/t/logstash-reindexing-task-behaves-irratic/126075)
**Author:** [@rolfje](https://discuss.elastic.co/u/rolfje)\ **Replies:** 0\ **Last updated:** [March 29, 2018, 11:07am UTC](https://discuss.elastic.co/t/logstash-reindexing-task-behaves-irratic/126075 "2018-03-29T11:07:54Z")
We are using a GelfAppender and Logstash to send log entries to ElasticSearch. Log entries are sent to indices based on current date (one index per day of logging so we can easly clean up). We also index MDC (Mapped Diag… --- ## [Translate results in incorrect (surprise) value](https://discuss.elastic.co/t/translate-results-in-incorrect-surprise-value/126061)
**Author:** [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Replies:** 2\ **Last updated:** [March 29, 2018, 9:58am UTC](https://discuss.elastic.co/t/translate-results-in-incorrect-surprise-value/126061 "2018-03-29T09:58:53Z")
We have syslog input events with severity 6 which we want to map to priority 1: translate { field =\> "severity" destination =\> "priority" dictionary =\> { "0" =\> 5 "1" =\> 4 "2" =\> 3 "3" =\> 3 "4" =\> 2… --- ## [Translog exploding](https://discuss.elastic.co/t/translog-exploding/118877)
**Author:** [@Chaka](https://discuss.elastic.co/u/Chaka)\ **Replies:** 1\ **Last updated:** [March 29, 2018, 9:48am UTC](https://discuss.elastic.co/t/translog-exploding/118877 "2018-03-29T09:48:09Z")
Translog files (.tlog) files are becoming very big and server runs out of disk space quite quickly. Stored data is exclusively loaded via Logstash, which pulls it from Mysql. Considering my problem is disk space, my re… --- ## [Unable to index the array(from mongodb) using the logstash](https://discuss.elastic.co/t/unable-to-index-the-array-from-mongodb-using-the-logstash/125740)
**Author:** [@surendar27](https://discuss.elastic.co/u/surendar27)\ **Replies:** 1\ **Last updated:** [March 29, 2018, 9:32am UTC](https://discuss.elastic.co/t/unable-to-index-the-array-from-mongodb-using-the-logstash/125740 "2018-03-29T09:32:26Z")
Hi, I am unable to index the array in elasticsearch using logstash. I have tried the split option but no luck. Below is the format of data i have(i.e is coming from mongodb), {"\_id"=\>BSON::ObjectId('5ab4f8fc36e9ca09d4cf… --- ## [Mask parts of the message](https://discuss.elastic.co/t/mask-parts-of-the-message/125693)
**Author:** [@Kissa\_Koira](https://discuss.elastic.co/u/Kissa_Koira)\ **Replies:** 3\ **Last updated:** [March 29, 2018, 7:40am UTC](https://discuss.elastic.co/t/mask-parts-of-the-message/125693 "2018-03-29T07:40:32Z")
I am sending application exceptions from winlogbeat to logstash. Some parts of the message might contain information I want to mask. For example User and Host. How can I do that? Matched string is: "message" =\> "27.03.2… --- ## [Logstash-input-jms](https://discuss.elastic.co/t/logstash-input-jms/126023)
**Author:** [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)\ **Replies:** 1\ **Last updated:** [March 29, 2018, 6:05am UTC](https://discuss.elastic.co/t/logstash-input-jms/126023 "2018-03-29T06:05:02Z")
hello guys, i have problem with logstash-input-jms plugin: this plugin don't have opportunity to reconnect to queue manager in network partitions cases. Example of JMS Exception that occur: \[2018-03-29T07:28:59,456\]\[… --- ## [Risk calculation](https://discuss.elastic.co/t/risk-calculation/125595)
**Author:** [@Bakkali\_Amine](https://discuss.elastic.co/u/Bakkali_Amine)\ **Replies:** 1\ **Last updated:** [March 28, 2018, 8:23pm UTC](https://discuss.elastic.co/t/risk-calculation/125595 "2018-03-28T20:23:40Z")
Is it possible to add a tool that calculate risk for logs that logstash manipulate ? Like AlienVault who use an operation. Thank you --- ## [Geoip plugin and private address](https://discuss.elastic.co/t/geoip-plugin-and-private-address/125863)
**Author:** [@rudok](https://discuss.elastic.co/u/rudok)\ **Replies:** 4\ **Last updated:** [March 28, 2018, 7:21pm UTC](https://discuss.elastic.co/t/geoip-plugin-and-private-address/125863 "2018-03-28T19:21:27Z")
Hi, I have IP ( private range ) , lot and lat coordinates (country, city ... information). Is there any way to create geoip point field in message that Elasticsearch and Kibana both understand from these variables? I… --- ## [How do I match URL](https://discuss.elastic.co/t/how-do-i-match-url/125984)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [March 28, 2018, 6:35pm UTC](https://discuss.elastic.co/t/how-do-i-match-url/125984 "2018-03-28T18:35:10Z")
Hi Team, How do I match URL from below pattern? As well as how do I exclude " and ,. Here I am starting.. "1078","http://www.innovationsystems.gr/RDFdk/","offline","malware\_download","emotet,heodo,exe" "%{INT:ID}","%… --- ## [Metricbeat Communication problem](https://discuss.elastic.co/t/metricbeat-communication-problem/125916)
**Author:** [@daithi\_dearg](https://discuss.elastic.co/u/daithi_dearg)\ **Replies:** 4\ **Last updated:** [March 28, 2018, 4:21pm UTC](https://discuss.elastic.co/t/metricbeat-communication-problem/125916 "2018-03-28T16:21:25Z")
Hello, Tearing my hair out at this early juncture. I have setup Logstash/Elastisearch/Kibana on a Redhat Server and I have setup MetricBeats on my Windows machine. Getting the following in the metricbeat logs: 2018-0… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=789) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=791)