# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=638 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 639 --- ## [Retaining Historical Data Outside of ElasticStack](https://discuss.elastic.co/t/retaining-historical-data-outside-of-elasticstack/158663)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 9\ **Last updated:** [December 4, 2018, 8:33pm UTC](https://discuss.elastic.co/t/retaining-historical-data-outside-of-elasticstack/158663 "2018-12-04T20:33:31Z")
We have a small, single node Elastic Stack configuration running. Once we reach about 2 billion documents, our installation starts to have trouble keeping up. For this reason, I've configured Curator to delete indexes … --- ## [Can't send CSV data to elastic using logstash](https://discuss.elastic.co/t/cant-send-csv-data-to-elastic-using-logstash/158778)
**Author:** [@lightonseo](https://discuss.elastic.co/u/lightonseo)\ **Replies:** 2\ **Last updated:** [December 4, 2018, 8:18pm UTC](https://discuss.elastic.co/t/cant-send-csv-data-to-elastic-using-logstash/158778 "2018-12-04T20:18:41Z")
Hi there, I m new to ELK and I m facing this issue since two days. I couldn't send CSV datas to elastic with logstash. I m running ELK on windows 10, I m using wamp server (note sure if this is relevant) I've read a l… --- ## [Pipelines not working but work when launched separately](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330)
**Author:** [@wheelq](https://discuss.elastic.co/u/wheelq)\ **Replies:** 12\ **Last updated:** [December 4, 2018, 7:43pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330 "2018-12-04T19:43:24Z")
Has anyone managed to get pipelines working? Here are my observations: My pipelines.yml file: pipeline.id: httppipeline path.config: "http.conf" pipeline.id: ftppipeline path.config: "ftp.conf" After launching … --- ## [Unable to extract gzip'ed output from File Plugin](https://discuss.elastic.co/t/unable-to-extract-gziped-output-from-file-plugin/159410)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 6:53pm UTC](https://discuss.elastic.co/t/unable-to-extract-gziped-output-from-file-plugin/159410 "2018-12-04T18:53:49Z")
Using the file output with gzip set to true and a file is created but when I try to decompress the file using 7zip, I get an error saying it can't open file as archive. Am I misunderstanding the gzip flag? --- ## [Load table with jdbc\_static filter with NULL integers](https://discuss.elastic.co/t/load-table-with-jdbc-static-filter-with-null-integers/159391)
**Author:** [@Micah\_Hunsberger](https://discuss.elastic.co/u/Micah_Hunsberger)\ **Replies:** 2\ **Last updated:** [December 4, 2018, 4:29pm UTC](https://discuss.elastic.co/t/load-table-with-jdbc-static-filter-with-null-integers/159391 "2018-12-04T16:29:56Z")
I'm trying to use the jdbc\_static filter in my logstash pipeline, but I am getting the following error: \[2018-12-04T10:07:38,952\]\[ERROR\]\[logstash.filters.jdbc.readwritedatabase\] Exception when filling lookup db from loa… --- ## [Logstash File Input - Crashes when Missing SinceDB Path](https://discuss.elastic.co/t/logstash-file-input-crashes-when-missing-sincedb-path/159393)
**Author:** [@londonx](https://discuss.elastic.co/u/londonx)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 3:45pm UTC](https://discuss.elastic.co/t/logstash-file-input-crashes-when-missing-sincedb-path/159393 "2018-12-04T15:45:19Z")
Logstash v 6.5.1 Just noticed that the Logstash file input crashes and creates an error if it can't find the path for a sincedb file specified in the file input section. Would be good if there was a switch to tell file … --- ## [Securing Beats to Logstash with Public Cert (version 6.5.1)](https://discuss.elastic.co/t/securing-beats-to-logstash-with-public-cert-version-6-5-1/159364)
**Author:** [@Andrew\_Griffiths](https://discuss.elastic.co/u/Andrew_Griffiths)\ **Replies:** 1\ **Last updated:** [December 4, 2018, 3:42pm UTC](https://discuss.elastic.co/t/securing-beats-to-logstash-with-public-cert-version-6-5-1/159364 "2018-12-04T15:42:05Z")
Hi, I have a public cert from globalsign for securing beats -\> logstash. I just want a simple way to encrypt the data without having to create hundreds of client certs. logstash-beats.conf input{ beats{ host =\> "0.0… --- ## [Delete documents in elasticsearch using logstash](https://discuss.elastic.co/t/delete-documents-in-elasticsearch-using-logstash/159162)
**Author:** [@vsingh](https://discuss.elastic.co/u/vsingh)\ **Replies:** 2\ **Last updated:** [December 4, 2018, 5:30am UTC](https://discuss.elastic.co/t/delete-documents-in-elasticsearch-using-logstash/159162 "2018-12-04T05:30:21Z")
Hello there, I am using elasticsearch 6.4.0 version on a single node windows machine. i am trying to delete documents from elastic search using logstash. delete.csv: id 20 21 logstash configuration file: input { … --- ## [RabbitMQ input plugin does not adhere to pipeline configuration](https://discuss.elastic.co/t/rabbitmq-input-plugin-does-not-adhere-to-pipeline-configuration/159384)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 2:40pm UTC](https://discuss.elastic.co/t/rabbitmq-input-plugin-does-not-adhere-to-pipeline-configuration/159384 "2018-12-04T14:40:05Z")
Hello, I've been trying to use the rabbitMQ input plugin for logstash with the Cisco AMP API. More specifically, i want to connect to a stream and continuously listen for events and send that to our ELK stack. However,… --- ## [Logstash and a CSV](https://discuss.elastic.co/t/logstash-and-a-csv/159041)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 6\ **Last updated:** [December 4, 2018, 2:27pm UTC](https://discuss.elastic.co/t/logstash-and-a-csv/159041 "2018-12-04T14:27:45Z")
Good Sunday Morning, I have a question. I have a directory full of CSV files roughly 110 gigs or so. I am trying to upload them all to ES but I keep getting an error. I know ES is working because I am using a tool to up… --- ## [Count length of field / number of characters in a field and add the result into a new field](https://discuss.elastic.co/t/count-length-of-field-number-of-characters-in-a-field-and-add-the-result-into-a-new-field/157705)
**Author:** [@ssi](https://discuss.elastic.co/u/ssi)\ **Replies:** 8\ **Last updated:** [December 4, 2018, 2:01pm UTC](https://discuss.elastic.co/t/count-length-of-field-number-of-characters-in-a-field-and-add-the-result-into-a-new-field/157705 "2018-12-04T14:01:29Z")
hi im trying to port the answer given here \`\`\` filter { ruby { code =\> " event\['indexname'\] = event\['somefield'\].length \> 10 ? 'long-fields' : 'short-fields' " } } \`\`\` and here ruby { code =\> "… --- ## [Error in structuring log file](https://discuss.elastic.co/t/error-in-structuring-log-file/158120)
**Author:** [@Sripal](https://discuss.elastic.co/u/Sripal)\ **Replies:** 10\ **Last updated:** [December 4, 2018, 1:58pm UTC](https://discuss.elastic.co/t/error-in-structuring-log-file/158120 "2018-12-04T13:58:32Z")
Hi all, I'm new to ELK stack. I'm trying to read the system error log, but i am unable to achieve. Here are my exploration. SysErr Lof file sample: \[9/3/18 12:19:34:644 IST\] 00000ff6 SystemErr R at org.apache.cx… --- ## [Excessive RAM usage, gets OOM killed](https://discuss.elastic.co/t/excessive-ram-usage-gets-oom-killed/157539)
**Author:** [@cy\_lir](https://discuss.elastic.co/u/cy_lir)\ **Replies:** 6\ **Last updated:** [December 4, 2018, 1:40pm UTC](https://discuss.elastic.co/t/excessive-ram-usage-gets-oom-killed/157539 "2018-12-04T13:40:20Z")
Hi ! I'm running 4 basic pipelines on logstash with persisted queue. Everything ran smoothly, but the output went down for a certain amount of time, and the queue filled up, up to the 1GB limit. When the output went bac… --- ## [Upload Files from Logstash](https://discuss.elastic.co/t/upload-files-from-logstash/159359)
**Author:** [@patrick007](https://discuss.elastic.co/u/patrick007)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 1:13pm UTC](https://discuss.elastic.co/t/upload-files-from-logstash/159359 "2018-12-04T13:13:55Z")
I want to upload a file to S3 after the execution of logstash is complete. How can I do it ? Any Ideas --- ## [How to turn off and turn on email alerts from logstash without restarting application components or logstash](https://discuss.elastic.co/t/how-to-turn-off-and-turn-on-email-alerts-from-logstash-without-restarting-application-components-or-logstash/159352)
**Author:** [@naga\_kunchala](https://discuss.elastic.co/u/naga_kunchala)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 12:10pm UTC](https://discuss.elastic.co/t/how-to-turn-off-and-turn-on-email-alerts-from-logstash-without-restarting-application-components-or-logstash/159352 "2018-12-04T12:10:50Z")
does logstash permit any run-time dynamic variables? The reason I ask is that I can imagine a case where some condition happens that triggers alert mails at a much higher frequency than we intended. So, it would be good … --- ## [Field NUMBER save as String](https://discuss.elastic.co/t/field-number-save-as-string/159349)
**Author:** [@Viti](https://discuss.elastic.co/u/Viti)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 11:52am UTC](https://discuss.elastic.co/t/field-number-save-as-string/159349 "2018-12-04T11:52:06Z")
Hi, I need to create an index for elasticsearch from logstash, but the fields number it's saved as String in elasticsearch. This is my log: 2019-12-08#22:19:04#2019120822190431913#2019-12-08T22:19:04.898+01:00#servido… --- ## [Simple config file but fail to start](https://discuss.elastic.co/t/simple-config-file-but-fail-to-start/159121)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 6\ **Last updated:** [December 4, 2018, 9:42am UTC](https://discuss.elastic.co/t/simple-config-file-but-fail-to-start/159121 "2018-12-04T09:42:11Z")
Here is my config file, but logstash won't start. input { file { path =\> \["c:/tmp/elk/logs/Rtp\*"\] type =\> "cfxlog" } } filter { grok { match =\> {"message" =\> "%{URIHOST:date1} %{HAPROXYTIME:time1} %{GREEDYDATA:… --- ## [Automatically transfer data with Logstash](https://discuss.elastic.co/t/automatically-transfer-data-with-logstash/159321)
**Author:** [@chrillebile](https://discuss.elastic.co/u/chrillebile)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 9:29am UTC](https://discuss.elastic.co/t/automatically-transfer-data-with-logstash/159321 "2018-12-04T09:29:30Z")
Hey, I'm trying to run a cronjob that's transferring new data from one elastic server to another every hour. I'm using kibana/elastic 6.5 on the new server where in management i have the logstash pipeline option where … --- ## [Calculate Delta - Aggregate Filter](https://discuss.elastic.co/t/calculate-delta-aggregate-filter/159288)
**Author:** [@karnamonkster](https://discuss.elastic.co/u/karnamonkster)\ **Replies:** 1\ **Last updated:** [December 4, 2018, 9:01am UTC](https://discuss.elastic.co/t/calculate-delta-aggregate-filter/159288 "2018-12-04T09:01:50Z")
Hi, I need to calculate delta consumption from a stream of events generated by multiple devices uniquely identified by an ID. Based on some other forums i could see this is possible using Logstash Aggregate filter He… --- ## [SMS via LOGSTASH](https://discuss.elastic.co/t/sms-via-logstash/159299)
**Author:** [@Atulya\_Sharma](https://discuss.elastic.co/u/Atulya_Sharma)\ **Replies:** 1\ **Last updated:** [December 4, 2018, 8:45am UTC](https://discuss.elastic.co/t/sms-via-logstash/159299 "2018-12-04T08:45:06Z")
Hi All, Can anyone tell me how to send alert message on mobile number using logstash .conf file. Kindly help. Thanks in Advance. --- ## [Arcsight+logstash integration](https://discuss.elastic.co/t/arcsight-logstash-integration/159310)
**Author:** [@sandeepnandula](https://discuss.elastic.co/u/sandeepnandula)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 8:14am UTC](https://discuss.elastic.co/t/arcsight-logstash-integration/159310 "2018-12-04T08:14:47Z")
Hi All, please suggest me how do i resolve the following error when i try to run the below command to get the smart connector for arcsight the command: bin/logstash --modules arcsight --setup -M "arcsight.var.input.s… --- ## [Integrating arcsight with logstash](https://discuss.elastic.co/t/integrating-arcsight-with-logstash/159308)
**Author:** [@jagan](https://discuss.elastic.co/u/jagan)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 8:08am UTC](https://discuss.elastic.co/t/integrating-arcsight-with-logstash/159308 "2018-12-04T08:08:48Z")
Hi all, Could you suggest me how to resolve the following error when i run the command as below: usr/share/logstash\>bin/logstash --modules arcsight --setup -M "arcsight.var.input.smartconnector.port"=5000 -M "arcs… --- ## [Error in Logstash gork filter](https://discuss.elastic.co/t/error-in-logstash-gork-filter/159304)
**Author:** [@Sripal](https://discuss.elastic.co/u/Sripal)\ **Replies:** 2\ **Last updated:** [December 4, 2018, 7:39am UTC](https://discuss.elastic.co/t/error-in-logstash-gork-filter/159304 "2018-12-04T07:39:01Z")
Hi, I tried to structure the syserr log, and i'm getting some error. Please anyone guide me how to proceed. Log file sample \[1/8/18 14:08:22:395 IST\] 0000007a SystemErr R com.ibm.ws.persistence.WsJpaProductDerivat… --- ## [Multiline log problem](https://discuss.elastic.co/t/multiline-log-problem/159297)
**Author:** [@trelo](https://discuss.elastic.co/u/trelo)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 6:45am UTC](https://discuss.elastic.co/t/multiline-log-problem/159297 "2018-12-04T06:45:27Z")
Hi, I've logs like this: {20151003010443,D, {1,1},61,62,5,761520333,6,L,"SOME TEXT 1",1, {"P", {6, {"S","SOME TEXT 2"}, {"S","SOME TEXT 3"} } },"",5,13,13,894575181,1, {1} } I wrote and tested grok filter with Kibana … --- ## [Private ip geoip from dictionary](https://discuss.elastic.co/t/private-ip-geoip-from-dictionary/156763)
**Author:** [@ppafford](https://discuss.elastic.co/u/ppafford)\ **Replies:** 7\ **Last updated:** [December 4, 2018, 4:41am UTC](https://discuss.elastic.co/t/private-ip-geoip-from-dictionary/156763 "2018-12-04T04:41:10Z")
Related question Private ip geoip JSON object / structure and some questions but this is option 2 some other useful resources https://cinhtau.net/2018/01/26/enrich-geopoints/ https://www.elastic.co/assets/bltc44f784ba… --- ## [IP Encode](https://discuss.elastic.co/t/ip-encode/159285)
**Author:** [@Mathu\_Bhalan\_G](https://discuss.elastic.co/u/Mathu_Bhalan_G)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 4:40am UTC](https://discuss.elastic.co/t/ip-encode/159285 "2018-12-04T04:40:52Z")
Hi All, I'm new to ELK and have requirement to encode the client ip in apache server log using the digit & alpha mapping as below. Is it doable ? ex. 198.0.0.1 \> BJI.A.A.B num alpha 0 A 1 B 2 C 3 D 4 E… --- ## [Private ip geoip JSON object / structure and some questions](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555)
**Author:** [@ppafford](https://discuss.elastic.co/u/ppafford)\ **Replies:** 14\ **Last updated:** [December 4, 2018, 3:32am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555 "2018-12-04T03:32:30Z")
I've seen a bunch of topics on how to map your private ip address to a geoip json object but I have a question on the fields and the structure. I'm following this thread Creating geoip data for internal networks but migh… --- ## [GROK Filter for class / thread regex help please](https://discuss.elastic.co/t/grok-filter-for-class-thread-regex-help-please/159266)
**Author:** [@Frrrrrunkis](https://discuss.elastic.co/u/Frrrrrunkis)\ **Replies:** 0\ **Last updated:** [December 4, 2018, 1:17am UTC](https://discuss.elastic.co/t/grok-filter-for-class-thread-regex-help-please/159266 "2018-12-04T01:17:19Z")
trying to tackle a expression but im a novice. Need to match the log4j thread name ${NOSPACE:thread} worked until I encountered spaces then spaces and brackets need to match the following mulesoftlogs mulesoftlogs w… --- ## [Can Logstash Mutate Rename method use a file](https://discuss.elastic.co/t/can-logstash-mutate-rename-method-use-a-file/159063)
**Author:** [@MariNZa](https://discuss.elastic.co/u/MariNZa)\ **Replies:** 2\ **Last updated:** [December 3, 2018, 8:51pm UTC](https://discuss.elastic.co/t/can-logstash-mutate-rename-method-use-a-file/159063 "2018-12-03T20:51:39Z")
Hi there, I do not if it possible, but I couldn't find any information regarding the renaming of the fields based on the dictionary. I would like to use Logstash to convert all the entry data to the 'standard' format f… --- ## [Float type issue, should it be resolved at the ingestion process or before?](https://discuss.elastic.co/t/float-type-issue-should-it-be-resolved-at-the-ingestion-process-or-before/159174)
**Author:** [@lightonseo](https://discuss.elastic.co/u/lightonseo)\ **Replies:** 1\ **Last updated:** [December 3, 2018, 8:22pm UTC](https://discuss.elastic.co/t/float-type-issue-should-it-be-resolved-at-the-ingestion-process-or-before/159174 "2018-12-03T20:22:34Z")
Hello, I have some problems with floats. The data is coming from a csv file and the source looks like that "name","avg\_rank" "richard","1,65" "john","8,34" "dan","0,78" "alicia","14,659" Now, because elasticsearch was… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=637) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=639)