# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=847 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 848 --- ## [Measuring the taking time](https://discuss.elastic.co/t/measuring-the-taking-time/113900)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 4\ **Last updated:** [January 4, 2018, 9:15am UTC](https://discuss.elastic.co/t/measuring-the-taking-time/113900 "2018-01-04T09:15:25Z")
hi everyone! is there any way to do measuring the time taken by logstash pipeline processing (input cloudwatch logs \> filter some patterns \> output to aws elasticsearch)???? i am using logstash v6.1.1 and not include x… --- ## [Loading a CSV file](https://discuss.elastic.co/t/loading-a-csv-file/113760)
**Author:** [@Rahul\_Pm](https://discuss.elastic.co/u/Rahul_Pm)\ **Replies:** 5\ **Last updated:** [January 4, 2018, 9:04am UTC](https://discuss.elastic.co/t/loading-a-csv-file/113760 "2018-01-04T09:04:34Z")
Hi, I'm trying to load a csv file (stored on my windows machine) into elasticsearch via logstash. I need to visualize it on Kibana. I have everything installed on my windows machine. P.S. It's my first time using Logst… --- ## [How to pull logs with specific date range from aws cloudwatch logs](https://discuss.elastic.co/t/how-to-pull-logs-with-specific-date-range-from-aws-cloudwatch-logs/113893)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 1\ **Last updated:** [January 4, 2018, 7:56am UTC](https://discuss.elastic.co/t/how-to-pull-logs-with-specific-date-range-from-aws-cloudwatch-logs/113893 "2018-01-04T07:56:28Z")
hi there! 1 ) i would like to pull the logs with specific date range or specific log from aws cloudwatch logs. for eg: i have 1-20 days logs. but i want to pull 3-9 days logs via logstash server. is that possible? 2… --- ## [Get 255 characters in logstash and store it as a new field](https://discuss.elastic.co/t/get-255-characters-in-logstash-and-store-it-as-a-new-field/113897)
**Author:** [@ranjith](https://discuss.elastic.co/u/ranjith)\ **Replies:** 3\ **Last updated:** [January 4, 2018, 7:42am UTC](https://discuss.elastic.co/t/get-255-characters-in-logstash-and-store-it-as-a-new-field/113897 "2018-01-04T07:42:34Z")
Hi all, if i want to extract first 255 characters from a field, and also the 255th character is space it should not be splitted , then it should split as 254th character. Currently i am using the following code for ext… --- ## [How to replace special characters to another character](https://discuss.elastic.co/t/how-to-replace-special-characters-to-another-character/111232)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 4\ **Last updated:** [January 4, 2018, 7:35am UTC](https://discuss.elastic.co/t/how-to-replace-special-characters-to-another-character/111232 "2018-01-04T07:35:21Z")
Hello! I want to replace below string hello\>\>\>\>hiii\>huha special characters to , and I achieve by the following way.. filter{ mutate { gsub =\> \[ "title", "\[\>/\]", "," \] } } It works but it replace every single … --- ## [Data not being recognized of specific type](https://discuss.elastic.co/t/data-not-being-recognized-of-specific-type/113965)
**Author:** [@sohenrane](https://discuss.elastic.co/u/sohenrane)\ **Replies:** 2\ **Last updated:** [January 3, 2018, 9:34pm UTC](https://discuss.elastic.co/t/data-not-being-recognized-of-specific-type/113965 "2018-01-03T21:34:27Z")
Hi, I ingested data using LogStash with its configuration and a template, but it seems like the data is not mapping to the data type specified in the template. LogStash's configuration: \[Testing\_configuration\] LogSta… --- ## [Logstash 6.1.1 docker error](https://discuss.elastic.co/t/logstash-6-1-1-docker-error/113849)
**Author:** [@fij29dkvn28](https://discuss.elastic.co/u/fij29dkvn28)\ **Replies:** 4\ **Last updated:** [January 3, 2018, 9:11pm UTC](https://discuss.elastic.co/t/logstash-6-1-1-docker-error/113849 "2018-01-03T21:11:11Z")
Just trying to create a small ELK instance with the new 6.1.1 docker containers. I'm trying to keep it as vanilla as possible (not creating a new docker build file for example) and secure (a password that can be set in o… --- ## [Logstash http output plugin throws "HTTP Output Failure"](https://discuss.elastic.co/t/logstash-http-output-plugin-throws-http-output-failure/112304)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 4\ **Last updated:** [January 3, 2018, 8:05pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-throws-http-output-failure/112304 "2018-01-03T20:05:28Z")
Hi, I am using Logstash Output Plugin to send data to another Logstash instance. Some events are successfully processed but for some events it is throwing the following error, {:timestamp=\>"2017-12-18T15:13:40.243000-0… --- ## [Splitting an event into multiple documents](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851)
**Author:** [@MattHarvey](https://discuss.elastic.co/u/MattHarvey)\ **Replies:** 4\ **Last updated:** [January 3, 2018, 7:29pm UTC](https://discuss.elastic.co/t/splitting-an-event-into-multiple-documents/113851 "2018-01-03T19:29:17Z")
I am having some trouble splitting fields and cloning events in logstash--this is the first time I am doing so. Here's my logstash config (I've removed the filter split plugin configuration as it is incorrect): input {… --- ## [Best practice for Kubernetes liveness probe?](https://discuss.elastic.co/t/best-practice-for-kubernetes-liveness-probe/113967)
**Author:** [@JamesB](https://discuss.elastic.co/u/JamesB)\ **Replies:** 0\ **Last updated:** [January 3, 2018, 5:38pm UTC](https://discuss.elastic.co/t/best-practice-for-kubernetes-liveness-probe/113967 "2018-01-03T17:38:25Z")
I'm going to run Logstash in Kubernetes and was wondering what the best practice was for knowing that logstash is alive and functioning? I can then create a liveness and readiness probe in Kubernetes, whether it's HTTP o… --- ## [Dead Letter Queue questions](https://discuss.elastic.co/t/dead-letter-queue-questions/113701)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 2\ **Last updated:** [January 3, 2018, 4:45pm UTC](https://discuss.elastic.co/t/dead-letter-queue-questions/113701 "2018-01-03T16:45:15Z")
Still trying to wrap my head around a few things regarding the Dead Letter Queue feature: I sometimes see my DLQ log files contain just a 1. What does this mean? I am running one config for my main pipeline and ano… --- ## [Logstash alert type =\>](https://discuss.elastic.co/t/logstash-alert-type/112898)
**Author:** [@kunam](https://discuss.elastic.co/u/kunam)\ **Replies:** 4\ **Last updated:** [December 26, 2017, 3:14am UTC](https://discuss.elastic.co/t/logstash-alert-type/112898 "2017-12-26T03:14:08Z")
Hi, I'm new to logstash, please bare me and help me in writing this. How to use logstash alert type in logstash config file. I had a log source called allyourbase and alert type should be mapped to "AllYourBase - Audi… --- ## [Environment variable in Logstash](https://discuss.elastic.co/t/environment-variable-in-logstash/112978)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 3\ **Last updated:** [January 3, 2018, 12:29pm UTC](https://discuss.elastic.co/t/environment-variable-in-logstash/112978 "2018-01-03T12:29:12Z")
I have the following use case: I am capturing some data from ActiveMQ bus and Storing it in ElasticSearch via Logstash. I am capturing some data ("version number") via Kafka and storing it in ElasticSearch. This variab… --- ## [XPack Not Installing on Logstash](https://discuss.elastic.co/t/xpack-not-installing-on-logstash/113757)
**Author:** [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\ **Replies:** 4\ **Last updated:** [January 3, 2018, 12:28pm UTC](https://discuss.elastic.co/t/xpack-not-installing-on-logstash/113757 "2018-01-03T12:28:37Z")
I stop Logstash, run the "bin/logstash-plugin install x-pack" command, I get the 'Install successful' reply and then start Logstash again. I now check the Logstash.yml file and can't see the following settings: xpack.m… --- ## [ERROR Unable to locate appender "${sys:ls.log.format}\_rolling" for logger config "root"](https://discuss.elastic.co/t/error-unable-to-locate-appender-sys-ls-log-format-rolling-for-logger-config-root/107942)
**Author:** [@Juliette](https://discuss.elastic.co/u/Juliette)\ **Replies:** 7\ **Last updated:** [January 3, 2018, 11:18am UTC](https://discuss.elastic.co/t/error-unable-to-locate-appender-sys-ls-log-format-rolling-for-logger-config-root/107942 "2018-01-03T11:18:41Z")
Hi, I need your help ! I'm working on Elasticsearch ELK since 2 weeks and I have some issues when a try to create my first pipeline. I saw many blog, article and documentation before coming here and disturb you. Versio… --- ## [Logstash consuming lot of CPU](https://discuss.elastic.co/t/logstash-consuming-lot-of-cpu/113131)
**Author:** [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Replies:** 7\ **Last updated:** [January 3, 2018, 9:50am UTC](https://discuss.elastic.co/t/logstash-consuming-lot-of-cpu/113131 "2018-01-03T09:50:38Z")
I am using Logstash 2.4 to process multiple files, therefore to process all files at time; launching multiple logstash instances using "logstash -f file.conf \< file1" on a Windows Server. I don't need to tail and these f… --- ## [How to monitor and constantly forward newly added documents from an elasticsearch index to a Kafka topic with logstash?](https://discuss.elastic.co/t/how-to-monitor-and-constantly-forward-newly-added-documents-from-an-elasticsearch-index-to-a-kafka-topic-with-logstash/113631)
**Author:** [@anon24174512](https://discuss.elastic.co/u/anon24174512)\ **Replies:** 4\ **Last updated:** [January 3, 2018, 9:19am UTC](https://discuss.elastic.co/t/how-to-monitor-and-constantly-forward-newly-added-documents-from-an-elasticsearch-index-to-a-kafka-topic-with-logstash/113631 "2018-01-03T09:19:33Z")
Dear all, Question in the title :wink: To be more precise, I have a message written to a Kafka topic, then to an Elasticsearch index using Logstash then written back from the index to another Kafka topic using again Log… --- ## [Error in updating document by id in elasticsearch output plugin](https://discuss.elastic.co/t/error-in-updating-document-by-id-in-elasticsearch-output-plugin/113721)
**Author:** [@yaser](https://discuss.elastic.co/u/yaser)\ **Replies:** 2\ **Last updated:** [January 3, 2018, 9:10am UTC](https://discuss.elastic.co/t/error-in-updating-document-by-id-in-elasticsearch-output-plugin/113721 "2018-01-03T09:10:57Z")
:slight\_smile: --- ## [How to calculate random no](https://discuss.elastic.co/t/how-to-calculate-random-no/113884)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 1\ **Last updated:** [January 3, 2018, 9:06am UTC](https://discuss.elastic.co/t/how-to-calculate-random-no/113884 "2018-01-03T09:06:46Z")
hello.. Anybody know how to generate random numbers between 0 to .5 using logstash. I use the following code but not working.. ruby { code =\> 'event.set("randomNumber", rand(100).to\_i())' } Thanks --- ## [Netflow Persistent Queue](https://discuss.elastic.co/t/netflow-persistent-queue/113869)
**Author:** [@hmmxp](https://discuss.elastic.co/u/hmmxp)\ **Replies:** 0\ **Last updated:** [January 3, 2018, 4:13am UTC](https://discuss.elastic.co/t/netflow-persistent-queue/113869 "2018-01-03T04:13:19Z")
Dear All, If my input type is Netflow and have persistent queue enabled, does the data will store into the persistent queue in the event of failure like lost connection with ES node? Also may i know what is the correla… --- ## [XML Filter fail](https://discuss.elastic.co/t/xml-filter-fail/113800)
**Author:** [@Callahan](https://discuss.elastic.co/u/Callahan)\ **Replies:** 1\ **Last updated:** [January 2, 2018, 9:23pm UTC](https://discuss.elastic.co/t/xml-filter-fail/113800 "2018-01-02T21:23:12Z")
Hi, I'm looking to extract an xml file into usable components. On all my configs to date, the inputs have usually been syslogs or files that allow key/value separation making things easier to separate. I have a file wi… --- ## [Logstash does not start; does not accept Java version](https://discuss.elastic.co/t/logstash-does-not-start-does-not-accept-java-version/113832)
**Author:** [@enrique3000](https://discuss.elastic.co/u/enrique3000)\ **Replies:** 0\ **Last updated:** [January 2, 2018, 8:26pm UTC](https://discuss.elastic.co/t/logstash-does-not-start-does-not-accept-java-version/113832 "2018-01-02T20:26:57Z")
Best regard. I would like to know why the Logstash (version 6.1.1) that I have installed locally, does not accept the Java version that is on my computer. After locating in DOS in the directory where I have it installed … --- ## [Ruby Regex to find and replace key field where value field is missing in KV pair](https://discuss.elastic.co/t/ruby-regex-to-find-and-replace-key-field-where-value-field-is-missing-in-kv-pair/113325)
**Author:** [@elk\_chaser](https://discuss.elastic.co/u/elk_chaser)\ **Replies:** 1\ **Last updated:** [January 2, 2018, 6:08pm UTC](https://discuss.elastic.co/t/ruby-regex-to-find-and-replace-key-field-where-value-field-is-missing-in-kv-pair/113325 "2018-01-02T18:08:55Z")
I have a JSON to start with where there are key fields that may or may not have value field; I have to remove those keys. Example JSON: str= '{"key1": "val1\_start\_:,?val1end", "key2": "key3": "val3\_start:,?val3end", "ke… --- ## [Grok parsing issue on Bitbucket/Sonar API Logs](https://discuss.elastic.co/t/grok-parsing-issue-on-bitbucket-sonar-api-logs/113737)
**Author:** [@karanarora](https://discuss.elastic.co/u/karanarora)\ **Replies:** 5\ **Last updated:** [January 2, 2018, 6:04pm UTC](https://discuss.elastic.co/t/grok-parsing-issue-on-bitbucket-sonar-api-logs/113737 "2018-01-02T18:04:58Z")
Hi, In my logstash pipeline, I'm trying to use grok to segregate/filter out meaningful bits of information from the API call output logs to Tools like Bitbucket/SonarQube etc. to fetch metrics data into Elasticsearch T… --- ## [Problems with parsing JSON file with \\n \\t and \\s](https://discuss.elastic.co/t/problems-with-parsing-json-file-with-n-t-and-s/113802)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 1\ **Last updated:** [January 2, 2018, 4:43pm UTC](https://discuss.elastic.co/t/problems-with-parsing-json-file-with-n-t-and-s/113802 "2018-01-02T16:43:59Z")
Hi everyone! I have the following file I need to process in logstash: { "DataChangeInfo" : "Archive Log Set archiveLogSet.25933761.25933688 Info: Thread# Sequence# FirstScn LastScn", "documentsList" : \[ { "commandScn… --- ## [Getting S3 file metadata into Kibana through logstash S3 input plugin](https://discuss.elastic.co/t/getting-s3-file-metadata-into-kibana-through-logstash-s3-input-plugin/113482)
**Author:** [@dsanpedro](https://discuss.elastic.co/u/dsanpedro)\ **Replies:** 1\ **Last updated:** [January 2, 2018, 4:19pm UTC](https://discuss.elastic.co/t/getting-s3-file-metadata-into-kibana-through-logstash-s3-input-plugin/113482 "2018-01-02T16:19:31Z")
Hello. Is there a way to get S3 file metadata into Kibana using the logstash S3 plugin or any other method? For example, we have files that when you run 'aws s3api head-object --bucket bucketname --key filename' return … --- ## [Does the JSON filter create \`null\` value if parsing empty string?](https://discuss.elastic.co/t/does-the-json-filter-create-null-value-if-parsing-empty-string/113801)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 2\ **Last updated:** [January 2, 2018, 4:19pm UTC](https://discuss.elastic.co/t/does-the-json-filter-create-null-value-if-parsing-empty-string/113801 "2018-01-02T16:19:27Z")
I have been trying to diagnose a couple issues with my Logstash pipeline, but this would explain it if true. If the JSON filter tries to parse a field that is an empty string, it seems like it could be changing the field… --- ## [Missing Converter handling for full class Error](https://discuss.elastic.co/t/missing-converter-handling-for-full-class-error/113630)
**Author:** [@nickl1](https://discuss.elastic.co/u/nickl1)\ **Replies:** 2\ **Last updated:** [January 2, 2018, 3:20pm UTC](https://discuss.elastic.co/t/missing-converter-handling-for-full-class-error/113630 "2018-01-02T15:20:42Z")
Hi all, I'm trying to use the Logstash JDBC plugin to sync my Postgres database. For some reason, midway through the data transfer, I get this error: Exception when executing JDBC query {:exception=\>#\ **Author:** [@Phani\_Nadiminti](https://discuss.elastic.co/u/Phani_Nadiminti)\ **Replies:** 1\ **Last updated:** [January 2, 2018, 12:44pm UTC](https://discuss.elastic.co/t/pushing-logs-from-mysql-to-aws-es-using-logstash/113753 "2018-01-02T12:44:36Z") Hi , I am using logstash 5.6.3 and trying to push mysql table data to aws ES service(which is open to My Server . I have followed the following plugin to integrate AWS ES service But this is not working out i am gett… --- ## [JDBC input plugin error for MongoDB](https://discuss.elastic.co/t/jdbc-input-plugin-error-for-mongodb/113086)
**Author:** [@Sarra\_Bouhenni](https://discuss.elastic.co/u/Sarra_Bouhenni)\ **Replies:** 1\ **Last updated:** [January 2, 2018, 12:30pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-error-for-mongodb/113086 "2018-01-02T12:30:59Z")
Hello, I am using Logstash to replcate a database from MongoDB to Elasticsearch using Logstash 6.0.0, here is my config file: input{ jdbc{ jdbc\_driver\_library =\> "/usr/share/logstash/driver/mongodb-driver-3… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=846) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=848)