# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=876 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 877 --- ## [CSV parse failure and some untraceable records](https://discuss.elastic.co/t/csv-parse-failure-and-some-untraceable-records/107499)
**Author:** [@Aditi\_Shah](https://discuss.elastic.co/u/Aditi_Shah)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 7:25am UTC](https://discuss.elastic.co/t/csv-parse-failure-and-some-untraceable-records/107499 "2017-11-14T07:25:43Z")
I have like 2 million records, out of which 40 did not get read, 11 of them are giving CSV parse failure and these rows are just like the other rows. Remaining 29, I cannot track which rows are they. There is no informa… --- ## [Best practice](https://discuss.elastic.co/t/best-practice/106559)
**Author:** [@Rahulkumar](https://discuss.elastic.co/u/Rahulkumar)\ **Replies:** 7\ **Last updated:** [November 14, 2017, 6:57am UTC](https://discuss.elastic.co/t/best-practice/106559 "2017-11-14T06:57:09Z")
I wan to parse logfile(which is at client side) and and display content to kibana(which is hosted in offshore). I am planning to use logstash at client side and elastic search and kibana both locally. I had heard abou… --- ## [How to improve Performance](https://discuss.elastic.co/t/how-to-improve-performance/107167)
**Author:** [@yoga-sara](https://discuss.elastic.co/u/yoga-sara)\ **Replies:** 5\ **Last updated:** [November 14, 2017, 6:46am UTC](https://discuss.elastic.co/t/how-to-improve-performance/107167 "2017-11-14T06:46:01Z")
Hi, I want to know which improves performance. I have two methods like below, single pipeline with single filter having multiple plugin. 2.Single conf having multiple pipeline (multiple filter and multiple output) T… --- ## [Index output not generated to ElasticSearch](https://discuss.elastic.co/t/index-output-not-generated-to-elasticsearch/107429)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 6:45am UTC](https://discuss.elastic.co/t/index-output-not-generated-to-elasticsearch/107429 "2017-11-14T06:45:05Z")
In my filter I did: mutate { add\_field =\> { "\[@metadata\]\[indexname\]" =\> "CMServer" } convert =\> { "cputimeinmillisec" =\> "float" } } and my output is: output {… --- ## [Filter on a number in a log catch](https://discuss.elastic.co/t/filter-on-a-number-in-a-log-catch/107316)
**Author:** [@pir1981](https://discuss.elastic.co/u/pir1981)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 6:27am UTC](https://discuss.elastic.co/t/filter-on-a-number-in-a-log-catch/107316 "2017-11-14T06:27:25Z")
Hi, I have a message filebeat -- ERROR -- Nov 11 23:59:34 server25 smartd\[12032\]: Device: /dev/sda \[SAT\], SMART Prefailure Atribute: 1Raw\_Read\_Error\_Rate changes from XX to YY Where XX is a number and YY is a number i… --- ## [Unable to change output index according to event fields](https://discuss.elastic.co/t/unable-to-change-output-index-according-to-event-fields/107240)
**Author:** [@pravj](https://discuss.elastic.co/u/pravj)\ **Replies:** 3\ **Last updated:** [November 14, 2017, 6:25am UTC](https://discuss.elastic.co/t/unable-to-change-output-index-according-to-event-fields/107240 "2017-11-14T06:25:46Z")
I want to use different indices for "filebeat" and "metricbeat" input sources, shipping events to the logstash. Here is the logstash.conf we are using. input { beats { ssl =\> false host =\> "0.0.… --- ## [Can you use variables in logstash to account for multiple WebSphere hosts?](https://discuss.elastic.co/t/can-you-use-variables-in-logstash-to-account-for-multiple-websphere-hosts/107208)
**Author:** [@Blake\_2112](https://discuss.elastic.co/u/Blake_2112)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 6:19am UTC](https://discuss.elastic.co/t/can-you-use-variables-in-logstash-to-account-for-multiple-websphere-hosts/107208 "2017-11-14T06:19:36Z")
I currently have logstash configured to process WebSphere SystemOut.log files. No issues there. I have created a mutate construct within the filter section to enable us to add hostname, nodename, etc., so that we can pro… --- ## [Overriding logstash.yml in Docker not working](https://discuss.elastic.co/t/overriding-logstash-yml-in-docker-not-working/107197)
**Author:** [@dolanmk](https://discuss.elastic.co/u/dolanmk)\ **Replies:** 1\ **Last updated:** [November 13, 2017, 9:45pm UTC](https://discuss.elastic.co/t/overriding-logstash-yml-in-docker-not-working/107197 "2017-11-13T21:45:34Z")
Hi, I am trying to run Logstash in the Docker Toolbox (I have Windows 7) and I am having a few difficulties. The default logstash.yml from the Logstash image has x-pack preferences listed in it. I want to override this… --- ## [Json parsing](https://discuss.elastic.co/t/json-parsing/105464)
**Author:** [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Replies:** 4\ **Last updated:** [November 13, 2017, 9:14pm UTC](https://discuss.elastic.co/t/json-parsing/105464 "2017-11-13T21:14:10Z")
Hi folks, could you please help to understand 'json' data processing eg.: json data file: ssl.json: \[ { "certname": "host1", "environment": "nk", "name": "ruby", "value": { "platform": "x86\_64-linux", "sitedi… --- ## [Can't install logstash plugins, trying to install them breaks the logstash installation](https://discuss.elastic.co/t/cant-install-logstash-plugins-trying-to-install-them-breaks-the-logstash-installation/103689)
**Author:** [@A.McBride](https://discuss.elastic.co/u/A.McBride)\ **Replies:** 4\ **Last updated:** [November 13, 2017, 7:29pm UTC](https://discuss.elastic.co/t/cant-install-logstash-plugins-trying-to-install-them-breaks-the-logstash-installation/103689 "2017-11-13T19:29:28Z")
I'm trying to install the email plugin for logstash by using "logstash-plugin install logstash-output-email", however when I do it I get the following error: Validating logstash-output-email Installing logstash-output-… --- ## [Can't install x-pack for logstash 6.0](https://discuss.elastic.co/t/cant-install-x-pack-for-logstash-6-0/104733)
**Author:** [@Cornoualis](https://discuss.elastic.co/u/Cornoualis)\ **Replies:** 7\ **Last updated:** [November 13, 2017, 5:25pm UTC](https://discuss.elastic.co/t/cant-install-x-pack-for-logstash-6-0/104733 "2017-11-13T17:25:13Z")
Hi, I tried to install x-pack for logstash 6.0 using the command "logstash-plugin install x-pack" and I have the following errors: ⇒ ./logstash-plugin install x-pack … --- ## [Logstash elasticsearch output plugin takes extremely long time](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-takes-extremely-long-time/107198)
**Author:** [@shaowencheng](https://discuss.elastic.co/u/shaowencheng)\ **Replies:** 4\ **Last updated:** [November 13, 2017, 4:20pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-takes-extremely-long-time/107198 "2017-11-13T16:20:26Z")
Could someone help identify the problem and advice? Thanks in advance. The following logstash stats are obtained from the logstash management API: curl –X GET http://127.0.0.1/\_node/stats Cluster configuration is as f… --- ## [\[ERROR\]\[logstash.pipeline \] Error registering plugin {:plugin=\>"#\ **Author:** [@ppiety](https://discuss.elastic.co/u/ppiety)\ **Replies:** 0\ **Last updated:** [November 13, 2017, 3:55pm UTC](https://discuss.elastic.co/t/error-logstash-pipeline-error-registering-plugin-plugin-logstash-filterdelegator-0x5223bdbe/107423 "2017-11-13T15:55:46Z") I am new to logstash. Getting the error below. Any thoughts appreciated. ubuntu@ip-172-31-32-110:~/Downloads/logstash-5.6.4$ bin/logstash -f "helpdesk.config" Sending Logstash's logs to /home/ubuntu/Downloads/logstas… --- ## [New help assigning a date to @timestamp](https://discuss.elastic.co/t/new-help-assigning-a-date-to-timestamp/107107)
**Author:** [@mhallman](https://discuss.elastic.co/u/mhallman)\ **Replies:** 10\ **Last updated:** [November 13, 2017, 2:41pm UTC](https://discuss.elastic.co/t/new-help-assigning-a-date-to-timestamp/107107 "2017-11-13T14:41:25Z")
Hello, NOOB here. Trying to assign a date that I have constructed from other fields in a csv to @timestamp. The field that is concatenated shows up in the imported data but I can get it into the @timestamp field. Her… --- ## [Google PubSub Input Plugin: "Error: certificate verify failed"](https://discuss.elastic.co/t/google-pubsub-input-plugin-error-certificate-verify-failed/105564)
**Author:** [@livioso](https://discuss.elastic.co/u/livioso)\ **Replies:** 1\ **Last updated:** [November 13, 2017, 2:35pm UTC](https://discuss.elastic.co/t/google-pubsub-input-plugin-error-certificate-verify-failed/105564 "2017-11-13T14:35:42Z")
I'm fairly new to Logstash. Really like it so far! :slight\_smile: I'm struggling to make the Google PubSub Plugin work, I followed all the setup steps (creating a topic, creating a dedicated service account etc. pp.) My… --- ## [Logtstash kv](https://discuss.elastic.co/t/logtstash-kv/107267)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 2\ **Last updated:** [November 13, 2017, 12:06pm UTC](https://discuss.elastic.co/t/logtstash-kv/107267 "2017-11-13T12:06:22Z")
I converted my mysql field value to kv using kv filter... filter{ kv{ field\_split =\> "&?" } } but when i saw the result....it was in key and value form only...but in string codes...means my line break to kv form bu… --- ## [Unsupported platform: x86\_64-windows](https://discuss.elastic.co/t/unsupported-platform-x86-64-windows/107375)
**Author:** [@linfeng627](https://discuss.elastic.co/u/linfeng627)\ **Replies:** 1\ **Last updated:** [November 13, 2017, 11:36am UTC](https://discuss.elastic.co/t/unsupported-platform-x86-64-windows/107375 "2017-11-13T11:36:23Z")
\[image\] --- ## [Having environment variable in SQL file is not working](https://discuss.elastic.co/t/having-environment-variable-in-sql-file-is-not-working/107204)
**Author:** [@Karthik\_S](https://discuss.elastic.co/u/Karthik_S)\ **Replies:** 0\ **Last updated:** [November 10, 2017, 3:07pm UTC](https://discuss.elastic.co/t/having-environment-variable-in-sql-file-is-not-working/107204 "2017-11-10T15:07:04Z")
Hi, I've tried below things but 1st one is working and the 2nd one is failing. In command line i did "export SCHEMANAME=TEST " 1st One: Worked Conf file Plugin: JDBC statement=\> "select \* from ${SCHEMANAME}.TABLE1 … --- ## [Logstash indexes number](https://discuss.elastic.co/t/logstash-indexes-number/107335)
**Author:** [@cdra](https://discuss.elastic.co/u/cdra)\ **Replies:** 3\ **Last updated:** [November 13, 2017, 8:16am UTC](https://discuss.elastic.co/t/logstash-indexes-number/107335 "2017-11-13T08:16:43Z")
Default installation create a logstash index and a winlogbeat index everyday. Where could i find this configuration ?? What should I do if I wanted to create a index by week (for example...) --- ## [Filter Kafka JSON messages with Logstash grok](https://discuss.elastic.co/t/filter-kafka-json-messages-with-logstash-grok/107251)
**Author:** [@Konstantin\_Blag](https://discuss.elastic.co/u/Konstantin_Blag)\ **Replies:** 1\ **Last updated:** [November 13, 2017, 6:43am UTC](https://discuss.elastic.co/t/filter-kafka-json-messages-with-logstash-grok/107251 "2017-11-13T06:43:53Z")
I try to filter kafka json messages only for one Country - Germany (DE). To do that I have to write a grok expression. Can anyone help me in writing a grok pattern for this json? {"table":"ORDERS","type":"I","payload":{… --- ## [Getting errors when accessing localhost:5044 Invalid Frame Type,Received:69 and 84](https://discuss.elastic.co/t/getting-errors-when-accessing-localhost-5044-invalid-frame-type-received-69-and-84/107222)
**Author:** [@yakiza](https://discuss.elastic.co/u/yakiza)\ **Replies:** 1\ **Last updated:** [November 13, 2017, 6:40am UTC](https://discuss.elastic.co/t/getting-errors-when-accessing-localhost-5044-invalid-frame-type-received-69-and-84/107222 "2017-11-13T06:40:50Z")
Hello i have installed the ELK stack following the official documentaiton provided from the website i have installed the Elasticsearch Logstash Filebeat Kibana all the services are running with no errors but when i… --- ## [Install .gem file offline in Logstash](https://discuss.elastic.co/t/install-gem-file-offline-in-logstash/107207)
**Author:** [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Replies:** 3\ **Last updated:** [November 13, 2017, 6:28am UTC](https://discuss.elastic.co/t/install-gem-file-offline-in-logstash/107207 "2017-11-13T06:28:37Z")
Hi i am trying to install the logstash-codec-json\_lines plugin on one of our Logstash server which does not have internet access . I get following error below message when trying to install the plugin . I have tried to… --- ## [\[Solved\] ELK - Logstash not processing data](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318)
**Author:** [@Adam\_Black](https://discuss.elastic.co/u/Adam_Black)\ **Replies:** 6\ **Last updated:** [November 13, 2017, 12:32am UTC](https://discuss.elastic.co/t/solved-elk-logstash-not-processing-data/107318 "2017-11-13T00:32:29Z")
for starters, please excuse me if I am not posting the correct details or in the correct format. If there is anything I need to post, please let me know. History: I was running security onion and upgraded to their Alph… --- ## [Create nested field for aggregate plugin output?](https://discuss.elastic.co/t/create-nested-field-for-aggregate-plugin-output/106087)
**Author:** [@khommel](https://discuss.elastic.co/u/khommel)\ **Replies:** 1\ **Last updated:** [November 12, 2017, 5:40pm UTC](https://discuss.elastic.co/t/create-nested-field-for-aggregate-plugin-output/106087 "2017-11-12T17:40:04Z")
I'm using the Aggregate plugin to calculate a value. I'm trying to set the value in a nested field, creating the field at the same time. My code is: code =\> "event.set('\[msg\]\[duration\]', event.get('\[msg\]\[time\]') - map\['… --- ## [Can I store the each filter outputs in elasticSearch?](https://discuss.elastic.co/t/can-i-store-the-each-filter-outputs-in-elasticsearch/106637)
**Author:** [@Harim\_Tong](https://discuss.elastic.co/u/Harim_Tong)\ **Replies:** 2\ **Last updated:** [November 12, 2017, 11:34am UTC](https://discuss.elastic.co/t/can-i-store-the-each-filter-outputs-in-elasticsearch/106637 "2017-11-12T11:34:53Z")
e.g. the filters have 3 phases: 1.store all the orignal messages; 2.filter the validate data; 3.do the business logic; I want to save all of the 3 phase results for debug and troubleshoot,is it possible? or only can … --- ## [Reindex Elasticsearch using new mapping](https://discuss.elastic.co/t/reindex-elasticsearch-using-new-mapping/106534)
**Author:** [@sreejithps303](https://discuss.elastic.co/u/sreejithps303)\ **Replies:** 6\ **Last updated:** [November 12, 2017, 4:46am UTC](https://discuss.elastic.co/t/reindex-elasticsearch-using-new-mapping/106534 "2017-11-12T04:46:27Z")
Hi , I have 2 new fields added to the index and i want to renidex the old indices to include these two fields. I have a mapping csv file where i have an existing field and corresponding value for two new fields. I wan… --- ## [\[Logstash\] - Exception in Pipelineworker](https://discuss.elastic.co/t/logstash-exception-in-pipelineworker/105191)
**Author:** [@azam](https://discuss.elastic.co/u/azam)\ **Replies:** 3\ **Last updated:** [November 11, 2017, 9:51pm UTC](https://discuss.elastic.co/t/logstash-exception-in-pipelineworker/105191 "2017-11-11T21:51:11Z")
Hi everyone I am a new user to ELK stack. I tried to setup elastic stack on my local machine but logstash doesn't work for me. I tried to debug, it showed an error "Exception in pipelineworker". Please help ! My logsta… --- ## [Logstash Translate Filter Can't Convert String Into Hash Loading Dictionary](https://discuss.elastic.co/t/logstash-translate-filter-cant-convert-string-into-hash-loading-dictionary/107112)
**Author:** [@cchamberlain](https://discuss.elastic.co/u/cchamberlain)\ **Replies:** 4\ **Last updated:** [November 11, 2017, 5:45am UTC](https://discuss.elastic.co/t/logstash-translate-filter-cant-convert-string-into-hash-loading-dictionary/107112 "2017-11-11T05:45:48Z")
Hello, May I please request your advice? I am on Windows Server 2012 R2, Logstash 5.4.0. I am enriching either SQL Server results or text (shown) via a dictionary YAML file. However, the error is : error=\>"LogStash:… --- ## [How to join fields from multi events into single event?](https://discuss.elastic.co/t/how-to-join-fields-from-multi-events-into-single-event/106364)
**Author:** [@djgerhab](https://discuss.elastic.co/u/djgerhab)\ **Replies:** 4\ **Last updated:** [November 10, 2017, 8:48pm UTC](https://discuss.elastic.co/t/how-to-join-fields-from-multi-events-into-single-event/106364 "2017-11-10T20:48:13Z")
Hi, I feel like my issue should be a common one, however I have spent days trying to find the answer to no avail... Here is what I am trying to accomplish: I have data from 2 separate databases that share a common prima… --- ## [How to index 2 indexes using conditions?](https://discuss.elastic.co/t/how-to-index-2-indexes-using-conditions/107221)
**Author:** [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Replies:** 1\ **Last updated:** [November 10, 2017, 7:43pm UTC](https://discuss.elastic.co/t/how-to-index-2-indexes-using-conditions/107221 "2017-11-10T19:43:34Z")
Using 5.6.2 stack Hi, I have the following pipeline and it works well... But I would like to index a specific "application" to 2 indexes. Based on the field "executor" split below. input { beats { port =\> "5043" … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=875) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=877)