# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=346 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 347 --- ## [Add colonne in CSV file](https://discuss.elastic.co/t/add-colonne-in-csv-file/235981)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 0\ **Last updated:** [June 5, 2020, 4:01pm UTC](https://discuss.elastic.co/t/add-colonne-in-csv-file/235981 "2020-06-05T16:01:19Z")
hello everyone , please i need ur help i want add new columns in my logstash config file but i don't how do it ?? there is my config file ?? input { beats { port =\> "5044" } } filter { csv { separator =\> "," colu… --- ## [Push log files from different system and visualize it in kibana which is installed only in one system](https://discuss.elastic.co/t/push-log-files-from-different-system-and-visualize-it-in-kibana-which-is-installed-only-in-one-system/235935)
**Author:** [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\ **Replies:** 3\ **Last updated:** [June 5, 2020, 10:27pm UTC](https://discuss.elastic.co/t/push-log-files-from-different-system-and-visualize-it-in-kibana-which-is-installed-only-in-one-system/235935 "2020-06-05T22:27:26Z")
Hi, I want to push the log files from different machines where different log files are being generated. Now I want that whenever a log file is generated in any system, it gets updated into kibana . So that this way I do… --- ## [XML Parsing incomplete](https://discuss.elastic.co/t/xml-parsing-incomplete/235979)
**Author:** [@Noe253](https://discuss.elastic.co/u/Noe253)\ **Replies:** 2\ **Last updated:** [June 5, 2020, 5:03pm UTC](https://discuss.elastic.co/t/xml-parsing-incomplete/235979 "2020-06-05T17:03:27Z")
Hello everyone, I'm sending xml logs into Logstash using Filebeat, it worked perfectly the first time I set up my configuration, but since I restarted Logstash service using "systemctl restart logstash.service" (I'm on … --- ## [Logstash error - Output](https://discuss.elastic.co/t/logstash-error-output/235840)
**Author:** [@jgonzalez](https://discuss.elastic.co/u/jgonzalez)\ **Replies:** 4\ **Last updated:** [June 5, 2020, 4:58pm UTC](https://discuss.elastic.co/t/logstash-error-output/235840 "2020-06-05T16:58:29Z")
Hello All, I have an error in my logstash config file and i have not able to find it. Please if someone can help me This is the error: \[2020-06-04T15:00:07,633\]\[ERROR\]\[logstash.agent \] Failed to execute act… --- ## ["File does not contain valid private key:/usr/share/logstash/ssl/server.key" error in input pipeline Logstash. How fix it?](https://discuss.elastic.co/t/file-does-not-contain-valid-private-key-usr-share-logstash-ssl-server-key-error-in-input-pipeline-logstash-how-fix-it/235980)
**Author:** [@Andrew\_Foxis](https://discuss.elastic.co/u/Andrew_Foxis)\ **Replies:** 0\ **Last updated:** [June 5, 2020, 3:58pm UTC](https://discuss.elastic.co/t/file-does-not-contain-valid-private-key-usr-share-logstash-ssl-server-key-error-in-input-pipeline-logstash-how-fix-it/235980 "2020-06-05T15:58:11Z")
Hi all! I get certificate from MS CA. Then convert it: openssl pkcs12 -in test.pfx -out ca.crt -nodes -nokeys -cacerts openssl pkcs12 -in test.pfx -clcerts -nokeys -out server.crt openssl pkcs12 -in test.p… --- ## [Pipeline to pipeline communication - logstash data](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-logstash-data/235914)
**Author:** [@Pankaj\_Jadhav](https://discuss.elastic.co/u/Pankaj_Jadhav)\ **Replies:** 2\ **Last updated:** [June 5, 2020, 3:46pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-logstash-data/235914 "2020-06-05T15:46:59Z")
Pipeline1 :- ds\_test.conf input { generator { count =\> 1 lines =\> \[ '' \] } } filter { mutate { add\_field =\> { "\[eventHeader\]\[applicationId\]" =\> "A" "\[eventHeader\]\[objectCode\]" =\> 1 } } mutate { add\_field =\> { "… --- ## [Gelf input with multiline](https://discuss.elastic.co/t/gelf-input-with-multiline/235918)
**Author:** [@111228](https://discuss.elastic.co/u/111228)\ **Replies:** 3\ **Last updated:** [June 5, 2020, 3:25pm UTC](https://discuss.elastic.co/t/gelf-input-with-multiline/235918 "2020-06-05T15:25:27Z")
Hello all Tell me how to configure correctly PHP errors with Stack trace in multiline. I have such messages 1 message ) \[05-Jun-2020 11:35:39 Europe/Kiev\] PHP Fatal error: require(): Failed opening required 'askldja… --- ## [Logstash tcp input load balancing](https://discuss.elastic.co/t/logstash-tcp-input-load-balancing/235924)
**Author:** [@daniele.saccon](https://discuss.elastic.co/u/daniele.saccon)\ **Replies:** 1\ **Last updated:** [June 5, 2020, 2:10pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-load-balancing/235924 "2020-06-05T14:10:18Z")
Hi everybody, I have 2 logstash instances in 2 different hosts with the same configurations and a source that sends in tcp. It's possible to have a logstash cluster? Or do I have to put a load balancer that send events… --- ## [How can I flatten an array in my document](https://discuss.elastic.co/t/how-can-i-flatten-an-array-in-my-document/235807)
**Author:** [@genehunter29009](https://discuss.elastic.co/u/genehunter29009)\ **Replies:** 5\ **Last updated:** [June 5, 2020, 8:58am UTC](https://discuss.elastic.co/t/how-can-i-flatten-an-array-in-my-document/235807 "2020-06-05T08:58:15Z")
I am attempting to flatten an array either in logstash or ingestion processor. "clientType" : "Desktop Browser", "stringProperties" : \[ { "key" : "usertype", "applicationId" : "Client", "internalApplicationId" : "A… --- ## [ELK Integration with SAP HANA](https://discuss.elastic.co/t/elk-integration-with-sap-hana/235913)
**Author:** [@Shivakarasani](https://discuss.elastic.co/u/Shivakarasani)\ **Replies:** 0\ **Last updated:** [June 5, 2020, 8:43am UTC](https://discuss.elastic.co/t/elk-integration-with-sap-hana/235913 "2020-06-05T08:43:17Z")
anyone integrated ELK with SAP HANA DB, I am trying to integrate using JDBC connector and not able to connect. Is there any document available for this? I have not an appropriate one till now. my conf file jdbc { jd… --- ## [How to split log data into message fields when we have dynamic logs](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725)
**Author:** [@hemant\_472](https://discuss.elastic.co/u/hemant_472)\ **Replies:** 4\ **Last updated:** [June 5, 2020, 8:40am UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725 "2020-06-05T08:40:26Z")
i have created some log patterns in logstash but when i get new log pattern it is parsed into the kibana as a message filed but is not split up into other fields. I would like to know do i need to add log patterns for e… --- ## [Limit of total fields \[1000\] in index \[p\_bs\_api\_abcd\_2020\_23\] has been exceeded](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476)
**Author:** [@Saravana\_Maadavan](https://discuss.elastic.co/u/Saravana_Maadavan)\ **Replies:** 8\ **Last updated:** [June 5, 2020, 8:25am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-p-bs-api-abcd-2020-23-has-been-exceeded/235476 "2020-06-05T08:25:05Z")
I have followed the steps discussed in link Increase total fields limit via creation of index in logstash but not able to resolve it. Kindly find below few snippets of the configuration files. output { elasticsearch { … --- ## [Data filter input](https://discuss.elastic.co/t/data-filter-input/235796)
**Author:** [@seddikalaouiismaili](https://discuss.elastic.co/u/seddikalaouiismaili)\ **Replies:** 2\ **Last updated:** [June 5, 2020, 7:56am UTC](https://discuss.elastic.co/t/data-filter-input/235796 "2020-06-05T07:56:26Z")
Hello For a technical constraint we want to change the format of the data in the logs processed by filebeat . On the logs there are time fields that will be modified to milliseconds in input, in this case we want to ap… --- ## [Logstash stop working silently (workers die out)](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722)
**Author:** [@Aldin\_Osmanagic](https://discuss.elastic.co/u/Aldin_Osmanagic)\ **Replies:** 2\ **Last updated:** [June 5, 2020, 7:34am UTC](https://discuss.elastic.co/t/logstash-stop-working-silently-workers-die-out/235722 "2020-06-05T07:34:50Z")
Hello, At the customer site, we are using ElasticStack to gather syslogs from the network devices. Logstash is receiving around 4000 syslogs/sec. We are using Logstash version 6.8.8. The problem is that every 2-5 hour… --- ## [Pipeline to pipeline communication - logstash](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-logstash/235821)
**Author:** [@Pankaj\_Jadhav](https://discuss.elastic.co/u/Pankaj_Jadhav)\ **Replies:** 8\ **Last updated:** [June 5, 2020, 7:09am UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-logstash/235821 "2020-06-05T07:09:10Z")
Pipeline1 kafka { bootstrap\_servers =\> "172.20.188.11:9092,172.20.188.31:9092,172.20.188.30:9092" topics =\> \["DS\_Test"\] decorate\_events =\> true group\_id =\> "ds\_test" codec =\> json max\_poll\_recor… --- ## [Adding a field (via logstash) to a monthly index with a template](https://discuss.elastic.co/t/adding-a-field-via-logstash-to-a-monthly-index-with-a-template/235651)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 3\ **Last updated:** [June 5, 2020, 6:51am UTC](https://discuss.elastic.co/t/adding-a-field-via-logstash-to-a-monthly-index-with-a-template/235651 "2020-06-05T06:51:03Z")
I understand that the template is applied at index creation, and I was wandering what will happen if I add a field via logstash in the middle of the month? the added field will be usable for agregations and searches? j… --- ## [Insert old MongoDB data for ES](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626)
**Author:** [@FernandoMOTA](https://discuss.elastic.co/u/FernandoMOTA)\ **Replies:** 12\ **Last updated:** [June 4, 2020, 8:26pm UTC](https://discuss.elastic.co/t/insert-old-mongodb-data-for-es/235626 "2020-06-04T20:26:03Z")
My logstash configuration for inserting data in Elasticsearch inserts only new data, it does not insert the old DB documents. My .conf file is as follows: input { mongodb { uri =\> 'mongodb://localhost:27017… --- ## [Problem with splitted logstash config file in logstash 7.7](https://discuss.elastic.co/t/problem-with-splitted-logstash-config-file-in-logstash-7-7/235788)
**Author:** [@jammann](https://discuss.elastic.co/u/jammann)\ **Replies:** 2\ **Last updated:** [June 4, 2020, 7:52pm UTC](https://discuss.elastic.co/t/problem-with-splitted-logstash-config-file-in-logstash-7-7/235788 "2020-06-04T19:52:30Z")
We're migrating from logstash 6.5.4 to 7.7.0. In 6.5.4 we had our config files all in one directory, with a naming convention like 01\_. We pointed path.config to that directory and logstash concatenated the files as if i… --- ## [Use the right codec field "date"](https://discuss.elastic.co/t/use-the-right-codec-field-date/235751)
**Author:** [@souma](https://discuss.elastic.co/u/souma)\ **Replies:** 1\ **Last updated:** [June 4, 2020, 7:51pm UTC](https://discuss.elastic.co/t/use-the-right-codec-field-date/235751 "2020-06-04T19:51:15Z")
Hi everyone, i use the logstash file below to send data to elasticsearch. input{ file{ path =\> "C:/Users/SOUMAYA/Desktop/bureau.csv" start\_position =\> "beginning" sincedb\_path =\> "NUL" codec… --- ## [How to create ruby class variables mapped to a given ID?](https://discuss.elastic.co/t/how-to-create-ruby-class-variables-mapped-to-a-given-id/234887)
**Author:** [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Replies:** 9\ **Last updated:** [June 4, 2020, 7:26pm UTC](https://discuss.elastic.co/t/how-to-create-ruby-class-variables-mapped-to-a-given-id/234887 "2020-06-04T19:26:52Z")
Hi, I'm using below Ruby filter to find the elapsed time between a start and end event for a given user. The code works fine with the events coming in sequence. But, I have event logs coming from multiple users. I have … --- ## [Issue Developing Filter Plugin on Windows](https://discuss.elastic.co/t/issue-developing-filter-plugin-on-windows/234932)
**Author:** [@Micah\_Hunsberger](https://discuss.elastic.co/u/Micah_Hunsberger)\ **Replies:** 1\ **Last updated:** [June 4, 2020, 6:43pm UTC](https://discuss.elastic.co/t/issue-developing-filter-plugin-on-windows/234932 "2020-06-04T18:43:32Z")
I am trying to write a new logstash filter, and I can't seem to get the rspec tests to run. I followed the guide posted here: https://www.elastic.co/guide/en/logstash/current/filter-new-plugin.html this is my process: P… --- ## [Date parsing errors I need help to debug](https://discuss.elastic.co/t/date-parsing-errors-i-need-help-to-debug/235733)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 4\ **Last updated:** [June 4, 2020, 6:35pm UTC](https://discuss.elastic.co/t/date-parsing-errors-i-need-help-to-debug/235733 "2020-06-04T18:35:45Z")
I am getting this error in my logstash error logs: logstash\[24891\]: \[2020-06-04T12:27:22,478\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\]\[5a0b38d56348d6bc3ed40ef3de0fd75852bd567bf4973ad3b30fa5a1538d879b\] Could not ind… --- ## [Logstash shut down](https://discuss.elastic.co/t/logstash-shut-down/235614)
**Author:** [@ajames](https://discuss.elastic.co/u/ajames)\ **Replies:** 9\ **Last updated:** [June 4, 2020, 5:08pm UTC](https://discuss.elastic.co/t/logstash-shut-down/235614 "2020-06-04T17:08:14Z")
Sending Logstash logs to C:/ElasticSearch/logstash-7.7.0/logs which is now configured via log4j2.properties \[2020-06-03T11:12:10,384\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file because m… --- ## [Does ILM work in Logstash with custom index names?](https://discuss.elastic.co/t/does-ilm-work-in-logstash-with-custom-index-names/235816)
**Author:** [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Replies:** 0\ **Last updated:** [June 4, 2020, 4:50pm UTC](https://discuss.elastic.co/t/does-ilm-work-in-logstash-with-custom-index-names/235816 "2020-06-04T16:50:18Z")
Hi there, we are currently using a single, generic "logstash-\*" ElasticSearch output which works quite well with the default ILM defaults of 30 days or 50GB and the automatic alias naming convention of logstash-2020.04.0… --- ## [Server 2016 Logstash 7.7 continually stops and restarts with errors](https://discuss.elastic.co/t/server-2016-logstash-7-7-continually-stops-and-restarts-with-errors/235633)
**Author:** [@Elkmaster20](https://discuss.elastic.co/u/Elkmaster20)\ **Replies:** 2\ **Last updated:** [June 4, 2020, 3:28pm UTC](https://discuss.elastic.co/t/server-2016-logstash-7-7-continually-stops-and-restarts-with-errors/235633 "2020-06-04T15:28:42Z")
Here is error in the log file: \[2020-06-01T00:04:16,445\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"7.7.0"} \[2020-06-01T00:04:16,986\]\[INFO \]\[logstash.config.source.local.configpathloader\]… --- ## [What is the command to check the status of logstash in windows](https://discuss.elastic.co/t/what-is-the-command-to-check-the-status-of-logstash-in-windows/235774)
**Author:** [@Rahul\_Ravichandran](https://discuss.elastic.co/u/Rahul_Ravichandran)\ **Replies:** 0\ **Last updated:** [June 4, 2020, 1:59pm UTC](https://discuss.elastic.co/t/what-is-the-command-to-check-the-status-of-logstash-in-windows/235774 "2020-06-04T13:59:53Z")
Hi How do i check for the logstash status in windows. What is the command that is used ? --- ## [Cluster ( or High Avalibility) at Logstash level](https://discuss.elastic.co/t/cluster-or-high-avalibility-at-logstash-level/235338)
**Author:** [@Yashwant\_Shettigar](https://discuss.elastic.co/u/Yashwant_Shettigar)\ **Replies:** 3\ **Last updated:** [June 4, 2020, 1:47pm UTC](https://discuss.elastic.co/t/cluster-or-high-avalibility-at-logstash-level/235338 "2020-06-04T13:47:10Z")
Hi Team, I need to know how can I achieve HA(High Availability) at logstash level (2 logstash servers) Reason: When we route logs from Firewall devices, Storage devices, etc we can define only IP address for log mana… --- ## [How to get filter value metadata in input](https://discuss.elastic.co/t/how-to-get-filter-value-metadata-in-input/235674)
**Author:** [@Pankaj\_Jadhav](https://discuss.elastic.co/u/Pankaj_Jadhav)\ **Replies:** 6\ **Last updated:** [June 4, 2020, 1:29pm UTC](https://discuss.elastic.co/t/how-to-get-filter-value-metadata-in-input/235674 "2020-06-04T13:29:41Z")
Hi Guys, input{ kafka { bootstrap\_servers =\> "172.20.188.11:9092,172.20.188.31:9092,172.20.188.30:9092" topics =\> \["DS\_Test"\] decorate\_events =\> true group\_id =\> "ds\_test" codec =\> json max\_poll\_records =\> "100" … --- ## [Does kv filter exclude keys supports regex match](https://discuss.elastic.co/t/does-kv-filter-exclude-keys-supports-regex-match/235732)
**Author:** [@kannan\_raj](https://discuss.elastic.co/u/kannan_raj)\ **Replies:** 1\ **Last updated:** [June 4, 2020, 1:21pm UTC](https://discuss.elastic.co/t/does-kv-filter-exclude-keys-supports-regex-match/235732 "2020-06-04T13:21:19Z")
Hi Team, I would like to know that kv filter exclude keys support regex. kv { allow\_duplicate\_values =\> false field\_split =\> ",\\s" exclude\_keys =\> \["^\[0-9\]+"\] } I have tried … --- ## [Logstash convert number to string in array of objects](https://discuss.elastic.co/t/logstash-convert-number-to-string-in-array-of-objects/235741)
**Author:** [@Mahebub\_Sayyed](https://discuss.elastic.co/u/Mahebub_Sayyed)\ **Replies:** 1\ **Last updated:** [June 4, 2020, 1:19pm UTC](https://discuss.elastic.co/t/logstash-convert-number-to-string-in-array-of-objects/235741 "2020-06-04T13:19:23Z")
Hello, I have 2 logstash flows writing to the same index. For first flow value for "code" is number and for second flow its string. Please find the sample below "Response": { "ResponsePayLoad": { "name": "OK", … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=345) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=347)