# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=213 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 214 --- ## [Change the index from winglogbeat to metricbeat](https://discuss.elastic.co/t/change-the-index-from-winglogbeat-to-metricbeat/277988)
**Author:** [@Angelo\_Bryan](https://discuss.elastic.co/u/Angelo_Bryan)\ **Replies:** 0\ **Last updated:** [July 6, 2021, 6:03pm UTC](https://discuss.elastic.co/t/change-the-index-from-winglogbeat-to-metricbeat/277988 "2021-07-06T18:03:47Z")
Hello elastic group, I have a question, I am trying to change the index from winglogbeat to metricbeat making the following configuration in the logstash (conf.d), but it has not worked for me, I would like to have your … --- ## [Data Validation in Logstash](https://discuss.elastic.co/t/data-validation-in-logstash/277922)
**Author:** [@Pallavibhushan](https://discuss.elastic.co/u/Pallavibhushan)\ **Replies:** 2\ **Last updated:** [July 6, 2021, 4:43pm UTC](https://discuss.elastic.co/t/data-validation-in-logstash/277922 "2021-07-06T16:43:58Z")
Hi, I am trying data validation in logstash. Example : column 1 in CSV should only allow number (double data type). No other text should be allowed if any other string found then trigger mail. Similarly, need date va… --- ## [Capturing servername and appending/propogating it to all other lines of logfile](https://discuss.elastic.co/t/capturing-servername-and-appending-propogating-it-to-all-other-lines-of-logfile/277876)
**Author:** [@Nanelastic](https://discuss.elastic.co/u/Nanelastic)\ **Replies:** 2\ **Last updated:** [July 6, 2021, 3:39pm UTC](https://discuss.elastic.co/t/capturing-servername-and-appending-propogating-it-to-all-other-lines-of-logfile/277876 "2021-07-06T15:39:18Z")
Hello Guys, I am very new to ELK stack. I am trying to parse below mentioned log using logstash config. I figured out the grok pattern for last 3 lines. I need help in capturing the server name mentioned in either line… --- ## [Monitoring Logstash with Metricbeat](https://discuss.elastic.co/t/monitoring-logstash-with-metricbeat/277635)
**Author:** [@Bilanda](https://discuss.elastic.co/u/Bilanda)\ **Replies:** 2\ **Last updated:** [July 6, 2021, 1:42pm UTC](https://discuss.elastic.co/t/monitoring-logstash-with-metricbeat/277635 "2021-07-06T13:42:22Z")
Hello, I'm trying to monitor Logstash with Metricbeat. I already enabled Elasticsearch X-Pack with metricbeat. I followed instructions regarding Logstash monitoring with metricbeat. In "Stack monitoring", i can see m… --- ## [Send to different Logstash pipelines based on input](https://discuss.elastic.co/t/send-to-different-logstash-pipelines-based-on-input/277959)
**Author:** [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Replies:** 1\ **Last updated:** [July 6, 2021, 1:08pm UTC](https://discuss.elastic.co/t/send-to-different-logstash-pipelines-based-on-input/277959 "2021-07-06T13:08:57Z")
Hi i have two logstash pipelines in Pipelines.yml. I need to redirect events to appropriate pipelines based on input. If input is "abc", send to pipeline "pipeline 1", else send to "pipeline2". I cant handle this at sou… --- ## [Http filter plugin adds \_jsonparsefailure in tag](https://discuss.elastic.co/t/http-filter-plugin-adds-jsonparsefailure-in-tag/277744)
**Author:** [@priyaankaa](https://discuss.elastic.co/u/priyaankaa)\ **Replies:** 4\ **Last updated:** [July 6, 2021, 9:47am UTC](https://discuss.elastic.co/t/http-filter-plugin-adds-jsonparsefailure-in-tag/277744 "2021-07-06T09:47:55Z")
My logstash pipeline - input { elasticsearch { hosts =\> "http://localhost:9200" index =\> "s1test-processcode-temp" user =\> "elastic" password =\> "elastic" schedule =\> "\*/10 \* \* \* \*" } } filter { j… --- ## [How to index percolator field via logstash and json\_encode filter?](https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860)
**Author:** [@akb](https://discuss.elastic.co/u/akb)\ **Replies:** 1\ **Last updated:** [July 6, 2021, 8:26am UTC](https://discuss.elastic.co/t/how-to-index-percolator-field-via-logstash-and-json-encode-filter/277860 "2021-07-06T08:26:44Z")
I'm trying to index documents and populate a percolator field via logstash: Index definition PUT test\_percolate { "mappings": { "\_doc": { "properties": { "search\_name": { "type": "text" … --- ## [Unable to Connect Logstash with Elastic](https://discuss.elastic.co/t/unable-to-connect-logstash-with-elastic/277723)
**Author:** [@iraed91](https://discuss.elastic.co/u/iraed91)\ **Replies:** 1\ **Last updated:** [July 6, 2021, 3:14am UTC](https://discuss.elastic.co/t/unable-to-connect-logstash-with-elastic/277723 "2021-07-06T03:14:41Z")
Hi All! I just enabled X-PACK on my elasticsearch and was able to do all required config for elastic and kibana. Now, I have logstash installed in another host and communicating to logstash over the internet on HTTPs pr… --- ## [Ingest data via Modbus and BACnet communication protocols to Elastic](https://discuss.elastic.co/t/ingest-data-via-modbus-and-bacnet-communication-protocols-to-elastic/277598)
**Author:** [@Valdos116](https://discuss.elastic.co/u/Valdos116)\ **Replies:** 0\ **Last updated:** [July 1, 2021, 8:56pm UTC](https://discuss.elastic.co/t/ingest-data-via-modbus-and-bacnet-communication-protocols-to-elastic/277598 "2021-07-01T20:56:08Z")
Hi Team, I'm trying to ingest data from field devices that talk Modbus TCP and BACnet IP to Elastic Logstash. Any help would be appreciated. Thanks --- ## [Gzip compression results to uncompressable package](https://discuss.elastic.co/t/gzip-compression-results-to-uncompressable-package/277867)
**Author:** [@yodog](https://discuss.elastic.co/u/yodog)\ **Replies:** 1\ **Last updated:** [July 5, 2021, 6:49pm UTC](https://discuss.elastic.co/t/gzip-compression-results-to-uncompressable-package/277867 "2021-07-05T18:49:41Z")
so, after a few years, any way to work around this problem? same problem on logstash 7.13.1 zgrep '2021' zimbra.log.2021-01-30-1613154813.gz gzip: zimbra.log.2021-01-30-1613154813.gz: invalid compressed data--forma… --- ## [Http poller plugin automatically splits json response](https://discuss.elastic.co/t/http-poller-plugin-automatically-splits-json-response/277859)
**Author:** [@pjohnson](https://discuss.elastic.co/u/pjohnson)\ **Replies:** 0\ **Last updated:** [July 5, 2021, 3:54pm UTC](https://discuss.elastic.co/t/http-poller-plugin-automatically-splits-json-response/277859 "2021-07-05T15:54:01Z")
I'm using the http\_poller plugin with an endpoint that returns a json array and then I'm processing the response in a ruby script. It seems like http\_poller is automatically splitting the body into separate events for ea… --- ## [HTTP Filter - Unnest Array](https://discuss.elastic.co/t/http-filter-unnest-array/277778)
**Author:** [@RoteEdition](https://discuss.elastic.co/u/RoteEdition)\ **Replies:** 2\ **Last updated:** [July 5, 2021, 2:22pm UTC](https://discuss.elastic.co/t/http-filter-unnest-array/277778 "2021-07-05T14:22:17Z")
Hi there, I need to unnest an array (based on the workaround mentioned in this previous thread) into seperate documents. I am calling an API and getting in return an array of results. Each of these results should prefe… --- ## [Logstash buff/cache high utilization linux](https://discuss.elastic.co/t/logstash-buff-cache-high-utilization-linux/277836)
**Author:** [@Adixon\_Diaz](https://discuss.elastic.co/u/Adixon_Diaz)\ **Replies:** 0\ **Last updated:** [July 5, 2021, 12:57pm UTC](https://discuss.elastic.co/t/logstash-buff-cache-high-utilization-linux/277836 "2021-07-05T12:57:17Z")
Hi comunity, I'm seeing the utilization resources on my S.O Linux that I only use for Logstash. I could see that the buff/cache is very high. In another hand i would like to know if i configured correctly the utiliz… --- ## [Microsoft DNS - parsing different length of domain name](https://discuss.elastic.co/t/microsoft-dns-parsing-different-length-of-domain-name/277430)
**Author:** [@gmaimbourg](https://discuss.elastic.co/u/gmaimbourg)\ **Replies:** 4\ **Last updated:** [July 5, 2021, 11:45am UTC](https://discuss.elastic.co/t/microsoft-dns-parsing-different-length-of-domain-name/277430 "2021-07-05T11:45:32Z")
Hello, I'am working on Microsoft DNS log parsing and i don't know how can i parse different lengh of domain name. Here is different examples of what i want to get : (2)ui(12)powerreviews(3)com(0) into ui.powerreviews.… --- ## [Error connecting to sql database](https://discuss.elastic.co/t/error-connecting-to-sql-database/277226)
**Author:** [@francieliton\_araujo](https://discuss.elastic.co/u/francieliton_araujo)\ **Replies:** 3\ **Last updated:** [July 5, 2021, 8:10am UTC](https://discuss.elastic.co/t/error-connecting-to-sql-database/277226 "2021-07-05T08:10:59Z")
I need help to connect to the sql server database. I downloaded a connector and already put n specific folder, but it's giving an error with the following. Note; I can already connect to mysql database but sql server w… --- ## [Oracle BLOB column with jdbc input to base64](https://discuss.elastic.co/t/oracle-blob-column-with-jdbc-input-to-base64/276830)
**Author:** [@alexanders](https://discuss.elastic.co/u/alexanders)\ **Replies:** 3\ **Last updated:** [July 5, 2021, 7:54am UTC](https://discuss.elastic.co/t/oracle-blob-column-with-jdbc-input-to-base64/276830 "2021-07-05T07:54:03Z")
When fetching an Oracle BLOB column with logstash's jdbc input, I try to convert it to base64 using the following filter: ruby { code =\> 'event.set("b64", Base64.strict\_encode64(event.get("blob\_column")))' } wh… --- ## [Http\_poller - URL from another field](https://discuss.elastic.co/t/http-poller-url-from-another-field/277722)
**Author:** [@RoteEdition](https://discuss.elastic.co/u/RoteEdition)\ **Replies:** 2\ **Last updated:** [July 5, 2021, 6:33am UTC](https://discuss.elastic.co/t/http-poller-url-from-another-field/277722 "2021-07-05T06:33:48Z")
Hi there, I am trying to get some data (regarding incoming calls) from an API. I am using http\_poller and everything seems to work. The Problem: The URL has a date parameter in it. I dont want to update the URL everyda… --- ## [Logstash config for nested log](https://discuss.elastic.co/t/logstash-config-for-nested-log/277680)
**Author:** [@kriss332](https://discuss.elastic.co/u/kriss332)\ **Replies:** 10\ **Last updated:** [July 4, 2021, 1:55pm UTC](https://discuss.elastic.co/t/logstash-config-for-nested-log/277680 "2021-07-04T13:55:45Z")
Hello everyone... I have multiple event logs from windows. Converted them to csv using EvtxEcmd.exe. I combined lots of event logs into one csv file. I need to upload it to ELK. below are the column names- RecordNumb… --- ## [Log fields are not parsing as expected using grok filter](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746)
**Author:** [@prat](https://discuss.elastic.co/u/prat)\ **Replies:** 5\ **Last updated:** [July 4, 2021, 11:35pm UTC](https://discuss.elastic.co/t/log-fields-are-not-parsing-as-expected-using-grok-filter/277746 "2021-07-04T23:35:22Z")
Hi All, I am using below configuration and expecting it to get parse as per below fields. below is the pipeline.conf file. input { beats { port =\> "5044" } } filter { grok { match =\> { "mes… --- ## [Extract data from CSV column](https://discuss.elastic.co/t/extract-data-from-csv-column/277745)
**Author:** [@anon33720113](https://discuss.elastic.co/u/anon33720113)\ **Replies:** 3\ **Last updated:** [July 4, 2021, 8:49pm UTC](https://discuss.elastic.co/t/extract-data-from-csv-column/277745 "2021-07-04T20:49:44Z")
Hey I have logs as CSV and one column which is in string and has multiple values that need to extracted and stored as new fields. The column name is comment and it's value is VPN token auth failed. Destination was: 62.… --- ## [Logstash kafka input consumer group not finding all partitions](https://discuss.elastic.co/t/logstash-kafka-input-consumer-group-not-finding-all-partitions/277741)
**Author:** [@Bakkali\_Amine](https://discuss.elastic.co/u/Bakkali_Amine)\ **Replies:** 0\ **Last updated:** [July 4, 2021, 4:09pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-consumer-group-not-finding-all-partitions/277741 "2021-07-04T16:09:30Z")
I have a topic with two partitions but logstash consume only one of them. "Adding newly assigned partitions: partition-0", it is missing the other partition. And when I delete my logstash pod (I'm on kubernetes) it rand… --- ## [Imap Input - Java OutOfMemoryError](https://discuss.elastic.co/t/imap-input-java-outofmemoryerror/277720)
**Author:** [@RoteEdition](https://discuss.elastic.co/u/RoteEdition)\ **Replies:** 0\ **Last updated:** [July 4, 2021, 4:54am UTC](https://discuss.elastic.co/t/imap-input-java-outofmemoryerror/277720 "2021-07-04T04:54:17Z")
Hi there, i am trying to import about ~1000 Mails from my Inbox. I marked the first e-mail as unread, got the UID and activated uid\_tracking. The first ~60 mails got fetched like intended. I don't think it's my configu… --- ## [HEXADECIMAL string to Binary and Binary to decimal in log-stash using ruby filter](https://discuss.elastic.co/t/hexadecimal-string-to-binary-and-binary-to-decimal-in-log-stash-using-ruby-filter/277715)
**Author:** [@Kamikaze\_K](https://discuss.elastic.co/u/Kamikaze_K)\ **Replies:** 2\ **Last updated:** [July 4, 2021, 2:30am UTC](https://discuss.elastic.co/t/hexadecimal-string-to-binary-and-binary-to-decimal-in-log-stash-using-ruby-filter/277715 "2021-07-04T02:30:05Z")
I am working with logstash and I am using ruby filter to change HEX to decimal which works fine for one set of data. What I use for this is ; ruby{ code =\> "event.set('xxx', event.get('xx').hex)" } I am looking for s… --- ## [Json @timestamp vs date {}](https://discuss.elastic.co/t/json-timestamp-vs-date/277713)
**Author:** [@danielmotaleite](https://discuss.elastic.co/u/danielmotaleite)\ **Replies:** 2\ **Last updated:** [July 4, 2021, 12:23am UTC](https://discuss.elastic.co/t/json-timestamp-vs-date/277713 "2021-07-04T00:23:14Z")
From what i see, one should use in logstash the date {} to modify the event date, but setting the @timestamp field to seems to also work, at least when using json {}, @timestamp object seems to set the date correctly. I… --- ## ["\_dateparsefailure"](https://discuss.elastic.co/t/dateparsefailure/277700)
**Author:** [@RoteEdition](https://discuss.elastic.co/u/RoteEdition)\ **Replies:** 3\ **Last updated:** [July 3, 2021, 5:12pm UTC](https://discuss.elastic.co/t/dateparsefailure/277700 "2021-07-03T17:12:21Z")
Hi there, i am trying to parse a .CSV file with a pretty horrible date format. I am using the Date filter: date { match =\> \[ "HelperDate", "dMMyyyy", "ddMMyyyy" \] } Some example data: 6112019 -\> … --- ## [Logstash split log and insert it separately into elasticsearch](https://discuss.elastic.co/t/logstash-split-log-and-insert-it-separately-into-elasticsearch/277694)
**Author:** [@shifenglim](https://discuss.elastic.co/u/shifenglim)\ **Replies:** 1\ **Last updated:** [July 3, 2021, 4:18pm UTC](https://discuss.elastic.co/t/logstash-split-log-and-insert-it-separately-into-elasticsearch/277694 "2021-07-03T16:18:56Z")
I am writing a logstash config file and a log I am receiving is giving me issues, the team sent me multiple logs merged into one eg. message: \[logitem(aaa=1, bbb=1, ccc=1), logitem(aaa=2, bbb=2, ccc=2), logitem(aaa=3, b… --- ## [Vertica connection with ELK](https://discuss.elastic.co/t/vertica-connection-with-elk/277691)
**Author:** [@simran96](https://discuss.elastic.co/u/simran96)\ **Replies:** 0\ **Last updated:** [July 2, 2021, 10:33pm UTC](https://discuss.elastic.co/t/vertica-connection-with-elk/277691 "2021-07-02T22:33:49Z")
hi, im trying to connect Vertica with ELastic search and writing the conf file for the same. input { jdbc { jdbc\_validate\_connection =\> true … --- ## [Convert timestamp to ISO8601](https://discuss.elastic.co/t/convert-timestamp-to-iso8601/277528)
**Author:** [@mostpha456](https://discuss.elastic.co/u/mostpha456)\ **Replies:** 1\ **Last updated:** [July 2, 2021, 9:19pm UTC](https://discuss.elastic.co/t/convert-timestamp-to-iso8601/277528 "2021-07-02T21:19:48Z")
Hello, I am trying to convert the default field @timestamp to ISO8601 format, but i am having a lot of tags: \_grokparsefailure, \_dateparsefailure, ERROR\_ISO8601\_TIMESTAMP\_FIELD. Do you have any idea how to deal with t… --- ## [Tranformate my date to timestamp field](https://discuss.elastic.co/t/tranformate-my-date-to-timestamp-field/276413)
**Author:** [@Denilson-Semedo](https://discuss.elastic.co/u/Denilson-Semedo)\ **Replies:** 2\ **Last updated:** [July 2, 2021, 9:08pm UTC](https://discuss.elastic.co/t/tranformate-my-date-to-timestamp-field/276413 "2021-07-02T21:08:03Z")
Hy I have this json date. {"type":"audit\_entry","created":"5/20/2021, 11:12:42 PM","colaborador\_id":"cf7dc62b-dde9-4980-89d8-96eb5707876e","request\_method":"PUT","ajax":false,"route":"/stock/artigos/8c443bfe-d077-46d2-8… --- ## [How to change date format in logstash](https://discuss.elastic.co/t/how-to-change-date-format-in-logstash/277642)
**Author:** [@rknd](https://discuss.elastic.co/u/rknd)\ **Replies:** 1\ **Last updated:** [July 2, 2021, 8:01pm UTC](https://discuss.elastic.co/t/how-to-change-date-format-in-logstash/277642 "2021-07-02T20:01:04Z")
I have a logs like below; 04/12/2020 5:12:56 PM 05/12/2020 6:13:36 AM I want to match this fields and change to this format in below; 2020-12-04 17:12:56 2020-12-04 06:13:36 How can i do this in logstash ? Thanks fo… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=212) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=214)