# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=537 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 538 --- ## [Enrich port data with a new description field](https://discuss.elastic.co/t/enrich-port-data-with-a-new-description-field/183645)
**Author:** [@Duked](https://discuss.elastic.co/u/Duked)\ **Replies:** 2\ **Last updated:** [May 31, 2019, 12:52pm UTC](https://discuss.elastic.co/t/enrich-port-data-with-a-new-description-field/183645 "2019-05-31T12:52:45Z")
Hi all, I'm parsing logs that contains src/dst port information. Now for every port I want to enrich the data by adding a description field. To make it clear: Port 22 -\> description = "SSH Traffic" Port 80 -\> descript… --- ## [Filebeat dashboard with logstash](https://discuss.elastic.co/t/filebeat-dashboard-with-logstash/183716)
**Author:** [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Replies:** 2\ **Last updated:** [May 31, 2019, 12:30pm UTC](https://discuss.elastic.co/t/filebeat-dashboard-with-logstash/183716 "2019-05-31T12:30:27Z")
Hello Team, My architecture is Filebeat-\>Logstash-\>Elasticsearch-\>Kibana. I am using logstash so i can't use filebeat module. Because filebeat module send data directly to elasticsearch. So i want some extra parsing w… --- ## [Extract server name used by FileBeats to connect to LogStash](https://discuss.elastic.co/t/extract-server-name-used-by-filebeats-to-connect-to-logstash/183495)
**Author:** [@funduraghu](https://discuss.elastic.co/u/funduraghu)\ **Replies:** 3\ **Last updated:** [May 31, 2019, 11:45am UTC](https://discuss.elastic.co/t/extract-server-name-used-by-filebeats-to-connect-to-logstash/183495 "2019-05-31T11:45:14Z")
I've set up multiple FileBeats instances on different machines to send syslog to a single central LogStash server. I have created ssl certs and have setup SSL communication (as explained in documentation) between FileBe… --- ## [How to get the one document's field and update with another document's fields\[transfer the field's value from one event to another event\]](https://discuss.elastic.co/t/how-to-get-the-one-documents-field-and-update-with-another-documents-fields-transfer-the-fields-value-from-one-event-to-another-event/183498)
**Author:** [@Md\_Iburahimsha\_S](https://discuss.elastic.co/u/Md_Iburahimsha_S)\ **Replies:** 3\ **Last updated:** [May 31, 2019, 11:09am UTC](https://discuss.elastic.co/t/how-to-get-the-one-documents-field-and-update-with-another-documents-fields-transfer-the-fields-value-from-one-event-to-another-event/183498 "2019-05-31T11:09:38Z")
I have a requirement to update the one document field based on the another document example \_id : 5 doesn't has any field like "risk", but another \_id : 4 has a field risk =\> high . \_id=\> 4 is the 1st event and followed… --- ## [Do we have solr input plugin for logstash?](https://discuss.elastic.co/t/do-we-have-solr-input-plugin-for-logstash/183540)
**Author:** [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Replies:** 1\ **Last updated:** [May 31, 2019, 9:29am UTC](https://discuss.elastic.co/t/do-we-have-solr-input-plugin-for-logstash/183540 "2019-05-31T09:29:50Z")
I have to query some records from solr and push to kafka using logstash. logstash have input plugin for elastic search but I couldn't find one for solr. is there a way/plugin to connect with solr from logstash? thanks… --- ## [Logstash-output-google\_pubsub giving \`pubsub.googleapis.com: Name or service not known\` error](https://discuss.elastic.co/t/logstash-output-google-pubsub-giving-pubsub-googleapis-com-name-or-service-not-known-error/183693)
**Author:** [@Anurag\_Mishra1](https://discuss.elastic.co/u/Anurag_Mishra1)\ **Replies:** 0\ **Last updated:** [May 31, 2019, 9:28am UTC](https://discuss.elastic.co/t/logstash-output-google-pubsub-giving-pubsub-googleapis-com-name-or-service-not-known-error/183693 "2019-05-31T09:28:53Z")
I using logstatsh output google pubsub plugin to consume my GCP pub sub messages. I have successfully installed elasticsearch, kibana, logstatsh and output plugin. I am following the tutorial on this page. After generat… --- ## [Issue with getting events over TCP output to TCP input in logstash](https://discuss.elastic.co/t/issue-with-getting-events-over-tcp-output-to-tcp-input-in-logstash/183506)
**Author:** [@darshan](https://discuss.elastic.co/u/darshan)\ **Replies:** 1\ **Last updated:** [May 31, 2019, 9:27am UTC](https://discuss.elastic.co/t/issue-with-getting-events-over-tcp-output-to-tcp-input-in-logstash/183506 "2019-05-31T09:27:08Z")
Hi i am try to sending events over tcp output to tcp input in logstash but i am not getting event in input side. When i run in debug(input side logstash) mode its showing some messages like TCP output configuration out… --- ## [Logstash output to FTP](https://discuss.elastic.co/t/logstash-output-to-ftp/183680)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 1\ **Last updated:** [May 31, 2019, 9:21am UTC](https://discuss.elastic.co/t/logstash-output-to-ftp/183680 "2019-05-31T09:21:21Z")
Hello, We have a use case, where our admins would like to receive Logstash's json output files directly to their FTP using non-standard ftp port. I tried using file output plugin with below path, but it fails. ftp://l… --- ## [Logstash ingesting gzip json file using http\_poller](https://discuss.elastic.co/t/logstash-ingesting-gzip-json-file-using-http-poller/183682)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 0\ **Last updated:** [May 31, 2019, 8:33am UTC](https://discuss.elastic.co/t/logstash-ingesting-gzip-json-file-using-http-poller/183682 "2019-05-31T08:33:01Z")
Hello. I want to do the direct ingestion of the json.gzip file over http\_poller. input { http\_poller { urls =\> { url\_1 =\> "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2019.json.gz" } request\_… --- ## [Variable in path](https://discuss.elastic.co/t/variable-in-path/183307)
**Author:** [@Sam\_Kisada](https://discuss.elastic.co/u/Sam_Kisada)\ **Replies:** 3\ **Last updated:** [May 31, 2019, 8:24am UTC](https://discuss.elastic.co/t/variable-in-path/183307 "2019-05-31T08:24:16Z")
Hi! I need some help with declaring variable in dicrianary\_path I have few filebeats which sending data to logstash from few different sources. Data is separeted to dictianary and raw, there is no chance to change it b… --- ## [Logstash Filter creating Duplicate documents](https://discuss.elastic.co/t/logstash-filter-creating-duplicate-documents/183584)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 9\ **Last updated:** [May 31, 2019, 7:28am UTC](https://discuss.elastic.co/t/logstash-filter-creating-duplicate-documents/183584 "2019-05-31T07:28:26Z")
I have below logstash filter which i am using to filter the raw data provided below, i works while tested with debugger but the problem is its creating duplicate documents for the same entry ie for the data sample pro… --- ## [Extract Common Name from incoming ssl connection and add field in LogStash](https://discuss.elastic.co/t/extract-common-name-from-incoming-ssl-connection-and-add-field-in-logstash/183670)
**Author:** [@funduraghu](https://discuss.elastic.co/u/funduraghu)\ **Replies:** 0\ **Last updated:** [May 31, 2019, 7:11am UTC](https://discuss.elastic.co/t/extract-common-name-from-incoming-ssl-connection-and-add-field-in-logstash/183670 "2019-05-31T07:11:49Z")
I am using ssl connections from client (FileBeat/syslog-ng/other) to send logs to LogStash. I would like to extract the Common Name from the ssl connection to identify the client. How can I extract the required informa… --- ## [Logstash is closing connections and Clients get Connection closed errors, especially at low volumes](https://discuss.elastic.co/t/logstash-is-closing-connections-and-clients-get-connection-closed-errors-especially-at-low-volumes/183668)
**Author:** [@ramsateesh](https://discuss.elastic.co/u/ramsateesh)\ **Replies:** 1\ **Last updated:** [May 31, 2019, 6:43am UTC](https://discuss.elastic.co/t/logstash-is-closing-connections-and-clients-get-connection-closed-errors-especially-at-low-volumes/183668 "2019-05-31T06:43:53Z")
Hi I am running a http\_input plugin and kafka output plugin. At high volumes like around 300 tps everything works just fine... but in lower tps logstash is closing client connections and they end up with an error that… --- ## [Logstash configuration always run sql mode](https://discuss.elastic.co/t/logstash-configuration-always-run-sql-mode/183210)
**Author:** [@akyong](https://discuss.elastic.co/u/akyong)\ **Replies:** 1\ **Last updated:** [May 31, 2019, 3:04am UTC](https://discuss.elastic.co/t/logstash-configuration-always-run-sql-mode/183210 "2019-05-31T03:04:15Z")
Hi folks, I just found out that my logtash always run query set sql\_mode='no\_engine\_substitution,strict\_trans\_tables' every 10s. i see that user api\_elastics always run that query every 10s in mysql log history. thi… --- ## [Help with Logstash security features](https://discuss.elastic.co/t/help-with-logstash-security-features/183597)
**Author:** [@Mrcow](https://discuss.elastic.co/u/Mrcow)\ **Replies:** 11\ **Last updated:** [May 31, 2019, 12:41am UTC](https://discuss.elastic.co/t/help-with-logstash-security-features/183597 "2019-05-31T00:41:42Z")
I just enabled on my elasticsearch 7.1 the security to login with user and passwords, but logstash stop working, im getting this error \[2019-05-30T12:58:41,053\]\[ERROR\]\[logstash.outputs.elasticsearch\] Encountered a retry… --- ## [Logstash has duplicate timestamp field](https://discuss.elastic.co/t/logstash-has-duplicate-timestamp-field/183557)
**Author:** [@ramzey1981](https://discuss.elastic.co/u/ramzey1981)\ **Replies:** 4\ **Last updated:** [May 31, 2019, 12:04am UTC](https://discuss.elastic.co/t/logstash-has-duplicate-timestamp-field/183557 "2019-05-31T00:04:44Z")
I have some cassandra logs that use this custom pattern for logging MILLISECOND (\\d{3}) JAVALOGBACKTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}\[T \]%{HOUR}:%{MINUTE}:%{SECOND},%{MILLISECOND} CASS\_BASE ^%{LOGLEVEL:level} \\\[… --- ## [Aggregation help](https://discuss.elastic.co/t/aggregation-help/183622)
**Author:** [@grazer](https://discuss.elastic.co/u/grazer)\ **Replies:** 0\ **Last updated:** [May 30, 2019, 10:24pm UTC](https://discuss.elastic.co/t/aggregation-help/183622 "2019-05-30T22:24:11Z")
the input file has so many date stamps, I want to aggregate on month or year. when I create the index and I start to define it I only get the @timestamp option which is the date the csv file was imported I want to be ab… --- ## [Delete lines in Elasticsearch previous deleted in SQL DB](https://discuss.elastic.co/t/delete-lines-in-elasticsearch-previous-deleted-in-sql-db/183281)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 3\ **Last updated:** [May 30, 2019, 7:54pm UTC](https://discuss.elastic.co/t/delete-lines-in-elasticsearch-previous-deleted-in-sql-db/183281 "2019-05-30T19:54:23Z")
Hello, In my logstash file, I input my data from a mysql database and then do some transformations and insert in elasticsearch. I used the action update to always have the most updated information. However, if I delete … --- ## [Logstash delete postgresql module fields](https://discuss.elastic.co/t/logstash-delete-postgresql-module-fields/183598)
**Author:** [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Replies:** 0\ **Last updated:** [May 30, 2019, 7:07pm UTC](https://discuss.elastic.co/t/logstash-delete-postgresql-module-fields/183598 "2019-05-30T19:07:48Z")
Hi I have an ELK platform version 7.1.0, to fix a timestamp inconsistency between my servers and Kibana I created a logstash filter, this filter read the timestamp from the logs and applied it in the filed @timestamp … --- ## [How to remove a line break from a field in logstash](https://discuss.elastic.co/t/how-to-remove-a-line-break-from-a-field-in-logstash/183572)
**Author:** [@Carlos.limaElaw](https://discuss.elastic.co/u/Carlos.limaElaw)\ **Replies:** 1\ **Last updated:** [May 30, 2019, 4:18pm UTC](https://discuss.elastic.co/t/how-to-remove-a-line-break-from-a-field-in-logstash/183572 "2019-05-30T16:18:54Z")
Hello, everyone! I'm trying to ingest a csv like this in logstash using csv filter: column1;column2;column3;column4;column5; 201905; DADOS PARA O CADASTRO;2019-05-22 09:48:08;2019-05-22 10:06:05;"Criar acesso para … --- ## [Receiving SNMP V3 traps](https://discuss.elastic.co/t/receiving-snmp-v3-traps/183585)
**Author:** [@romain1](https://discuss.elastic.co/u/romain1)\ **Replies:** 0\ **Last updated:** [May 30, 2019, 6:09pm UTC](https://discuss.elastic.co/t/receiving-snmp-v3-traps/183585 "2019-05-30T18:09:28Z")
Hello, Can the snmptrap input plugin for logstash receive snmp version 3 traps ? If not, are there other solutions ? I would like to collect snmp v3 traps sent by network devices. Thanks you --- ## [Special characters - replace](https://discuss.elastic.co/t/special-characters-replace/183156)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 2\ **Last updated:** [May 30, 2019, 5:05pm UTC](https://discuss.elastic.co/t/special-characters-replace/183156 "2019-05-30T17:05:27Z")
Hello, I have special characters in my fields like ã, á, à and others (in portuguese language we use them). There is a any to replace them? I tried to used ruby gsub like this "event.set('subProcessName', event.get('sub… --- ## [Inconsistent date matching pattern: "M/d/yyyy h:mm:ss a"](https://discuss.elastic.co/t/inconsistent-date-matching-pattern-m-d-yyyy-hss-a/183394)
**Author:** [@rlwmmw](https://discuss.elastic.co/u/rlwmmw)\ **Replies:** 2\ **Last updated:** [May 30, 2019, 4:59pm UTC](https://discuss.elastic.co/t/inconsistent-date-matching-pattern-m-d-yyyy-hss-a/183394 "2019-05-30T16:59:23Z")
Scratching my head with this one... Note: It does not appear to be the space issue that sometimes plagues single-digit fields. Any ideas? Ubuntu 16.04.6 LTS logstash 7.1.0 openjdk version "1.8.0\_212-ojdkbuild" Ope… --- ## [Get filter value in add\_field](https://discuss.elastic.co/t/get-filter-value-in-add-field/182946)
**Author:** [@kentatoi](https://discuss.elastic.co/u/kentatoi)\ **Replies:** 4\ **Last updated:** [May 30, 2019, 4:32pm UTC](https://discuss.elastic.co/t/get-filter-value-in-add-field/182946 "2019-05-30T16:32:52Z")
Hi, I would like to get the path of my logs from Filebeat split it and get the name of my app\_serv as a new field. I am using ELK-stack 6.6.0 Here that is what I am trying : input { stdin { } } filter { grok { … --- ## [Logstash 7.1.0 doesn't stop even if using --pipeline.unsafe\_shutdown](https://discuss.elastic.co/t/logstash-7-1-0-doesnt-stop-even-if-using-pipeline-unsafe-shutdown/183546)
**Author:** [@Ged](https://discuss.elastic.co/u/Ged)\ **Replies:** 0\ **Last updated:** [May 30, 2019, 2:10pm UTC](https://discuss.elastic.co/t/logstash-7-1-0-doesnt-stop-even-if-using-pipeline-unsafe-shutdown/183546 "2019-05-30T14:10:16Z")
Hi guys, I've just tested Logstash 7.1.0 against my previous version 6.3.1. I've noticed when trying to stop 7.1.0 version with SIGTERM (using --pipeline.unsafe\_shutdown option ) it cannot stop. I see: \[2019-05-30T09:… --- ## [Regex to match a specific key value pair in Logstash](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167)
**Author:** [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Replies:** 8\ **Last updated:** [May 30, 2019, 12:56pm UTC](https://discuss.elastic.co/t/regex-to-match-a-specific-key-value-pair-in-logstash/183167 "2019-05-30T12:56:13Z")
I'm looking to remove specific key=value pair that are inside a STRING. Say, input event is as follows: { "ABC": "10119707", "Request\_StartTime": "1558952196175", "Severity": "INFO", "UUID": "481e8cfa-399c-4996-a4d… --- ## [One or more required cgroup files or directories not found: /proc/self/cgroup](https://discuss.elastic.co/t/one-or-more-required-cgroup-files-or-directories-not-found-proc-self-cgroup/183510)
**Author:** [@Aizaz\_Syed](https://discuss.elastic.co/u/Aizaz_Syed)\ **Replies:** 1\ **Last updated:** [May 30, 2019, 12:34pm UTC](https://discuss.elastic.co/t/one-or-more-required-cgroup-files-or-directories-not-found-proc-self-cgroup/183510 "2019-05-30T12:34:24Z")
I have configured the logstash configuration as below to read the logs generated by spring boot to push into elastic search, but index is not creating even the logstash started successfully, Please let me know how to fi… --- ## [How to delete records from elastic search which are removed from DB using LOGSTASH JDBC PLUGIN?](https://discuss.elastic.co/t/how-to-delete-records-from-elastic-search-which-are-removed-from-db-using-logstash-jdbc-plugin/183522)
**Author:** [@Soumyapriya\_Sahoo](https://discuss.elastic.co/u/Soumyapriya_Sahoo)\ **Replies:** 0\ **Last updated:** [May 30, 2019, 12:27pm UTC](https://discuss.elastic.co/t/how-to-delete-records-from-elastic-search-which-are-removed-from-db-using-logstash-jdbc-plugin/183522 "2019-05-30T12:27:32Z")
How to delete records from elastic search which are removed from DB using LOGSTASH JDBC PLUGIN? --- ## [Ingesting a csv file with logstash](https://discuss.elastic.co/t/ingesting-a-csv-file-with-logstash/183076)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 12\ **Last updated:** [May 30, 2019, 11:58am UTC](https://discuss.elastic.co/t/ingesting-a-csv-file-with-logstash/183076 "2019-05-30T11:58:39Z")
Hello, I am trying to ingest a csv file with logstash. Can someone check my logstash.conf file. I want the content to be stored in elasticsearch as well as to be printed on stdout. input { file{ path =\> \["/home/abhi… --- ## [Unable to safely shutdown Logstash](https://discuss.elastic.co/t/unable-to-safely-shutdown-logstash/183500)
**Author:** [@agiorgi](https://discuss.elastic.co/u/agiorgi)\ **Replies:** 0\ **Last updated:** [May 30, 2019, 10:02am UTC](https://discuss.elastic.co/t/unable-to-safely-shutdown-logstash/183500 "2019-05-30T10:02:30Z")
Hi there, I have 3 Pipelines running, one is a syslog listener, the other 2 are JDBC inputs. As per title says, i can't safely shutdown logstash as it hangs while trying to shutdown itself. Both if i try to restart th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=536) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=538)