# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=382 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 383 --- ## [Issue Transfer Data to ElasticSearch](https://discuss.elastic.co/t/issue-transfer-data-to-elasticsearch/225825)
**Author:** [@Bernard.Thoven](https://discuss.elastic.co/u/Bernard.Thoven)\ **Replies:** 4\ **Last updated:** [March 31, 2020, 11:09am UTC](https://discuss.elastic.co/t/issue-transfer-data-to-elasticsearch/225825 "2020-03-31T11:09:29Z")
Hi, I'm new using ELK and i'm trying to connect logstash to oracle DB. I try to run logstash.conf and create the index, but not all the data got send to kibana. Any suggestion ? Regards, Bernard --- ## [Logstash filter with regex](https://discuss.elastic.co/t/logstash-filter-with-regex/225837)
**Author:** [@Robo](https://discuss.elastic.co/u/Robo)\ **Replies:** 0\ **Last updated:** [March 31, 2020, 11:05am UTC](https://discuss.elastic.co/t/logstash-filter-with-regex/225837 "2020-03-31T11:05:20Z")
We are using following filter condition to distinguish if the value of a field is a string or an object: if \[log\]\[outgoing\_message\]\[body\] =~ /.\*/ { mutate { rename =\> \["\[log\]\[outgoing\_message\]\[body\]", "\[lo… --- ## [Import CSV with different column names to same field](https://discuss.elastic.co/t/import-csv-with-different-column-names-to-same-field/223725)
**Author:** [@brunojpsantos](https://discuss.elastic.co/u/brunojpsantos)\ **Replies:** 8\ **Last updated:** [March 31, 2020, 11:02am UTC](https://discuss.elastic.co/t/import-csv-with-different-column-names-to-same-field/223725 "2020-03-31T11:02:19Z")
Hi guys, I have 1 index in elasticsearch that I want to aggregate 2 CSV files data. So, I create the index and the mapping: index: test mapping with the fields: ID, Date, Name, Reference, Subject, Operator I have 2 C… --- ## [How can I check if the log file path contains a certain string in the logstash configuration using regex?](https://discuss.elastic.co/t/how-can-i-check-if-the-log-file-path-contains-a-certain-string-in-the-logstash-configuration-using-regex/224042)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 3\ **Last updated:** [March 31, 2020, 10:10am UTC](https://discuss.elastic.co/t/how-can-i-check-if-the-log-file-path-contains-a-certain-string-in-the-logstash-configuration-using-regex/224042 "2020-03-31T10:10:37Z")
I use filebeat to send logs to logstash, based on their file names - these logs are sent to specific indexes in elasticsearch. Filebeat works fine, logstash receives log files, but I can't get the regular expression in m… --- ## [Logstash and permiscious mode - no data?](https://discuss.elastic.co/t/logstash-and-permiscious-mode-no-data/225831)
**Author:** [@Rysiu](https://discuss.elastic.co/u/Rysiu)\ **Replies:** 0\ **Last updated:** [March 31, 2020, 10:06am UTC](https://discuss.elastic.co/t/logstash-and-permiscious-mode-no-data/225831 "2020-03-31T10:06:45Z")
Hi, I have a problem. The problem is partly described on: Using logstash to listen on mirrored ports (similar to Beats) · Issue #9948 · elastic/logstash · GitHub Logstash does not want to read the data sent in permisc… --- ## [Can Logstash output a read JSON-line as an event with multiple fields?](https://discuss.elastic.co/t/can-logstash-output-a-read-json-line-as-an-event-with-multiple-fields/225803)
**Author:** [@Mattness](https://discuss.elastic.co/u/Mattness)\ **Replies:** 2\ **Last updated:** [March 31, 2020, 9:10am UTC](https://discuss.elastic.co/t/can-logstash-output-a-read-json-line-as-an-event-with-multiple-fields/225803 "2020-03-31T09:10:27Z")
Hello, so I am trying to feed logdata to Elasticsearch. At first I tried Filebeat but that method did just read 1 line of JSON and converted it into one field called "message". For that reason I switched to Logstash on… --- ## [Can i have in logstash 2 different inputs with 2 different ports](https://discuss.elastic.co/t/can-i-have-in-logstash-2-different-inputs-with-2-different-ports/225809)
**Author:** [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Replies:** 4\ **Last updated:** [March 31, 2020, 9:07am UTC](https://discuss.elastic.co/t/can-i-have-in-logstash-2-different-inputs-with-2-different-ports/225809 "2020-03-31T09:07:51Z")
Hello can i have in logstash input section 2 different inputs with 2 different ports ? input { gelf { port\_udp =\> 5041 use\_udp =\> true }, http{ port\_http =\> 5043 } } In the case that the answer is yes can i al… --- ## [Logstash Beat Plugin fatal error](https://discuss.elastic.co/t/logstash-beat-plugin-fatal-error/225818)
**Author:** [@zebulyon](https://discuss.elastic.co/u/zebulyon)\ **Replies:** 0\ **Last updated:** [March 31, 2020, 8:45am UTC](https://discuss.elastic.co/t/logstash-beat-plugin-fatal-error/225818 "2020-03-31T08:45:35Z")
Hello, I'm using a docker compose configuration to run the ELK-stack and it has been working smoothly until recently and I have no Idea whats causing the error. The problem comes in the form that when I'm trying to star… --- ## [Logstash is not outputting read files](https://discuss.elastic.co/t/logstash-is-not-outputting-read-files/225636)
**Author:** [@Mattness](https://discuss.elastic.co/u/Mattness)\ **Replies:** 5\ **Last updated:** [March 31, 2020, 8:04am UTC](https://discuss.elastic.co/t/logstash-is-not-outputting-read-files/225636 "2020-03-31T08:04:28Z")
Hello, I want to use logstash to index some logfiles of mine. My config, with different paths I have tried: input { file { id =\> "my\_files" #path =\> \[ "C:\\Users\\Me\\Desktop\\files\\\*\*\\\*.log" \] #path =\> \[ "C:/User… --- ## [How to drop a nested field that has an empty value?](https://discuss.elastic.co/t/how-to-drop-a-nested-field-that-has-an-empty-value/225649)
**Author:** [@Robo](https://discuss.elastic.co/u/Robo)\ **Replies:** 2\ **Last updated:** [March 31, 2020, 7:57am UTC](https://discuss.elastic.co/t/how-to-drop-a-nested-field-that-has-an-empty-value/225649 "2020-03-31T07:57:11Z")
it's possible to drop a nested field in case the value is empty? example: "log":{"business\_information":""} background: this is a workaround to avoid mapping issues, as other application logs are creating fields like… --- ## [Ruby filter: escaping single quote inside the back quote expression](https://discuss.elastic.co/t/ruby-filter-escaping-single-quote-inside-the-back-quote-expression/225615)
**Author:** [@deostroll](https://discuss.elastic.co/u/deostroll)\ **Replies:** 1\ **Last updated:** [March 31, 2020, 6:24am UTC](https://discuss.elastic.co/t/ruby-filter-escaping-single-quote-inside-the-back-quote-expression/225615 "2020-03-31T06:24:30Z")
I am trying to run a powershell command to get the system date. In a windows command line you type this: powershell -command (date).ToString('yyyyMMddhhmmss') It outputs the current time in the required format. However… --- ## [SSL/TLS on Logstash syslog input plugin without XPACK/ NGINX or any third party tools](https://discuss.elastic.co/t/ssl-tls-on-logstash-syslog-input-plugin-without-xpack-nginx-or-any-third-party-tools/225151)
**Author:** [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Replies:** 1\ **Last updated:** [March 31, 2020, 5:37am UTC](https://discuss.elastic.co/t/ssl-tls-on-logstash-syslog-input-plugin-without-xpack-nginx-or-any-third-party-tools/225151 "2020-03-31T05:37:35Z")
Hi I want to secure the communication to logstash 7.4.2. Im using Syslog Input Plugin listening over specified port. How can I enable SSL/TLS on syslog? Regards Ram --- ## [Logstash muitiple pipeline to pipeline configue](https://discuss.elastic.co/t/logstash-muitiple-pipeline-to-pipeline-configue/225777)
**Author:** [@judragon\_dark](https://discuss.elastic.co/u/judragon_dark)\ **Replies:** 1\ **Last updated:** [March 31, 2020, 5:22am UTC](https://discuss.elastic.co/t/logstash-muitiple-pipeline-to-pipeline-configue/225777 "2020-03-31T05:22:49Z")
I'd like to do a multi-pipeline. - pipeline.id: beats-server input { beats { port =\> 5044 } } output { if \[type\] == log1 { pipeline { send\_to =\> log1 } } else if \[type\] == log2 { pipeline { send\_t… --- ## [Performance Tuning for related docs](https://discuss.elastic.co/t/performance-tuning-for-related-docs/225774)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 0\ **Last updated:** [March 31, 2020, 2:55am UTC](https://discuss.elastic.co/t/performance-tuning-for-related-docs/225774 "2020-03-31T02:55:20Z")
Hey, it's me again =D so, in my quest to crush those damn RRAS logs I've found myself in a pickle. I had read somewhere that if I wanted my registries to be read in a linear stream I had to setup the pipeline to use a si… --- ## [How to completely disable Xpack?](https://discuss.elastic.co/t/how-to-completely-disable-xpack/225755)
**Author:** [@Alexander\_Popov](https://discuss.elastic.co/u/Alexander_Popov)\ **Replies:** 3\ **Last updated:** [March 30, 2020, 11:47pm UTC](https://discuss.elastic.co/t/how-to-completely-disable-xpack/225755 "2020-03-30T23:47:19Z")
I have elasticsearch 7.4 running on AWS Elastic Search with Open Distro user management with curl and basic auth - I can successful call elasticsearch root: { "name": "0ea36812a52dca373179a5cbe60cfa8d", "cluster\_… --- ## [JDBC MONGO and objectId type](https://discuss.elastic.co/t/jdbc-mongo-and-objectid-type/225760)
**Author:** [@7cinquante](https://discuss.elastic.co/u/7cinquante)\ **Replies:** 0\ **Last updated:** [March 30, 2020, 10:46pm UTC](https://discuss.elastic.co/t/jdbc-mongo-and-objectid-type/225760 "2020-03-30T22:46:59Z")
For people who are looking for convert objectId type from mongo to string in logstash statement =\> "db.getCollection('YOUR\_COLLECTION').aggregate(\[ { $project: { '\_id': {$toString: '$\_id'}, … --- ## [Centralized Pipeline Management Error](https://discuss.elastic.co/t/centralized-pipeline-management-error/225599)
**Author:** [@earl\_potter](https://discuss.elastic.co/u/earl_potter)\ **Replies:** 4\ **Last updated:** [March 30, 2020, 5:58pm UTC](https://discuss.elastic.co/t/centralized-pipeline-management-error/225599 "2020-03-30T17:58:04Z")
Trying to get Centralized Pipeline Management working. I have a small deployment on Elastic Cloud so I know security is enabled. But when I try to run logstash, I get the following error: \`Setting "xpack.management.en… --- ## [Java Heap Space](https://discuss.elastic.co/t/java-heap-space/225496)
**Author:** [@hariskhalique](https://discuss.elastic.co/u/hariskhalique)\ **Replies:** 2\ **Last updated:** [March 30, 2020, 3:46pm UTC](https://discuss.elastic.co/t/java-heap-space/225496 "2020-03-30T15:46:08Z")
Hi, I am getting OutOfMemoryError following are the JVM sizes \> intx CompilerThreadStackSize = 0 {pd product} \> uintx ErgoHeapSizeLimit =… --- ## [Logstash Json Split Data is NOT loading correctly and gives Http Status 409 at logstash log file](https://discuss.elastic.co/t/logstash-json-split-data-is-not-loading-correctly-and-gives-http-status-409-at-logstash-log-file/225700)
**Author:** [@Ranga\_Bhupal\_G](https://discuss.elastic.co/u/Ranga_Bhupal_G)\ **Replies:** 2\ **Last updated:** [March 30, 2020, 3:34pm UTC](https://discuss.elastic.co/t/logstash-json-split-data-is-not-loading-correctly-and-gives-http-status-409-at-logstash-log-file/225700 "2020-03-30T15:34:48Z")
Hi, We have a table that contains JSON data as text/string. Logstash has below configuration that uses JDBC, JSON, SPLIT, HTTP and ELASTICSEARCH plugins. Logstash error contains Http Status 409 and Data is NOT loading… --- ## [Getting \_dateparsefailure error](https://discuss.elastic.co/t/getting-dateparsefailure-error/225613)
**Author:** [@niudaye123](https://discuss.elastic.co/u/niudaye123)\ **Replies:** 1\ **Last updated:** [March 30, 2020, 3:13pm UTC](https://discuss.elastic.co/t/getting-dateparsefailure-error/225613 "2020-03-30T15:13:01Z")
Hi, I am getting \_dateparsefailure in tags, Not sure why it is. I checked the date match format and did not see any issue though, Any idea what's the issue? thanks! Here is the original message: Mar 27 16:48:18.875 PD… --- ## [Logstash, split field and remove field](https://discuss.elastic.co/t/logstash-split-field-and-remove-field/225410)
**Author:** [@SBH](https://discuss.elastic.co/u/SBH)\ **Replies:** 3\ **Last updated:** [March 30, 2020, 3:09pm UTC](https://discuss.elastic.co/t/logstash-split-field-and-remove-field/225410 "2020-03-30T15:09:10Z")
Hi, working on my logstash conf file. I have a remove field like this: remove\_field =\> \["\[data\]\[tasks\]\[data\]"\] And then i have a split field like this: split {field =\> "\[data\]\[tasks\]"} But when i see it in Discover… --- ## [Fingerprint- SHA256 takes a lot more space then MURMUR3](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574)
**Author:** [@aurimas](https://discuss.elastic.co/u/aurimas)\ **Replies:** 4\ **Last updated:** [March 30, 2020, 2:00pm UTC](https://discuss.elastic.co/t/fingerprint-sha256-takes-a-lot-more-space-then-murmur3/225574 "2020-03-30T14:00:34Z")
Hi, I have problem with duplicate documents, so I am using method with Logstash described here: It seems to do the job as after dedublication I get smaller number of documents. But then I get another problem, then … --- ## [Read timed out .. Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down!](https://discuss.elastic.co/t/read-timed-out-attempted-to-send-a-bulk-request-to-elasticsearch-but-elasticsearch-appears-to-be-unreachable-or-down/225680)
**Author:** [@jl2035](https://discuss.elastic.co/u/jl2035)\ **Replies:** 0\ **Last updated:** [March 30, 2020, 1:18pm UTC](https://discuss.elastic.co/t/read-timed-out-attempted-to-send-a-bulk-request-to-elasticsearch-but-elasticsearch-appears-to-be-unreachable-or-down/225680 "2020-03-30T13:18:41Z")
Hi, I've found thousands of forum post on this topic, but not a single solution. This is my desperate attempt to solve my problem creating my own thread. Any help would be awesome, this is really difficult to setup. I'm… --- ## [Move all fields to a subfield?](https://discuss.elastic.co/t/move-all-fields-to-a-subfield/225563)
**Author:** [@sliddjur](https://discuss.elastic.co/u/sliddjur)\ **Replies:** 2\ **Last updated:** [March 30, 2020, 10:29am UTC](https://discuss.elastic.co/t/move-all-fields-to-a-subfield/225563 "2020-03-30T10:29:00Z")
I want to move all my fields to a specified subfield. However, I don't know all the field names. Alternatively if there is a way for grok filter to set the target field or similar? I found an old post that did this, wh… --- ## [Logstash Unexpected character ('-' (code 45)): was expecting comma to separate Array entries](https://discuss.elastic.co/t/logstash-unexpected-character-code-45-was-expecting-comma-to-separate-array-entries/224821)
**Author:** [@Rahul\_Ravichandran](https://discuss.elastic.co/u/Rahul_Ravichandran)\ **Replies:** 5\ **Last updated:** [March 30, 2020, 8:05am UTC](https://discuss.elastic.co/t/logstash-unexpected-character-code-45-was-expecting-comma-to-separate-array-entries/224821 "2020-03-30T08:05:51Z")
Hi i am trying to move my application logs to kibana through logstash. 1.This is my logstash.conf file: input { file { path =\> "/home/ubuntu/\*" } } filter { json { source =\> "message" } … --- ## [How to make logstash parse file in a parallel manner](https://discuss.elastic.co/t/how-to-make-logstash-parse-file-in-a-parallel-manner/225568)
**Author:** [@Mohsin\_Alam](https://discuss.elastic.co/u/Mohsin_Alam)\ **Replies:** 2\ **Last updated:** [March 30, 2020, 7:33am UTC](https://discuss.elastic.co/t/how-to-make-logstash-parse-file-in-a-parallel-manner/225568 "2020-03-30T07:33:24Z")
hey i am using 3 vms and on 2 of my vm i am running instance of logstash and what i notice is when i push logs to both the vm's at the same time logs from vm1 are first processed by logstash and then vm2 logs are process… --- ## [If grok\_parse failure Instead of drop. I want to process everything as greedy data](https://discuss.elastic.co/t/if-grok-parse-failure-instead-of-drop-i-want-to-process-everything-as-greedy-data/225585)
**Author:** [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Replies:** 3\ **Last updated:** [March 30, 2020, 7:33am UTC](https://discuss.elastic.co/t/if-grok-parse-failure-instead-of-drop-i-want-to-process-everything-as-greedy-data/225585 "2020-03-30T07:33:20Z")
filter { if \[type\] == "chef-client" { grok { match =\> {"message" =\> "\\\[%{TIMESTAMP\_ISO8601:logtimestamp}\\\] %{SPACE}%{LOGLEVEL:loglevel}:%{SPACE}%{GREEDYDATA:message}"} … --- ## [Issue while parsing xml data from a file](https://discuss.elastic.co/t/issue-while-parsing-xml-data-from-a-file/225132)
**Author:** [@Varun\_Saraf](https://discuss.elastic.co/u/Varun_Saraf)\ **Replies:** 4\ **Last updated:** [March 30, 2020, 6:06am UTC](https://discuss.elastic.co/t/issue-while-parsing-xml-data-from-a-file/225132 "2020-03-30T06:06:49Z")
I have xml data in a file and I am trying to parse it as a single event and visualise it in kibana below is the Configuration file for logstash. input { file { path =\> "/Users/varsaraf/outxr.xml" start\_position =\> "… --- ## [Is there a long-term road map for Helm 3 support?](https://discuss.elastic.co/t/is-there-a-long-term-road-map-for-helm-3-support/225438)
**Author:** [@aidan-melen](https://discuss.elastic.co/u/aidan-melen)\ **Replies:** 1\ **Last updated:** [March 30, 2020, 5:09am UTC](https://discuss.elastic.co/t/is-there-a-long-term-road-map-for-helm-3-support/225438 "2020-03-30T05:09:27Z")
In response to this article: https://www.elastic.co/blog/alpha-helm-charts-for-elasticsearch-kibana-and-cncf-membership Is there a long-term road map for Helm 3 support? The official Elasticsearch and Kibana charts work… --- ## [Mutate --\> convert doesn't work](https://discuss.elastic.co/t/mutate-convert-doesnt-work/225551)
**Author:** [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)\ **Replies:** 3\ **Last updated:** [March 30, 2020, 4:31am UTC](https://discuss.elastic.co/t/mutate-convert-doesnt-work/225551 "2020-03-30T04:31:38Z")
Hi Team, I'm trying to change a field data type from string to integer but it doesn't work. see below example: In the above example, neededFieldinInt and number\_of\_documents I need in integer and below is the code … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=381) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=383)