# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=498 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 499 --- ## [Logstash Grok Pattern-Problem with finding every ")"](https://discuss.elastic.co/t/logstash-grok-pattern-problem-with-finding-every/192034)
**Author:** [@Dennis\_Bohs](https://discuss.elastic.co/u/Dennis_Bohs)\ **Replies:** 5\ **Last updated:** [August 6, 2019, 7:57am UTC](https://discuss.elastic.co/t/logstash-grok-pattern-problem-with-finding-every/192034 "2019-08-06T07:57:53Z")
Hi there and thanks for your time in advance. I did build my first grok-pattern myself to differ some fields in my events. The filter looks like this: filter { grok { match =\> { "message" =\> "\\A%{TIMEST… --- ## [How many pipelines can at max be configured in pipelines.yml](https://discuss.elastic.co/t/how-many-pipelines-can-at-max-be-configured-in-pipelines-yml/193920)
**Author:** [@bharatwaj](https://discuss.elastic.co/u/bharatwaj)\ **Replies:** 1\ **Last updated:** [August 6, 2019, 7:00am UTC](https://discuss.elastic.co/t/how-many-pipelines-can-at-max-be-configured-in-pipelines-yml/193920 "2019-08-06T07:00:26Z")
I wish to know the maximum number of pipelines that can be successfully configured in pipelines.yml --- ## [Logstash HTTP\_Poller tls version](https://discuss.elastic.co/t/logstash-http-poller-tls-version/193875)
**Author:** [@tn-mikel](https://discuss.elastic.co/u/tn-mikel)\ **Replies:** 4\ **Last updated:** [August 6, 2019, 3:09am UTC](https://discuss.elastic.co/t/logstash-http-poller-tls-version/193875 "2019-08-06T03:09:44Z")
Hello, I am trying to use the HTTP\_Poller v5.0.1 to contact the CloudFlare API to pull some traffic and security logs. However, when calling the URI, Cloudflare API complains that the version of TLS I am using is only 1… --- ## [\[Multiline\]Data following newline is ignored](https://discuss.elastic.co/t/multiline-data-following-newline-is-ignored/193733)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 4\ **Last updated:** [August 6, 2019, 1:33am UTC](https://discuss.elastic.co/t/multiline-data-following-newline-is-ignored/193733 "2019-08-06T01:33:34Z")
Hi, I have a csv file that contains a double quoted field with a empty line. I am trying to convert this to a single line using the multiline codec so I can filter it later on using the csv filter. csv example number… --- ## [Multiple codecs? running multiline codec during filter?](https://discuss.elastic.co/t/multiple-codecs-running-multiline-codec-during-filter/193894)
**Author:** [@tedder](https://discuss.elastic.co/u/tedder)\ **Replies:** 4\ **Last updated:** [August 6, 2019, 12:52am UTC](https://discuss.elastic.co/t/multiple-codecs-running-multiline-codec-during-filter/193894 "2019-08-06T00:52:12Z")
I have log entries coming in as a JSON payload. I'm using a json codec, then splitting the entries in a filter. My JSON looks something like this: payload = { "date": "...", "source": "...", "entries": \["line1", "line2"… --- ## [Grok - Parsing a String with Spaces](https://discuss.elastic.co/t/grok-parsing-a-string-with-spaces/193855)
**Author:** [@jakecramer17](https://discuss.elastic.co/u/jakecramer17)\ **Replies:** 5\ **Last updated:** [August 5, 2019, 9:34pm UTC](https://discuss.elastic.co/t/grok-parsing-a-string-with-spaces/193855 "2019-08-05T21:34:21Z")
For a while I've been trying to parse a syslog input in Logstash. I've encountered something like this: 2019-08-05 08:55:15 - jcramer(string with user information) - Successfully updated profile (this is an example mea… --- ## [Logstash pipelines and conditionals in config](https://discuss.elastic.co/t/logstash-pipelines-and-conditionals-in-config/193416)
**Author:** [@oliverlucas1](https://discuss.elastic.co/u/oliverlucas1)\ **Replies:** 6\ **Last updated:** [August 5, 2019, 9:00pm UTC](https://discuss.elastic.co/t/logstash-pipelines-and-conditionals-in-config/193416 "2019-08-05T21:00:29Z")
Hi All, Although the subject sounds interesting (maybe for some) but I don't really have a clue what I'm doing! I've been trying to search for an answer but no luck so I'm seeing if someone can help or point me to somet… --- ## [Location Mapping](https://discuss.elastic.co/t/location-mapping/193615)
**Author:** [@aaditya\_alekar](https://discuss.elastic.co/u/aaditya_alekar)\ **Replies:** 39\ **Last updated:** [August 5, 2019, 7:59pm UTC](https://discuss.elastic.co/t/location-mapping/193615 "2019-08-05T19:59:24Z")
I want solution for location mapping for my dataset. Following is the link for my dataset: I am not able to get location type geo\_point for mapping. Kindly advise. --- ## [Google Cloud Security Command Center (SCC) Logstash Plugin](https://discuss.elastic.co/t/google-cloud-security-command-center-scc-logstash-plugin/192774)
**Author:** [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\ **Replies:** 3\ **Last updated:** [August 5, 2019, 5:46pm UTC](https://discuss.elastic.co/t/google-cloud-security-command-center-scc-logstash-plugin/192774 "2019-08-05T17:46:04Z")
I looked into Logstash plugins today, and I noticed there were two plugins for Google cloud: google\_cloud\_storage and google\_pubsub. I was just thinking if there is a short term plan/project to come up with a plugin for … --- ## [Logstash JDBC Oracle](https://discuss.elastic.co/t/logstash-jdbc-oracle/193846)
**Author:** [@Adriano\_F](https://discuss.elastic.co/u/Adriano_F)\ **Replies:** 1\ **Last updated:** [August 5, 2019, 4:34pm UTC](https://discuss.elastic.co/t/logstash-jdbc-oracle/193846 "2019-08-05T16:34:32Z")
Boa tarde, estou tentando utilizar o logstash com JDBC acessando oracle input { jdbc { jdbc\_validate\_connection =\> true jdbc\_connection\_string =\> "jdbc:oracle:thin:@vms-dbs-01:1521/nladm" jdbc\_user =\> "nladm" jdbc… --- ## [Logstash - Checking if a value or field exist in an array within an object](https://discuss.elastic.co/t/logstash-checking-if-a-value-or-field-exist-in-an-array-within-an-object/192294)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 4\ **Last updated:** [August 5, 2019, 4:09pm UTC](https://discuss.elastic.co/t/logstash-checking-if-a-value-or-field-exist-in-an-array-within-an-object/192294 "2019-08-05T16:09:37Z")
I am trying to check if a field exists within a pair in an array inside of a larger object. I've tried to just add the field as is to make sure i was using the proper syntax and it works. I've done mutate { add\_field … --- ## [Move json file to elasticsearch](https://discuss.elastic.co/t/move-json-file-to-elasticsearch/193644)
**Author:** [@Jim\_Palazzolo](https://discuss.elastic.co/u/Jim_Palazzolo)\ **Replies:** 26\ **Last updated:** [August 5, 2019, 3:58pm UTC](https://discuss.elastic.co/t/move-json-file-to-elasticsearch/193644 "2019-08-05T15:58:57Z")
trying to move a file from a folder using logstash to elasticsearch. here's my json.conf: input { file { path =\> "/usr/share/logstash/json" start\_position =\> "beginning" … --- ## [Logstash creates switch case using ruby code](https://discuss.elastic.co/t/logstash-creates-switch-case-using-ruby-code/193581)
**Author:** [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\ **Replies:** 6\ **Last updated:** [August 5, 2019, 3:03pm UTC](https://discuss.elastic.co/t/logstash-creates-switch-case-using-ruby-code/193581 "2019-08-05T15:03:49Z")
Hi all, Trying to do something like a switch case as below : if value \< 1 then field = 'low' else if value \>= 1 && \< 20 then field = 'medium' else if value \>= 20 && \< 50 then field = 'high' else field = 'veryHigh' but… --- ## [Ruby nooby looking for help installing logstash-input-okta plugin](https://discuss.elastic.co/t/ruby-nooby-looking-for-help-installing-logstash-input-okta-plugin/193821)
**Author:** [@bigdamhero](https://discuss.elastic.co/u/bigdamhero)\ **Replies:** 1\ **Last updated:** [August 5, 2019, 2:51pm UTC](https://discuss.elastic.co/t/ruby-nooby-looking-for-help-installing-logstash-input-okta-plugin/193821 "2019-08-05T14:51:16Z")
I am having difficulties due to not being very familiar with ruby and the fact that the security dept blocked access to github. I was able to obtain the files locally but when i go to edit the logstash Gemfile the comm… --- ## [Seraching value of variable](https://discuss.elastic.co/t/seraching-value-of-variable/193768)
**Author:** [@RickT](https://discuss.elastic.co/u/RickT)\ **Replies:** 2\ **Last updated:** [August 5, 2019, 12:59pm UTC](https://discuss.elastic.co/t/seraching-value-of-variable/193768 "2019-08-05T12:59:18Z")
hi, i'm searching to compare the value of a variable in a field (from "message" or from a specific field). I'm trying this, but without result : ... filter { mutate { add\_field =\> { "mails" =\> "joe, mary, pierre" } … --- ## [Logstash runs out of memory](https://discuss.elastic.co/t/logstash-runs-out-of-memory/193749)
**Author:** [@luc1](https://discuss.elastic.co/u/luc1)\ **Replies:** 0\ **Last updated:** [August 5, 2019, 8:19am UTC](https://discuss.elastic.co/t/logstash-runs-out-of-memory/193749 "2019-08-05T08:19:48Z")
Hello, Here is the issue: My logstash conf file is not sending files to Elasticsearch when it's full: filter{ json { source =\> "message" remove\_field =\>\[ "message" \] } … --- ## [Gork failures](https://discuss.elastic.co/t/gork-failures/193775)
**Author:** [@sharath\_1245](https://discuss.elastic.co/u/sharath_1245)\ **Replies:** 2\ **Last updated:** [August 5, 2019, 12:24pm UTC](https://discuss.elastic.co/t/gork-failures/193775 "2019-08-05T12:24:45Z")
Hi Team, We are getting multiple gork\_failures. Kindly provide how to reduce gork\_failure. --- ## [Logstash grok pattern failed check config](https://discuss.elastic.co/t/logstash-grok-pattern-failed-check-config/189939)
**Author:** [@Dave\_Hafid](https://discuss.elastic.co/u/Dave_Hafid)\ **Replies:** 22\ **Last updated:** [August 5, 2019, 11:29am UTC](https://discuss.elastic.co/t/logstash-grok-pattern-failed-check-config/189939 "2019-08-05T11:29:14Z")
Dear ALL, i try to create ssh grok pattern this is my pattern input { beats { port =\> 5044 host =\> "192.168.11.13" } } # Capture\_all\_MSG filter { if \[fileset\]\[name\] == "auth" { grok { match =\> { "message" =\>… --- ## [Logstash ssl error es input --\> output csv](https://discuss.elastic.co/t/logstash-ssl-error-es-input-output-csv/193759)
**Author:** [@subok](https://discuss.elastic.co/u/subok)\ **Replies:** 0\ **Last updated:** [August 5, 2019, 9:12am UTC](https://discuss.elastic.co/t/logstash-ssl-error-es-input-output-csv/193759 "2019-08-05T09:12:43Z")
I'm getting the following error when running a logstash: tric=\>true, charset=\>"UTF-8"\>, size=\>1000, scroll=\>"1m", docinfo=\>false, docinfo\_target=\>"@metadata", docinfo\_fields=\>\["\_index", "\_type", "\_id"\], ssl=\>false\> Err… --- ## [Decorate\_events returns the literal strin](https://discuss.elastic.co/t/decorate-events-returns-the-literal-strin/193574)
**Author:** [@mirajp](https://discuss.elastic.co/u/mirajp)\ **Replies:** 2\ **Last updated:** [August 5, 2019, 7:34am UTC](https://discuss.elastic.co/t/decorate-events-returns-the-literal-strin/193574 "2019-08-05T07:34:06Z")
Hi, I'm trying to use the decorate\_events feature in logstash to pull kafka details and put them into Elastic search. The issue is, in Elastic search, the value is being set to "%{\[metadata\]\[kafka\]\[topic\]}" rather than … --- ## [All output fail if one of tcp outputs fails](https://discuss.elastic.co/t/all-output-fail-if-one-of-tcp-outputs-fails/193703)
**Author:** [@dickens88](https://discuss.elastic.co/u/dickens88)\ **Replies:** 4\ **Last updated:** [August 5, 2019, 3:05am UTC](https://discuss.elastic.co/t/all-output-fail-if-one-of-tcp-outputs-fails/193703 "2019-08-05T03:05:33Z")
Guys, my logstash configuration has multiply outputs, like this: input{ syslog { port =\> 514 } } output{ tcp { host =\> "192.168.1.1" port =\> 1517 codec =\> line { format =\> "%{mes… --- ## [Can we use Filebeat with ssl and logstash netflow module?](https://discuss.elastic.co/t/can-we-use-filebeat-with-ssl-and-logstash-netflow-module/193452)
**Author:** [@jsandri](https://discuss.elastic.co/u/jsandri)\ **Replies:** 1\ **Last updated:** [August 5, 2019, 1:58am UTC](https://discuss.elastic.co/t/can-we-use-filebeat-with-ssl-and-logstash-netflow-module/193452 "2019-08-05T01:58:18Z")
Hey! I use Filebeat with netflow input and send logs to logstash. So far so good but then I try to setup the logstash netflow module and now some issues appear. First Filebeat sends logs over TCP and logstash netflow mo… --- ## [Avro SchemaParseError](https://discuss.elastic.co/t/avro-schemaparseerror/190757)
**Author:** [@DSPN](https://discuss.elastic.co/u/DSPN)\ **Replies:** 2\ **Last updated:** [August 4, 2019, 11:13pm UTC](https://discuss.elastic.co/t/avro-schemaparseerror/190757 "2019-08-04T23:13:57Z")
Hi, I'm using logstash to load logs from Kafka to Elastic. Running logstash on openshift. I'm receiving the below error when trying to read the logs from the kafka topic. Can anyone help me figure what I'm missing here. … --- ## [Logstash like forwarder of beats. Why? Benefits?](https://discuss.elastic.co/t/logstash-like-forwarder-of-beats-why-benefits/193202)
**Author:** [@sergiodcm00](https://discuss.elastic.co/u/sergiodcm00)\ **Replies:** 4\ **Last updated:** [August 4, 2019, 8:30am UTC](https://discuss.elastic.co/t/logstash-like-forwarder-of-beats-why-benefits/193202 "2019-08-04T08:30:18Z")
Hello, I'm newby in ELK world. I'm using beats (filebeat and others) with output to Logstash. LS only forward to ES ingest node (https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html): input { be… --- ## [Dissect field with unknown length?](https://discuss.elastic.co/t/dissect-field-with-unknown-length/193643)
**Author:** [@eclipsed450](https://discuss.elastic.co/u/eclipsed450)\ **Replies:** 2\ **Last updated:** [August 3, 2019, 5:37pm UTC](https://discuss.elastic.co/t/dissect-field-with-unknown-length/193643 "2019-08-03T17:37:00Z")
Hi, Based on other posts I've made, I've learned that I should be able dissect a field (in this case, it's eventData.ObjectName into sub-fields, however I'm having an issue where some messages don't have the full length… --- ## [Parsing AWS Logs - field extraction](https://discuss.elastic.co/t/parsing-aws-logs-field-extraction/193635)
**Author:** [@Randy-312](https://discuss.elastic.co/u/Randy-312)\ **Replies:** 0\ **Last updated:** [August 3, 2019, 1:23pm UTC](https://discuss.elastic.co/t/parsing-aws-logs-field-extraction/193635 "2019-08-03T13:23:23Z")
I'm a fan of how the beats/logstash modules work, and can help provide logstash an easier way to break messages up so that we have explicit fields that are not originally in the metadata (ie apache logs). I'm looking to… --- ## [Recommended way to force date fields to be UTC](https://discuss.elastic.co/t/recommended-way-to-force-date-fields-to-be-utc/193404)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 8\ **Last updated:** [August 3, 2019, 12:54am UTC](https://discuss.elastic.co/t/recommended-way-to-force-date-fields-to-be-utc/193404 "2019-08-03T00:54:53Z")
Hello, using LogStash and ElasticSearch 5.6.10. My documents have a few fields of type "date", besides @timestamp. The template includes, for each one of them, a section like this "my\_datefield\_1": { "type": "date",… --- ## [Remove duplicate documents that have the same field value](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604)
**Author:** [@gabrielconte14](https://discuss.elastic.co/u/gabrielconte14)\ **Replies:** 3\ **Last updated:** [August 2, 2019, 6:47pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604 "2019-08-02T18:47:20Z")
I'm trying to retrieve data from MongoDB and remove documents that have the same values on a specific field. For example: If the value of the field "device\_uuid" of multiple documents are equal, I would like to filter t… --- ## [Collect jmx tabular data and visualize it into kibana](https://discuss.elastic.co/t/collect-jmx-tabular-data-and-visualize-it-into-kibana/193488)
**Author:** [@HenryDuong](https://discuss.elastic.co/u/HenryDuong)\ **Replies:** 0\ **Last updated:** [August 2, 2019, 8:54am UTC](https://discuss.elastic.co/t/collect-jmx-tabular-data-and-visualize-it-into-kibana/193488 "2019-08-02T08:54:48Z")
Hi guys, I'm new with ELK. Currently, I'm working with logstash-input-jmx plugin to collect MBean data and visualize it into Kibana. So technically, I stuck at 2 points: JMX data, I can collect the attribute with numb… --- ## [Update document while uploading mapping with logstash](https://discuss.elastic.co/t/update-document-while-uploading-mapping-with-logstash/193596)
**Author:** [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Replies:** 0\ **Last updated:** [August 2, 2019, 4:43pm UTC](https://discuss.elastic.co/t/update-document-while-uploading-mapping-with-logstash/193596 "2019-08-02T16:43:33Z")
Hi all, I have some map for mapping and data on elasticsearch. My filter plugin is the following elasticsearch{ hosts =\> \["localhost:9200"\] index =\> "mapping\_av" query =\> "signature:%{\[id\]}" result\_siz… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=497) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=499)