# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=917 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 918 --- ## [Offline plugin install](https://discuss.elastic.co/t/offline-plugin-install/98850)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 4\ **Last updated:** [August 30, 2017, 3:18pm UTC](https://discuss.elastic.co/t/offline-plugin-install/98850 "2017-08-30T15:18:21Z")
I am attempting an offline install the logstash-filter-cidr plugin on logstash 5.4. I have tried: Downloading the gem from rubygems.org and executing: sudo /usr/share/logstash/bin/logstash-plugin install file:///tmp… --- ## [Handle empty node in XPATH for xml filter](https://discuss.elastic.co/t/handle-empty-node-in-xpath-for-xml-filter/98674)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 2\ **Last updated:** [August 30, 2017, 2:25pm UTC](https://discuss.elastic.co/t/handle-empty-node-in-xpath-for-xml-filter/98674 "2017-08-30T14:25:56Z")
Hi All I have a XML file with 100 records, in one of the record the xml node is missing as the value is not available in system The xml path is like this /Envelope/Body/response/table/row/description/text in other re… --- ## [Logstash 5.4.3 crash "Regexp Interrupted error(s)"](https://discuss.elastic.co/t/logstash-5-4-3-crash-regexp-interrupted-error-s/98757)
**Author:** [@Daniel314](https://discuss.elastic.co/u/Daniel314)\ **Replies:** 1\ **Last updated:** [August 30, 2017, 1:45pm UTC](https://discuss.elastic.co/t/logstash-5-4-3-crash-regexp-interrupted-error-s/98757 "2017-08-30T13:45:29Z")
I’m seeing an issue similar to https://github.com/elastic/logstash/issues/5851, except that their problem is with an older version of LogStash. Here’s a sample output I get from Logstash, version 5.4.3 (reformatted slig… --- ## [Non-optional base64 encoding in Avro codec](https://discuss.elastic.co/t/non-optional-base64-encoding-in-avro-codec/97373)
**Author:** [@robinklaassen](https://discuss.elastic.co/u/robinklaassen)\ **Replies:** 1\ **Last updated:** [August 30, 2017, 1:23pm UTC](https://discuss.elastic.co/t/non-optional-base64-encoding-in-avro-codec/97373 "2017-08-30T13:23:06Z")
We’re using Logstash to write event data to a Kafka queue. The consumer of this queue expects an Avro binary blob (or fragment, the terminology seems to differ sometimes). It seems however, that using the Avro codec prov… --- ## [External file for match](https://discuss.elastic.co/t/external-file-for-match/98663)
**Author:** [@Alexander\_Orlenko](https://discuss.elastic.co/u/Alexander_Orlenko)\ **Replies:** 2\ **Last updated:** [August 30, 2017, 1:15pm UTC](https://discuss.elastic.co/t/external-file-for-match/98663 "2017-08-30T13:15:15Z")
Hello, We analise syslog msg from dhcpd daemon. And want use GeoIP, but we use internal ip addresses. How match internal ip or host name to City. For example, @timestamp August 29th 2017, 12:39:10.754 host … --- ## [Parsing Logstash Data](https://discuss.elastic.co/t/parsing-logstash-data/98685)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 5\ **Last updated:** [August 30, 2017, 11:51am UTC](https://discuss.elastic.co/t/parsing-logstash-data/98685 "2017-08-30T11:51:55Z")
Hi - I have the below message coming to the ELK, and I’d like to parse those values and create a mapping (Key =\> Value) and add as separate field in Elastic Search. \[image\] Please let me know how could I do this. --- ## [Increase total fields limit via creation of index in logstash](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 6\ **Last updated:** [August 30, 2017, 11:50am UTC](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742 "2017-08-30T11:50:28Z")
My logstash conf has the following output: output { elasticsearch { hosts =\> \["IP1:9200","IP2:9200"\] index =\> "test-${+YYYY.MM.dd}" user =\> elastic password =\> supersecret } } … --- ## [Logstash send alert if there is no log](https://discuss.elastic.co/t/logstash-send-alert-if-there-is-no-log/98847)
**Author:** [@bilalyasar](https://discuss.elastic.co/u/bilalyasar)\ **Replies:** 0\ **Last updated:** [August 30, 2017, 11:22am UTC](https://discuss.elastic.co/t/logstash-send-alert-if-there-is-no-log/98847 "2017-08-30T11:22:42Z")
Hi, in logstash, i want to check that: whether specific log exist or not. and if it is not happened last 24 hours then i want to send alert. is there any plugin for that? --- ## [Create nested field with XPath](https://discuss.elastic.co/t/create-nested-field-with-xpath/98230)
**Author:** [@cadoman](https://discuss.elastic.co/u/cadoman)\ **Replies:** 3\ **Last updated:** [August 30, 2017, 8:08am UTC](https://discuss.elastic.co/t/create-nested-field-with-xpath/98230 "2017-08-30T08:08:49Z")
I have an XML structure \ \ \ \ \ \ \… --- ## [Logstash for multi output](https://discuss.elastic.co/t/logstash-for-multi-output/98650)
**Author:** [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\ **Replies:** 5\ **Last updated:** [August 30, 2017, 4:46am UTC](https://discuss.elastic.co/t/logstash-for-multi-output/98650 "2017-08-30T04:46:12Z")
Hi experts. I have this logstash input input { beats { type =\> "beat" port =\> 5044 ssl =\> false #ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" #ssl\_key =\> "/etc/pki/tls/private/lo… --- ## [Logstash filter event.get() is hard to debug](https://discuss.elastic.co/t/logstash-filter-event-get-is-hard-to-debug/98662)
**Author:** [@John\_ax](https://discuss.elastic.co/u/John_ax)\ **Replies:** 3\ **Last updated:** [August 30, 2017, 2:16am UTC](https://discuss.elastic.co/t/logstash-filter-event-get-is-hard-to-debug/98662 "2017-08-30T02:16:49Z")
Hi, I am using logstash 5.1.1, got lots of ERROR logs, so I try this: echo '{"msg":"xxxx", "@timestamp":"2017-08-28T08:16:54.943Z", "service":"x.y.z", "host":"ubuntu","timestamp":1503908214}' | ./logstash -e 'input {… --- ## [Input from JDBC - Convert to JSON & Push to SNS Topic](https://discuss.elastic.co/t/input-from-jdbc-convert-to-json-push-to-sns-topic/98395)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 7\ **Last updated:** [August 29, 2017, 7:59pm UTC](https://discuss.elastic.co/t/input-from-jdbc-convert-to-json-push-to-sns-topic/98395 "2017-08-29T19:59:44Z")
Team, Not sure if this is possible via one logstash pipeline so appreciate feedback. Data Source: Oracle DB. Extract using standard JDBC input plugin. Destination: SNS After extracting fields from DB e.g. lets say my… --- ## [Creating Grok Pattern for logstash config](https://discuss.elastic.co/t/creating-grok-pattern-for-logstash-config/98305)
**Author:** [@sunny1](https://discuss.elastic.co/u/sunny1)\ **Replies:** 6\ **Last updated:** [August 29, 2017, 6:26pm UTC](https://discuss.elastic.co/t/creating-grok-pattern-for-logstash-config/98305 "2017-08-29T18:26:23Z")
Hello everyone out there! I recently started using ELK so sorry if my question silly. I need help with modifying my grok filter to achieve particular set of data. Tried lot but could not so looking for help. Log file … --- ## [Number filtering logstash](https://discuss.elastic.co/t/number-filtering-logstash/98599)
**Author:** [@luciferdude](https://discuss.elastic.co/u/luciferdude)\ **Replies:** 8\ **Last updated:** [August 29, 2017, 4:53pm UTC](https://discuss.elastic.co/t/number-filtering-logstash/98599 "2017-08-29T16:53:28Z")
{"name": "hello", "xy": "ab", "ip": "123.456.78.910", "abc": "AB", "nbc": 12345, "cidr": "123.456.78.9/10", "last\_seen\_at": "2017-08-15", "first\_seen\_at": "2015-09-03", "categories": \["hello"\]} I have an array of numb… --- ## [How can I split fields](https://discuss.elastic.co/t/how-can-i-split-fields/98682)
**Author:** [@mikygee](https://discuss.elastic.co/u/mikygee)\ **Replies:** 2\ **Last updated:** [August 29, 2017, 3:52pm UTC](https://discuss.elastic.co/t/how-can-i-split-fields/98682 "2017-08-29T15:52:14Z")
Hello, I have a field that looks like that lists: list1 list2 The number of lists is not defined, here it’s 2 but it can be 3 or 4 I’d like to create new fields with new values lists1 =\> list1 lists2 =\> list2 Do y… --- ## [Importing into nested document using logstash](https://discuss.elastic.co/t/importing-into-nested-document-using-logstash/98407)
**Author:** [@aditya8003](https://discuss.elastic.co/u/aditya8003)\ **Replies:** 0\ **Last updated:** [August 25, 2017, 6:59pm UTC](https://discuss.elastic.co/t/importing-into-nested-document-using-logstash/98407 "2017-08-25T18:59:16Z")
How do I feed nested data like the one below using Logstash extracted from a CSV file? I checked out the below posts but couldnt find a solution built into logstash that helps convert a flat row from a database or an ex… --- ## [S3 Input Plugin Not Fully Processing ELB Access Logs](https://discuss.elastic.co/t/s3-input-plugin-not-fully-processing-elb-access-logs/98408)
**Author:** [@corey-hammerton](https://discuss.elastic.co/u/corey-hammerton)\ **Replies:** 1\ **Last updated:** [August 29, 2017, 2:08pm UTC](https://discuss.elastic.co/t/s3-input-plugin-not-fully-processing-elb-access-logs/98408 "2017-08-29T14:08:18Z")
We are using the s3 input plugin to process our ELB logs and analyze them in Elasticsearch. Frequently, but inconsistently, it stops processing the log part-way through the file. Running Logstash in debug mode hasn’t rev… --- ## [HTTP LogStash Request](https://discuss.elastic.co/t/http-logstash-request/98362)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 3\ **Last updated:** [August 29, 2017, 1:01pm UTC](https://discuss.elastic.co/t/http-logstash-request/98362 "2017-08-29T13:01:17Z")
Hi - I have a webservice URL (something like this “http://149.212.194.252:8001/ths/ResourceMonitor”) Where I’d get the output in the form of XML nodes, I need to read this data for every second and push it to ELK. Wou… --- ## [Grok's strange data conversions](https://discuss.elastic.co/t/groks-strange-data-conversions/98582)
**Author:** [@Cornoualis](https://discuss.elastic.co/u/Cornoualis)\ **Replies:** 4\ **Last updated:** [August 29, 2017, 12:04pm UTC](https://discuss.elastic.co/t/groks-strange-data-conversions/98582 "2017-08-29T12:04:38Z")
Hi, I have really weird conversions when using grok. I really don’t get why the final values are so different from the originals: \[grok\] I use the DATA grok pattern because I sometimes have numbers with exponents (eg:… --- ## [Alerting by slack,email etc.. when log level error](https://discuss.elastic.co/t/alerting-by-slack-email-etc-when-log-level-error/98654)
**Author:** [@Julien\_Hac](https://discuss.elastic.co/u/Julien_Hac)\ **Replies:** 1\ **Last updated:** [August 29, 2017, 9:47am UTC](https://discuss.elastic.co/t/alerting-by-slack-email-etc-when-log-level-error/98654 "2017-08-29T09:47:01Z")
Hello, I have installed the ELK Stack and i would like be notify when a log error happen in production. Today i have to go in Kibana et check Log level error but its too much manual… Is there a way to be notify automa… --- ## [Graphite-input-plugin configuration for logstash](https://discuss.elastic.co/t/graphite-input-plugin-configuration-for-logstash/98620)
**Author:** [@vijeshkk](https://discuss.elastic.co/u/vijeshkk)\ **Replies:** 1\ **Last updated:** [August 29, 2017, 9:27am UTC](https://discuss.elastic.co/t/graphite-input-plugin-configuration-for-logstash/98620 "2017-08-29T09:27:15Z")
We have graphite based metric data collection in our environment and now we are trying to use ELK stack to create centralized data repository and dashboards. Since the graphite based metric collection infrastructure is … --- ## [Elastic stack on Azure](https://discuss.elastic.co/t/elastic-stack-on-azure/97583)
**Author:** [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\ **Replies:** 3\ **Last updated:** [August 29, 2017, 8:13am UTC](https://discuss.elastic.co/t/elastic-stack-on-azure/97583 "2017-08-29T08:13:45Z")
The first thing i noticed was that Logstash is missing from the ARM template. Is there any particular reason why? Can anyone point me to a resource as to how I could deploy Logstash on Azure? I am new to deploying appli… --- ## [Use of Split Filter for more than 1 fields, it is possible](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 14\ **Last updated:** [August 29, 2017, 7:23am UTC](https://discuss.elastic.co/t/use-of-split-filter-for-more-than-1-fields-it-is-possible/97687 "2017-08-29T07:23:39Z")
Hi All my output is like this Field1 : A,B,C,D Field2: E,F,G,H Field3: W,X,Y,Z Field4: Q,R,S,T Now i am using split filter in conf file split {field =\> "\[Field1 \]"} split {field =\> "\[Field2\]"} split {field =\> … --- ## [Multiline on "\\n"](https://discuss.elastic.co/t/multiline-on-n/98618)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 1\ **Last updated:** [August 29, 2017, 5:38am UTC](https://discuss.elastic.co/t/multiline-on-n/98618 "2017-08-29T05:38:42Z")
I have the following log blah… blah… \\n lah… lah… \\n ah… ah… how do I split the three lines into arrays? --- ## [Logstash stops randomly with NoMethodError: undefined method \`split' for nil:NilClass Exception](https://discuss.elastic.co/t/logstash-stops-randomly-with-nomethoderror-undefined-method-split-for-nil-nilclass-exception/98596)
**Author:** [@amit.karir](https://discuss.elastic.co/u/amit.karir)\ **Replies:** 1\ **Last updated:** [August 29, 2017, 5:30am UTC](https://discuss.elastic.co/t/logstash-stops-randomly-with-nomethoderror-undefined-method-split-for-nil-nilclass-exception/98596 "2017-08-29T05:30:21Z")
Hi, Logstash stops running randomly with below exception message. Would really appreciate any pointers to resolve this issue: Exception in pipelineworker, the pipeline stopped processing new events, please check your … --- ## [Data not showing when linking Mysql with ElasticSearch using logstash](https://discuss.elastic.co/t/data-not-showing-when-linking-mysql-with-elasticsearch-using-logstash/98524)
**Author:** [@vaibhav2903](https://discuss.elastic.co/u/vaibhav2903)\ **Replies:** 1\ **Last updated:** [August 29, 2017, 5:28am UTC](https://discuss.elastic.co/t/data-not-showing-when-linking-mysql-with-elasticsearch-using-logstash/98524 "2017-08-29T05:28:36Z")
Hi! I am trying to link mysql with ElasticSearch using logstash. The database is getting connected but the schemas and the data are not being shown. My .conf file: input { \*\* jdbc {\*\* \*\* jdbc\_connection\_str… --- ## [Problems with Redis input](https://discuss.elastic.co/t/problems-with-redis-input/98537)
**Author:** [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\ **Replies:** 5\ **Last updated:** [August 29, 2017, 4:45am UTC](https://discuss.elastic.co/t/problems-with-redis-input/98537 "2017-08-29T04:45:26Z")
Hi experts I have config to use Redis input for logstash. Here is my input logstash input { redis { host =\> "\*\*\*.\*\*\*.\*\*.\*\*\*" type =\> "redis-input" data\_type =\> "list" key =\> "filebeat" } } Whenever… --- ## [Re: Filebeat \> Kafka \> Logstash \> ElasticSearch](https://discuss.elastic.co/t/re-filebeat-kafka-logstash-elasticsearch/97888)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 4\ **Last updated:** [August 29, 2017, 3:24am UTC](https://discuss.elastic.co/t/re-filebeat-kafka-logstash-elasticsearch/97888 "2017-08-29T03:24:54Z")
Anyone here why beats data such as beat name, version etc is not included after indexing in elasticsearch? --- ## [How to import 'File outputed' Metricbeat data into ES via Logstash](https://discuss.elastic.co/t/how-to-import-file-outputed-metricbeat-data-into-es-via-logstash/94986)
**Author:** [@Patrick\_Stevens](https://discuss.elastic.co/u/Patrick_Stevens)\ **Replies:** 4\ **Last updated:** [August 28, 2017, 5:53pm UTC](https://discuss.elastic.co/t/how-to-import-file-outputed-metricbeat-data-into-es-via-logstash/94986 "2017-08-28T17:53:21Z")
Hi, I am attempting to use Logstash to import Metricbeat ‘file output’ files into Elasticsearch. I am using file output on Metricbeat because the servers I’m monitoring are on a different domain than my ES, Logstash, and… --- ## [Patterns directory and relevance with elasticsearch](https://discuss.elastic.co/t/patterns-directory-and-relevance-with-elasticsearch/98457)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 2\ **Last updated:** [August 28, 2017, 2:34pm UTC](https://discuss.elastic.co/t/patterns-directory-and-relevance-with-elasticsearch/98457 "2017-08-28T14:34:09Z")
Hi Guys, I am just trying to find the relevance document on patterns directory and what is the function? I am bit confused on patterns directory. I mean lets say I am accpeting nginx data using logstash with appropriate… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=916) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=918)