# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=839 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 840 --- ## [Ingesting leef formatted log into logstash from other sources](https://discuss.elastic.co/t/ingesting-leef-formatted-log-into-logstash-from-other-sources/115937)
**Author:** [@Zorkmid](https://discuss.elastic.co/u/Zorkmid)\ **Replies:** 1\ **Last updated:** [January 17, 2018, 8:04pm UTC](https://discuss.elastic.co/t/ingesting-leef-formatted-log-into-logstash-from-other-sources/115937 "2018-01-17T20:04:17Z")
Hi All, I'd like to ship specific event logs from my Siem to ElasticSearch for further processing. However the output from the Siem is in the leef format and can not be changed. Has anyone on the lists done this? If … --- ## [How to set variables in config logstash](https://discuss.elastic.co/t/how-to-set-variables-in-config-logstash/115935)
**Author:** [@cgarcia](https://discuss.elastic.co/u/cgarcia)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 6:35pm UTC](https://discuss.elastic.co/t/how-to-set-variables-in-config-logstash/115935 "2018-01-17T18:35:38Z")
Hi Expert I need your help. How to set this variables in the config of logstash? ELASTIFLOW\_NETFLOW\_PORT The UDP port to listen on for Netflow messages 2055 ELASTIFLOW\_ES\_HOST The Elasticsearch host to which t… --- ## [Logstash JDBC document\_id not reading latest](https://discuss.elastic.co/t/logstash-jdbc-document-id-not-reading-latest/115610)
**Author:** [@bhrtjoshi](https://discuss.elastic.co/u/bhrtjoshi)\ **Replies:** 4\ **Last updated:** [January 17, 2018, 5:42pm UTC](https://discuss.elastic.co/t/logstash-jdbc-document-id-not-reading-latest/115610 "2018-01-17T17:42:13Z")
Hello There, I am referring this article https://www.elastic.co/blog/logstash-jdbc-input-plugin to ingest data from Oracle table. I need to get Oracle result set from some huge queries but before I do that I am playing … --- ## [Logstash - Old entries/ log files](https://discuss.elastic.co/t/logstash-old-entries-log-files/115926)
**Author:** [@kanwal](https://discuss.elastic.co/u/kanwal)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 4:49pm UTC](https://discuss.elastic.co/t/logstash-old-entries-log-files/115926 "2018-01-17T16:49:59Z")
Hi, I am reading IIS log entries from Logstash (version 6.1.1). Its not able to read the old log files. Please advice --- ## [Add new elements in XML array](https://discuss.elastic.co/t/add-new-elements-in-xml-array/115437)
**Author:** [@Benjamin\_Carriou](https://discuss.elastic.co/u/Benjamin_Carriou)\ **Replies:** 1\ **Last updated:** [January 17, 2018, 2:59pm UTC](https://discuss.elastic.co/t/add-new-elements-in-xml-array/115437 "2018-01-17T14:59:04Z")
Hi all ! I have the following XML file: \ \ \ \ \ **Author:** [@Suryakumar](https://discuss.elastic.co/u/Suryakumar)\ **Replies:** 1\ **Last updated:** [January 17, 2018, 2:56pm UTC](https://discuss.elastic.co/t/importing-csv-to-elasticsearch-using-logstash/115880 "2018-01-17T14:56:20Z") Hi, I am trying to import a CSV file to elasticsearch using Logstash. This is my config file: input { file { path =\> \["/home/suryakumar/Sample.csv"\] start\_position =\> "beginning" } … --- ## [Filter Nested Data based on pattern](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 5\ **Last updated:** [January 17, 2018, 1:59pm UTC](https://discuss.elastic.co/t/filter-nested-data-based-on-pattern/115510 "2018-01-17T13:59:44Z")
I have incoming JSON data that looks like: { "DataType1": { "system:subsystem" : \[{..some nested data fields}\] "DataType2": { "system:subsystem" : \[{..some nested data fields}\] } I am trying to separate the events, s… --- ## [Http input with ssl](https://discuss.elastic.co/t/http-input-with-ssl/115338)
**Author:** [@skydiablo](https://discuss.elastic.co/u/skydiablo)\ **Replies:** 2\ **Last updated:** [January 17, 2018, 1:52pm UTC](https://discuss.elastic.co/t/http-input-with-ssl/115338 "2018-01-17T13:52:54Z")
hey guys, im very happy with my current ES-Stack and all works fine... except my http input with SSL on :frowning: if i enable the SSL, there is no way to push stuff to my endpoint, this is my conf: input { http { … --- ## [How to get kv filter to ignore value\_split in data](https://discuss.elastic.co/t/how-to-get-kv-filter-to-ignore-value-split-in-data/115846)
**Author:** [@Nofar](https://discuss.elastic.co/u/Nofar)\ **Replies:** 2\ **Last updated:** [January 17, 2018, 11:58am UTC](https://discuss.elastic.co/t/how-to-get-kv-filter-to-ignore-value-split-in-data/115846 "2018-01-17T11:58:55Z")
My conf file: kv { source =\> "cp\_keyValueData" field\_split =\> ";" value\_split =\> ":" trim\_key =\> " " } data example: key1: val1; key2: val2; key3: https://site/?g={......"...;%20%20CLR%20;%20rv:11.0)"..}; key… --- ## [Heap Size in Logstash](https://discuss.elastic.co/t/heap-size-in-logstash/115872)
**Author:** [@devil\_srj7](https://discuss.elastic.co/u/devil_srj7)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 11:38am UTC](https://discuss.elastic.co/t/heap-size-in-logstash/115872 "2018-01-17T11:38:37Z")
While importing the table of size 9 GB in elasticsearch through Logstash i get the following error. \[2018-01-17T16:34:59,948\]\[INFO \]\[logstash.inputs.jdbc \] (120.731000s) SELECT \* from mis\_monthly\_work\_statistics \[20… --- ## [How to set /n as delimiter in dissect filter](https://discuss.elastic.co/t/how-to-set-n-as-delimiter-in-dissect-filter/112588)
**Author:** [@George\_Cherian](https://discuss.elastic.co/u/George_Cherian)\ **Replies:** 3\ **Last updated:** [January 17, 2018, 10:39am UTC](https://discuss.elastic.co/t/how-to-set-n-as-delimiter-in-dissect-filter/112588 "2018-01-17T10:39:25Z")
Hi Elastic Team, I need to filter of the field with \\n as the delimiter in dissect filter. Saw that logstash 6.0 has feature enabled. config.support\_escapes: true The filter what I have written is as follows; disse… --- ## [Logstash-sqs-output](https://discuss.elastic.co/t/logstash-sqs-output/115860)
**Author:** [@fenglilee](https://discuss.elastic.co/u/fenglilee)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 10:22am UTC](https://discuss.elastic.co/t/logstash-sqs-output/115860 "2018-01-17T10:22:30Z")
hi, guys. I am a little confused about the logstash sqs output. In the tutorial I seems the following configuration can work. output { sqs { id =\> "my\_plugin\_id" } } But I am wondering where is message sent, there… --- ## [Queueing LogEvents in Redis or similar needed?](https://discuss.elastic.co/t/queueing-logevents-in-redis-or-similar-needed/115308)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 6\ **Last updated:** [January 17, 2018, 10:14am UTC](https://discuss.elastic.co/t/queueing-logevents-in-redis-or-similar-needed/115308 "2018-01-17T10:14:14Z")
Hi, I sometimes find elastic architectures where Logstash, which parses the logs, is located behind a redis or kafka instance. What are the advantages of such a setup? Currently we are shipping our logs via filebeat (… --- ## [Skip lines in a log file](https://discuss.elastic.co/t/skip-lines-in-a-log-file/115276)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [January 17, 2018, 9:07am UTC](https://discuss.elastic.co/t/skip-lines-in-a-log-file/115276 "2018-01-17T09:07:05Z")
Hey i have a log file like this: Job Name : save\_start\_config-sftp Job Status: Success (0) Schedule Name : daily\_save\_start\_config User Name : mfservice Completion time: Mon Jan 8 06:00:21 … --- ## [S3 output behaviour when internet goes down?](https://discuss.elastic.co/t/s3-output-behaviour-when-internet-goes-down/115850)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 9:00am UTC](https://discuss.elastic.co/t/s3-output-behaviour-when-internet-goes-down/115850 "2018-01-17T09:00:57Z")
Hi, As per the subject, how does the S3 output react when the internet connection is interrupted? Are all temporarily files saved until the connection comes back up? If a connection fails during transmission, will the… --- ## [Netflow codec not working as expected](https://discuss.elastic.co/t/netflow-codec-not-working-as-expected/115822)
**Author:** [@JustinHoMi](https://discuss.elastic.co/u/JustinHoMi)\ **Replies:** 0\ **Last updated:** [January 17, 2018, 4:28am UTC](https://discuss.elastic.co/t/netflow-codec-not-working-as-expected/115822 "2018-01-17T04:28:31Z")
I've just installed the logstash 6.1.1.1-1 rpm on CentOS 7 and am having trouble getting the netflow codec to work as expected. It appears as if "/usr/share/logstash/modules/netflow/configuration/logstash/netflow.conf.er… --- ## [How to add another timestamp?](https://discuss.elastic.co/t/how-to-add-another-timestamp/115808)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [January 17, 2018, 3:01am UTC](https://discuss.elastic.co/t/how-to-add-another-timestamp/115808 "2018-01-17T03:01:12Z")
Hi, I'm ingesting some logs containing a @timestamp using the json codec. This is working but I'd also like to save the timestamp containing the date and time logstash ingested my logs. Can logstash create a new timest… --- ## [MultiplePipeline not working.. Lost all day :-(](https://discuss.elastic.co/t/multiplepipeline-not-working-lost-all-day/115802)
**Author:** [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Replies:** 1\ **Last updated:** [January 17, 2018, 12:05am UTC](https://discuss.elastic.co/t/multiplepipeline-not-working-lost-all-day/115802 "2018-01-17T00:05:17Z")
Trying to get multiple pipelines work from logstash and fed up with settings. Below are details logstash.yml path.data: /var/lib/logstash path.logs: /var/log/logstash Note that "path.config" is not present as I want … --- ## [License check warnings and logstash fails to work](https://discuss.elastic.co/t/license-check-warnings-and-logstash-fails-to-work/115783)
**Author:** [@GSCully](https://discuss.elastic.co/u/GSCully)\ **Replies:** 2\ **Last updated:** [January 16, 2018, 9:22pm UTC](https://discuss.elastic.co/t/license-check-warnings-and-logstash-fails-to-work/115783 "2018-01-16T21:22:58Z")
First off Logstash is not on the cluster it is a seperate vm, (Windows) After installing x-pack for monitoring I get an error spamming my console window, spamming th logs is this, \[logstash.licensechecker.licensereade… --- ## [JDBC\_Streaming Filter Plugin PoolTimeout Issue](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380)
**Author:** [@msugas19](https://discuss.elastic.co/u/msugas19)\ **Replies:** 7\ **Last updated:** [January 16, 2018, 8:38pm UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-plugin-pooltimeout-issue/115380 "2018-01-16T20:38:33Z")
Hello everyone. I am pulling data into logstash using the JDBC input plugin filter with a fetch size of 10,000 and for each of those records I am trying to perform another query to get child attributes for the parent doc… --- ## [Event field in logstash output](https://discuss.elastic.co/t/event-field-in-logstash-output/115717)
**Author:** [@Akaren](https://discuss.elastic.co/u/Akaren)\ **Replies:** 3\ **Last updated:** [January 16, 2018, 6:39pm UTC](https://discuss.elastic.co/t/event-field-in-logstash-output/115717 "2018-01-16T18:39:37Z")
Hi, in my logstash config in output part I have sns settings: sns { region =\> "eu-west-1" arn =\> "xxxx" } It works but every event has inside field called "SNSregion" that contains region (we get it from user gui...). … --- ## [Cannot convert type of nested field](https://discuss.elastic.co/t/cannot-convert-type-of-nested-field/115768)
**Author:** [@leafarlins](https://discuss.elastic.co/u/leafarlins)\ **Replies:** 0\ **Last updated:** [January 16, 2018, 6:03pm UTC](https://discuss.elastic.co/t/cannot-convert-type-of-nested-field/115768 "2018-01-16T18:03:07Z")
Hi. I am trying to convert a field read by a json filter in logstash. I am used to convert fields from other sources, but with this nested field from a json log file, I cannot convert a quoted number string to a integer. … --- ## [Logstash keeps crashing](https://discuss.elastic.co/t/logstash-keeps-crashing/114713)
**Author:** [@crapoulou](https://discuss.elastic.co/u/crapoulou)\ **Replies:** 9\ **Last updated:** [January 16, 2018, 5:38pm UTC](https://discuss.elastic.co/t/logstash-keeps-crashing/114713 "2018-01-16T17:38:26Z")
Hello, Every day, logstash is crashing and generating dozens of errors per second in the logs concerning Kafka. I run ELK 5.6.4, Java 1.8.0\_151, Kafka 0.10.0.1. I get 2 logstash servers for indexing linked with Kafka … --- ## [Dynamic template with logstash question](https://discuss.elastic.co/t/dynamic-template-with-logstash-question/115766)
**Author:** [@tomrade](https://discuss.elastic.co/u/tomrade)\ **Replies:** 0\ **Last updated:** [January 16, 2018, 5:30pm UTC](https://discuss.elastic.co/t/dynamic-template-with-logstash-question/115766 "2018-01-16T17:30:37Z")
Hi guyes, im having some issues with my template and I wanted to clear up if im using the path\_unmatch correctly. Im using the default logstash index template with one change. The keyword match "\*" now has a path\_unmat… --- ## [Define logstash's jdbc-input-plugin json message root object](https://discuss.elastic.co/t/define-logstashs-jdbc-input-plugin-json-message-root-object/115157)
**Author:** [@endersonmaia](https://discuss.elastic.co/u/endersonmaia)\ **Replies:** 2\ **Last updated:** [January 16, 2018, 5:16pm UTC](https://discuss.elastic.co/t/define-logstashs-jdbc-input-plugin-json-message-root-object/115157 "2018-01-16T17:16:55Z")
I'm using jdbc-input-plugin to get some orders data and send it to a rabbitmq queue. To make the json message compatible with the queue consuming system, I need to add a root named order. I'm getting { "id" : 123, "pr… --- ## [Best methods for polling logs from a web-client?](https://discuss.elastic.co/t/best-methods-for-polling-logs-from-a-web-client/115761)
**Author:** [@Jason-Terry](https://discuss.elastic.co/u/Jason-Terry)\ **Replies:** 0\ **Last updated:** [January 16, 2018, 4:54pm UTC](https://discuss.elastic.co/t/best-methods-for-polling-logs-from-a-web-client/115761 "2018-01-16T16:54:52Z")
Hey folks, just trying to get used to working with the ELK stack, I have set up a small project and got a mock API running, logging to a file, which is tailed by Logstash, and then pushed into Elasticsearch. My next step… --- ## [How to generate single object from multiple log lines in logstash filter?](https://discuss.elastic.co/t/how-to-generate-single-object-from-multiple-log-lines-in-logstash-filter/115215)
**Author:** [@mohits1995](https://discuss.elastic.co/u/mohits1995)\ **Replies:** 2\ **Last updated:** [January 16, 2018, 4:46pm UTC](https://discuss.elastic.co/t/how-to-generate-single-object-from-multiple-log-lines-in-logstash-filter/115215 "2018-01-16T16:46:03Z")
Hi, I am new to Logstash and Grok filter. I want to parse logs like these - 2018-01-11 17:17:16,071 | DEBUG | \[Thread-2\] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | CommittedVirtualMemorySize :: 401186816 … --- ## [Data conversion & enrichment with Logstash](https://discuss.elastic.co/t/data-conversion-enrichment-with-logstash/114921)
**Author:** [@Ranieri\_Mazili\_de\_Ol](https://discuss.elastic.co/u/Ranieri_Mazili_de_Ol)\ **Replies:** 1\ **Last updated:** [January 16, 2018, 4:44pm UTC](https://discuss.elastic.co/t/data-conversion-enrichment-with-logstash/114921 "2018-01-16T16:44:50Z")
I'm developing a product with Logstash + Elasticsearch + Kibana. I needed to make Logstash listen a JMS queue and I made it using JMS Input Plugin. I needed to convert the xml to json and I made it using JMS XML filter… --- ## [An unexpected error occurred! {:error=\>#\,](https://discuss.elastic.co/t/an-unexpected-error-occurred-error-argumenterror-setting-localhostck-monitoring-elasticsearch-username-hasnt-been-registered/115232)
**Author:** [@shaonbean](https://discuss.elastic.co/u/shaonbean)\ **Replies:** 1\ **Last updated:** [January 16, 2018, 4:31pm UTC](https://discuss.elastic.co/t/an-unexpected-error-occurred-error-argumenterror-setting-localhostck-monitoring-elasticsearch-username-hasnt-been-registered/115232 "2018-01-16T16:31:46Z")
env: elastic stack 6.1.1 centos6.9 when install x-pack,start logstash erros \[2018-01-12T15:31:58,463\]\[INFO \]\[logstash.modules.scaffold\] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/mod… --- ## [Index naming problem](https://discuss.elastic.co/t/index-naming-problem/115712)
**Author:** [@JohanLanderholm](https://discuss.elastic.co/u/JohanLanderholm)\ **Replies:** 1\ **Last updated:** [January 16, 2018, 1:43pm UTC](https://discuss.elastic.co/t/index-naming-problem/115712 "2018-01-16T13:43:40Z")
Hi, I'm trying to insert documents into an Elasticsearch index from a filebeat, through Logstash processing. My Logstash config looks like this: input { beats { port =\> 5044 } } output { elasticsearch { hosts =\>… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=838) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=840)