# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=719 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 720 --- ## [Trouble parsing logs from different devices using custom defined patterns](https://discuss.elastic.co/t/trouble-parsing-logs-from-different-devices-using-custom-defined-patterns/140704)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 0\ **Last updated:** [July 19, 2018, 10:05am UTC](https://discuss.elastic.co/t/trouble-parsing-logs-from-different-devices-using-custom-defined-patterns/140704 "2018-07-19T10:05:57Z")
Hi We want to have a directory with several files containing different grok expressions for different devices and vendors. So we created a directory called "patterns" in "/etc/logstash/conf.d/". In that folder we have t… --- ## [Pipeline.batch.delay ignored?](https://discuss.elastic.co/t/pipeline-batch-delay-ignored/140697)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 0\ **Last updated:** [July 19, 2018, 9:27am UTC](https://discuss.elastic.co/t/pipeline-batch-delay-ignored/140697 "2018-07-19T09:27:24Z")
hello all, I've got LS 6.3.1 with the following config: path.data: /var/lib/logstash pipeline.workers: 6 pipeline.batch.size: 1000 pipeline.batch.delay: 30000 path.config: /etc/logstash/conf.d queue.type: persisted pat… --- ## [Logstash--\> http\_output plugin --\>Telegram](https://discuss.elastic.co/t/logstash-http-output-plugin-telegram/140526)
**Author:** [@ysumit](https://discuss.elastic.co/u/ysumit)\ **Replies:** 1\ **Last updated:** [July 19, 2018, 8:39am UTC](https://discuss.elastic.co/t/logstash-http-output-plugin-telegram/140526 "2018-07-19T08:39:31Z")
How to send emoji as part of message to telegram bot using http\_ouput\_plugin of logstash in json format and post method. SY --- ## [How to run logstash as a service in CentOS](https://discuss.elastic.co/t/how-to-run-logstash-as-a-service-in-centos/140461)
**Author:** [@ramanna\_hk](https://discuss.elastic.co/u/ramanna_hk)\ **Replies:** 11\ **Last updated:** [July 19, 2018, 6:23am UTC](https://discuss.elastic.co/t/how-to-run-logstash-as-a-service-in-centos/140461 "2018-07-19T06:23:52Z")
I am able to run logstash successfully using below: /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/app.conf But it is not working, when i run following commands /usr/share/logstash/bin/logstash -f /etc/logst… --- ## [Elk server, reboot, 23 g was released](https://discuss.elastic.co/t/elk-server-reboot-23-g-was-released/140626)
**Author:** [@databeata](https://discuss.elastic.co/u/databeata)\ **Replies:** 3\ **Last updated:** [July 19, 2018, 7:09am UTC](https://discuss.elastic.co/t/elk-server-reboot-23-g-was-released/140626 "2018-07-19T07:09:30Z")
After reboot, disk /dev/sda1 used went down from 30 g to 7 g. The 23 g was released. It has happened few times. I have only elk running on it. Next time, how do I find what is taking that 23 g? before reboot: df -h a… --- ## [Logstash.config.sourceloader\] No configuration found in the configured sources](https://discuss.elastic.co/t/logstash-config-sourceloader-no-configuration-found-in-the-configured-sources/140440)
**Author:** [@Sunil\_Khullar](https://discuss.elastic.co/u/Sunil_Khullar)\ **Replies:** 1\ **Last updated:** [July 19, 2018, 6:33am UTC](https://discuss.elastic.co/t/logstash-config-sourceloader-no-configuration-found-in-the-configured-sources/140440 "2018-07-19T06:33:44Z")
I am unable to get multiple pipelines to work on Windows Individually I am able to run the config files, but not from inside the pipelines.yml LOG - \[2018-07-18T10:31:14,430\]\[DEBUG\]\[logstash.config.source.multilocal\] … --- ## [Logstash-plain.log is 17G in size](https://discuss.elastic.co/t/logstash-plain-log-is-17g-in-size/140643)
**Author:** [@databeata](https://discuss.elastic.co/u/databeata)\ **Replies:** 4\ **Last updated:** [July 19, 2018, 1:22am UTC](https://discuss.elastic.co/t/logstash-plain-log-is-17g-in-size/140643 "2018-07-19T01:22:09Z")
Why is logstash-plain.log 17G in size? ls -lh /path/logstash/path\_logs -rw-r--r-- 1 root root 4.8M Jul 17 23:59 logstash-plain-2018-07-17.log -rw-r--r-- 1 root root 17G Jul 18 22:43 logstash-plain.log cat /path/logs… --- ## [Problem with S3 Input](https://discuss.elastic.co/t/problem-with-s3-input/140412)
**Author:** [@jlawrenc](https://discuss.elastic.co/u/jlawrenc)\ **Replies:** 3\ **Last updated:** [July 18, 2018, 11:27pm UTC](https://discuss.elastic.co/t/problem-with-s3-input/140412 "2018-07-18T23:27:51Z")
This is all new to me, but I'm trying to pull Cisco managed S3 bucket gzip logs using the below input. The error message I'm seeing in /var/log/logstash/logstash.log is "Logstash S3 input, stop reading in the middle of t… --- ## ['appid' is not recognized as an internal or external command](https://discuss.elastic.co/t/appid-is-not-recognized-as-an-internal-or-external-command/140409)
**Author:** [@APJ](https://discuss.elastic.co/u/APJ)\ **Replies:** 9\ **Last updated:** [July 18, 2018, 11:14pm UTC](https://discuss.elastic.co/t/appid-is-not-recognized-as-an-internal-or-external-command/140409 "2018-07-18T23:14:00Z")
Hi all, I am following this link (Link) to display live weather data on Kibana. I used the logstash configuration explained in the pdf. Here is my logstash configuration: input { exec { command =\> "curl --silen… --- ## [Multiline issue with log](https://discuss.elastic.co/t/multiline-issue-with-log/140598)
**Author:** [@AGarcMan](https://discuss.elastic.co/u/AGarcMan)\ **Replies:** 5\ **Last updated:** [July 18, 2018, 10:05pm UTC](https://discuss.elastic.co/t/multiline-issue-with-log/140598 "2018-07-18T22:05:32Z")
Hi, I'm working with a log with the following structure (fragment): 2018-01-30 18:14:10.466554 (0x03000000:PCDataField ):folio = '201711231427280265' (CHARACTER) (0x03000000:PCDataFiel… --- ## [Conditionals within filter using fields generated by cloudwatch logs plugin](https://discuss.elastic.co/t/conditionals-within-filter-using-fields-generated-by-cloudwatch-logs-plugin/140609)
**Author:** [@vacovsky](https://discuss.elastic.co/u/vacovsky)\ **Replies:** 2\ **Last updated:** [July 18, 2018, 8:10pm UTC](https://discuss.elastic.co/t/conditionals-within-filter-using-fields-generated-by-cloudwatch-logs-plugin/140609 "2018-07-18T20:10:24Z")
I'm having an issue getting my grok filter to apply to VPC logs, and hoping someone here can help. filter { if "vpc-logs" in \[cloudwatch\_logs.log\_group\] { grok { patterns\_dir =\> \["/etc/logstash/p… --- ## [Logstash Ruby filter error](https://discuss.elastic.co/t/logstash-ruby-filter-error/140375)
**Author:** [@Darshan\_Parab](https://discuss.elastic.co/u/Darshan_Parab)\ **Replies:** 9\ **Last updated:** [July 18, 2018, 6:37pm UTC](https://discuss.elastic.co/t/logstash-ruby-filter-error/140375 "2018-07-18T18:37:41Z")
Hi All, I have written a ruby filter to check if a pattern present in a message and drop the event if pattern is not present. I'm getting below mentioned error after updating the logstash configuration, \[2018-07-17T23… --- ## [Gsub replacing \\\\n with \\n](https://discuss.elastic.co/t/gsub-replacing-n-with-n/140517)
**Author:** [@saramali](https://discuss.elastic.co/u/saramali)\ **Replies:** 10\ **Last updated:** [July 18, 2018, 4:11pm UTC](https://discuss.elastic.co/t/gsub-replacing-n-with-n/140517 "2018-07-18T16:11:27Z")
I want to replace '\\\\n' with '\\n' i am using the gsub method but cannot replace ruby { code =\> "@mystring=event.get('stockLines'); @mystring=@mystring.gsub('\\\\n', '\\n');" } --- ## [Logstash-input-kinesis plugin](https://discuss.elastic.co/t/logstash-input-kinesis-plugin/140230)
**Author:** [@rshadmon](https://discuss.elastic.co/u/rshadmon)\ **Replies:** 2\ **Last updated:** [July 18, 2018, 4:02pm UTC](https://discuss.elastic.co/t/logstash-input-kinesis-plugin/140230 "2018-07-18T16:02:50Z")
I was wondering if there is a way to not use DynamoDB in the logstash-input-kinesis plugin? If it must be used, can someone explain why? Thanks in advance! --- ## [Parse nested key-values](https://discuss.elastic.co/t/parse-nested-key-values/139771)
**Author:** [@tgaudin](https://discuss.elastic.co/u/tgaudin)\ **Replies:** 8\ **Last updated:** [July 18, 2018, 2:52pm UTC](https://discuss.elastic.co/t/parse-nested-key-values/139771 "2018-07-18T14:52:22Z")
Hi, I am trying to parse Oracle listener.log file. Here is a sample: 29-NOV. -2017 14:34:48 \* (CONNECT\_DATA=(CID=(PROGRAM=)(HOST=localhost)(USER=grid))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=203424000)… --- ## [Grok for Elasticsearch Logs](https://discuss.elastic.co/t/grok-for-elasticsearch-logs/139172)
**Author:** [@Ben96](https://discuss.elastic.co/u/Ben96)\ **Replies:** 1\ **Last updated:** [July 18, 2018, 2:48pm UTC](https://discuss.elastic.co/t/grok-for-elasticsearch-logs/139172 "2018-07-18T14:48:38Z")
Hi everyone, hope you have a nice day. We're currently running a full stack elk 5.6.9, all's good but i'm currently searching if anyone already ingest elasticsearch own logs into logstash ( to exploit them after ). I d… --- ## [Problem visualising nginx access logs in grafana](https://discuss.elastic.co/t/problem-visualising-nginx-access-logs-in-grafana/140579)
**Author:** [@nycterent](https://discuss.elastic.co/u/nycterent)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 2:31pm UTC](https://discuss.elastic.co/t/problem-visualising-nginx-access-logs-in-grafana/140579 "2018-07-18T14:31:42Z")
hi, I have had success pushing logs using filebeat to elasticsearch and visualising them in grafana, but when introduced logstash between filebeat and elasticsearch graphs are gone. in grafana i get this error: Fieldda… --- ## [Passing input values to logstash from external file](https://discuss.elastic.co/t/passing-input-values-to-logstash-from-external-file/139766)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 4\ **Last updated:** [July 18, 2018, 2:23pm UTC](https://discuss.elastic.co/t/passing-input-values-to-logstash-from-external-file/139766 "2018-07-18T14:23:29Z")
Hi All, I have been trying to use logstash http\_poller method to fetch data from one of my source. The plan is to pull data from a restapi source which requires an URL, user ccredentials i was able to pass these two fi… --- ## [Jdbc input- count error](https://discuss.elastic.co/t/jdbc-input-count-error/140217)
**Author:** [@bclipp770](https://discuss.elastic.co/u/bclipp770)\ **Replies:** 2\ **Last updated:** [July 18, 2018, 1:47pm UTC](https://discuss.elastic.co/t/jdbc-input-count-error/140217 "2018-07-18T13:47:16Z")
I'm pulling data from an advantage db, and I'm having issues with the query being run prior to statement I'm setting. \[SAP\]\[Advantage SQL Engine\]Unexpected token: LIMIT -- Expecting semicolon. -- Location of error in th… --- ## [Using gzip\_line codec](https://discuss.elastic.co/t/using-gzip-line-codec/140568)
**Author:** [@paresh\_Panda](https://discuss.elastic.co/u/paresh_Panda)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 1:30pm UTC](https://discuss.elastic.co/t/using-gzip-line-codec/140568 "2018-07-18T13:30:47Z")
Hi I am trying to evaluate the gzip\_line codec . Below is my use-case. I have my application trace files in CSV format and as days go it is gziped to gz files. Now I want to feed the gz files to logstash where I can ha… --- ## [Modify date format](https://discuss.elastic.co/t/modify-date-format/140521)
**Author:** [@niraj\_pandey](https://discuss.elastic.co/u/niraj_pandey)\ **Replies:** 1\ **Last updated:** [July 18, 2018, 11:56am UTC](https://discuss.elastic.co/t/modify-date-format/140521 "2018-07-18T11:56:19Z")
Hi , Is there any way to modify date format on logstash filter file. I am using the current once date { match =\> \[ "submit\_time\_unix", "UNIX" \] target =\> "submit\_time\_unix" } and the o/p is "June 22nd 2016, 01:54:… --- ## [Add new line character ruby filter](https://discuss.elastic.co/t/add-new-line-character-ruby-filter/140486)
**Author:** [@saramali](https://discuss.elastic.co/u/saramali)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 8:47am UTC](https://discuss.elastic.co/t/add-new-line-character-ruby-filter/140486 "2018-07-18T08:47:44Z")
I am trying to add a newline character in ruby filter. But when I add '\\n' it adds '\\\\n' and it is not taken as a new line character. --- ## [Syslog input, open only udp port](https://discuss.elastic.co/t/syslog-input-open-only-udp-port/134675)
**Author:** [@Sebastian\_Herrera](https://discuss.elastic.co/u/Sebastian_Herrera)\ **Replies:** 1\ **Last updated:** [June 28, 2018, 9:06am UTC](https://discuss.elastic.co/t/syslog-input-open-only-udp-port/134675 "2018-06-28T09:06:37Z")
Hello, i want to know if there is any possibility to open only the udp port for syslog. Now when i configure it open both tcp and udp. Regards --- ## [LogStash not sending some data to ElasticSearch](https://discuss.elastic.co/t/logstash-not-sending-some-data-to-elasticsearch/140496)
**Author:** [@Ragnaruk](https://discuss.elastic.co/u/Ragnaruk)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 9:34am UTC](https://discuss.elastic.co/t/logstash-not-sending-some-data-to-elasticsearch/140496 "2018-07-18T09:34:51Z")
Hello. I was running ELK Stack (6.2.3) with MetricBeat (6.2.4) installed on several Windows servers and there were no problems. Then, a need to monitor a Linux server (Red Hat Enterprise Linux 7.4) appeared: I installed… --- ## [Month names in english with pattern MMM](https://discuss.elastic.co/t/month-names-in-english-with-pattern-mmm/139736)
**Author:** [@Upsilon](https://discuss.elastic.co/u/Upsilon)\ **Replies:** 2\ **Last updated:** [July 18, 2018, 9:23am UTC](https://discuss.elastic.co/t/month-names-in-english-with-pattern-mmm/139736 "2018-07-18T09:23:07Z")
Hello, The @timestamp format I have is the following : « 2018-07-10T11:04:14.452Z ». And I would like to have in output this format : « Jul 10 13:09:12 ». Here is my configuration in order to do that : input { kafka … --- ## [Logstash cannot read /var/log/maillog](https://discuss.elastic.co/t/logstash-cannot-read-var-log-maillog/140331)
**Author:** [@robs](https://discuss.elastic.co/u/robs)\ **Replies:** 3\ **Last updated:** [July 18, 2018, 8:56am UTC](https://discuss.elastic.co/t/logstash-cannot-read-var-log-maillog/140331 "2018-07-18T08:56:07Z")
Hi all, I can't find this problem or a related answer in recent postings (after 2014) so I drop this here. I'm running this: CentOS 6.8 (final) Logstash 6.2.3 Elasticsearch 6.2.3 To allow logstash to read log files … --- ## [Parse Audit file](https://discuss.elastic.co/t/parse-audit-file/140479)
**Author:** [@Serak\_Shiferaw](https://discuss.elastic.co/u/Serak_Shiferaw)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 8:23am UTC](https://discuss.elastic.co/t/parse-audit-file/140479 "2018-07-18T08:23:32Z")
Hi All i wanted a specif patter of filter and i cant get my head arround it because my columns are not uniform for example i have line like Format TimeStamp || BusinessTimeStamp ||RuntimeMicroflowID ||MicroflowID ||U… --- ## [Load balance output between multiple hosts](https://discuss.elastic.co/t/load-balance-output-between-multiple-hosts/139694)
**Author:** [@xShadow](https://discuss.elastic.co/u/xShadow)\ **Replies:** 9\ **Last updated:** [July 18, 2018, 8:26am UTC](https://discuss.elastic.co/t/load-balance-output-between-multiple-hosts/139694 "2018-07-18T08:26:44Z")
Hi, I have a logstash instance with multiple hosts output. According to the documentation, logstash should load balance across the hosts specified. However, I am receiving the logs on both of the elasticsearch instance. … --- ## [Logstash to parse xml file](https://discuss.elastic.co/t/logstash-to-parse-xml-file/140147)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 5\ **Last updated:** [July 18, 2018, 7:45am UTC](https://discuss.elastic.co/t/logstash-to-parse-xml-file/140147 "2018-07-18T07:45:12Z")
I am new on ELK, and I need a help to use Kibana. I have 5 fields; one for the name of data, and for each name there is different type, and for each type there is different value for each object. My xml file is like that… --- ## [How to get shards from elasticsearch to logstash?](https://discuss.elastic.co/t/how-to-get-shards-from-elasticsearch-to-logstash/140449)
**Author:** [@deok](https://discuss.elastic.co/u/deok)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 5:53am UTC](https://discuss.elastic.co/t/how-to-get-shards-from-elasticsearch-to-logstash/140449 "2018-07-18T05:53:26Z")
i need to convert shards(elasticsearch)data to logFile.. how to set logstash config file ? ex) input { elasticsearch { hosts =\> "ip : port" ?? } output{ file{ path =\> \["/~~~path/~.log"\] } plz help me... [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=718) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=720)