# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1122 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1123 --- ## [Newbie: Grok Pattern for Cisco FirePower Conf file](https://discuss.elastic.co/t/newbie-grok-pattern-for-cisco-firepower-conf-file/56671)
**Author:** [@Tyler\_Siemers](https://discuss.elastic.co/u/Tyler_Siemers)\ **Replies:** 3\ **Last updated:** [July 29, 2016, 12:16pm UTC](https://discuss.elastic.co/t/newbie-grok-pattern-for-cisco-firepower-conf-file/56671 "2016-07-29T12:16:46Z")
I am trying to setup a basic test grok filter to grab syslog messages from "Cisco Sourcefire". To start I am only grabbing logs related to blocked events. Here is a sample log message: Jul 28 18:52:51 CentralFP1 URLb… --- ## [Logs from linux machine is unreadable](https://discuss.elastic.co/t/logs-from-linux-machine-is-unreadable/56722)
**Author:** [@babeesh](https://discuss.elastic.co/u/babeesh)\ **Replies:** 3\ **Last updated:** [July 29, 2016, 12:01pm UTC](https://discuss.elastic.co/t/logs-from-linux-machine-is-unreadable/56722 "2016-07-29T12:01:38Z")
I have logserver running on ubuntu 14.04. ELK versions are 2 or above(kibana 4.4.x). I am also using nagios to monitor systems and applications. Now I need to sent error messages from nagios to logserver. Nagios is in… --- ## [MySQL to ElasticSearch One DB multiple tables](https://discuss.elastic.co/t/mysql-to-elasticsearch-one-db-multiple-tables/56228)
**Author:** [@westealinfo](https://discuss.elastic.co/u/westealinfo)\ **Replies:** 2\ **Last updated:** [July 29, 2016, 10:50am UTC](https://discuss.elastic.co/t/mysql-to-elasticsearch-one-db-multiple-tables/56228 "2016-07-29T10:50:57Z")
Hello, I've just started using Elasticsearch. I have installed the JDBC Input Plugin to transfer data from our MySQL database to Elasticsearch. However, we have 200 tables in our database. We were wondering if there wa… --- ## [Http output - automatic\_retries retry period](https://discuss.elastic.co/t/http-output-automatic-retries-retry-period/56182)
**Author:** [@auerbecktj](https://discuss.elastic.co/u/auerbecktj)\ **Replies:** 1\ **Last updated:** [July 29, 2016, 10:48am UTC](https://discuss.elastic.co/t/http-output-automatic-retries-retry-period/56182 "2016-07-29T10:48:54Z")
I'm currently using the http output plugin in order to post to an API. I noticed that the default for automatic\_retries is just 1. We essentially want this to keep retrying until it succeeds, but I don't see a straightfo… --- ## [For straming data what should be configuration](https://discuss.elastic.co/t/for-straming-data-what-should-be-configuration/56136)
**Author:** [@Amit2015](https://discuss.elastic.co/u/Amit2015)\ **Replies:** 1\ **Last updated:** [July 29, 2016, 10:28am UTC](https://discuss.elastic.co/t/for-straming-data-what-should-be-configuration/56136 "2016-07-29T10:28:27Z")
Hi, I have tried batch processing using logstach and indexed data into Elasticsearch (from text file) . If I want to realtime streaming data index into ES what should I do? Please suggest. --- ## [Combine events via multiline?](https://discuss.elastic.co/t/combine-events-via-multiline/56717)
**Author:** [@raffis](https://discuss.elastic.co/u/raffis)\ **Replies:** 0\ **Last updated:** [July 29, 2016, 10:19am UTC](https://discuss.elastic.co/t/combine-events-via-multiline/56717 "2016-07-29T10:19:32Z")
Im trying to combine the following two log lines: \<133\>Jul 27 14:36:19 mta.local amavis\[6318\]: (06318-14) Passed CLEAN {RelayedOutbound}, ORIGINATING LOCAL \[1.1.1.1\]:42718 \ -\> \,\ **Author:** [@danielw](https://discuss.elastic.co/u/danielw)\ **Replies:** 2\ **Last updated:** [July 29, 2016, 8:46am UTC](https://discuss.elastic.co/t/grokfilter-for-json-field-not-working/56703 "2016-07-29T08:46:44Z") Hello, i'm a beginner of Logstash and Grok so please tell me if i have missunderstood a basic concept, there is no stupid answer even if it sounds too easy and obvious :slight\_smile: I got a working config for IIS Logs… --- ## [Regarding Pipeline Batch Sizing](https://discuss.elastic.co/t/regarding-pipeline-batch-sizing/56705)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [July 29, 2016, 8:22am UTC](https://discuss.elastic.co/t/regarding-pipeline-batch-sizing/56705 "2016-07-29T08:22:17Z")
Hi, We use beats to send most of our logs to logstash with the following parameters: bulk\_size: 2048 load\_balanced: yes workers: number of logstashes pipelining: 2 In logstash we have 8 worker threads with 2 cores a… --- ## [Help with LS stoping 30 seconds after start](https://discuss.elastic.co/t/help-with-ls-stoping-30-seconds-after-start/56513)
**Author:** [@dustin.liddick](https://discuss.elastic.co/u/dustin.liddick)\ **Replies:** 4\ **Last updated:** [July 29, 2016, 12:08am UTC](https://discuss.elastic.co/t/help-with-ls-stoping-30-seconds-after-start/56513 "2016-07-29T00:08:42Z")
Hello: My LS instance starts up just fine, and ~30-120 seconds into the initialization, logstash then stops. I will paste the error below. The settings that I have for my LS indexing node is VM - Red Hat 6.5 x64 2xvC… --- ## [Novice:something wrong with configuration error](https://discuss.elastic.co/t/novice-something-wrong-with-configuration-error/56614)
**Author:** [@Ankush\_Mehta](https://discuss.elastic.co/u/Ankush_Mehta)\ **Replies:** 2\ **Last updated:** [July 28, 2016, 8:03pm UTC](https://discuss.elastic.co/t/novice-something-wrong-with-configuration-error/56614 "2016-07-28T20:03:33Z")
I am a beginner in ELK and this is my first try in running logstash.I followed the following step: 1) Created a config file(ANSI encoding) in my local C:\\logstash-1.5.2\\conf\\logstash\_test.config.The contents are as below… --- ## [Logstash input s3 plugin](https://discuss.elastic.co/t/logstash-input-s3-plugin/56667)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 0\ **Last updated:** [July 28, 2016, 7:49pm UTC](https://discuss.elastic.co/t/logstash-input-s3-plugin/56667 "2016-07-28T19:49:40Z")
Any idea why am i seeing the below error on using the s3 plugin. A plugin had an unrecoverable error. Will restart this plugin. Plugin: "s3-access-log", bucket=\>"niraj-s3-log", prefix=\>"logs", region=\>"us-east-1", acce… --- ## [Prefix a field in logstash based on size/length of field](https://discuss.elastic.co/t/prefix-a-field-in-logstash-based-on-size-length-of-field/56664)
**Author:** [@nkapur](https://discuss.elastic.co/u/nkapur)\ **Replies:** 0\ **Last updated:** [July 28, 2016, 7:18pm UTC](https://discuss.elastic.co/t/prefix-a-field-in-logstash-based-on-size-length-of-field/56664 "2016-07-28T19:18:59Z")
Hello! I have a field (string) called clubcode. Valid incoming values are say, 014, 14, 22, 022, 111. I want to replace 14 with 014 and 22 with 022 so that I can aggregate using clubcode as term in Kibana. How do I achie… --- ## [Logstash not ingesting when new file added to the directory](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 16\ **Last updated:** [July 28, 2016, 6:00pm UTC](https://discuss.elastic.co/t/logstash-not-ingesting-when-new-file-added-to-the-directory/56206 "2016-07-28T18:00:16Z")
I have a logstash configured to ingest S3 access logs , i can successfully ingest data to Elasticsearch when i restart the logstash without any issues. But when new files gets added to the directory from where the Logsta… --- ## [Help with storing logs on e:\\](https://discuss.elastic.co/t/help-with-storing-logs-on-e/56401)
**Author:** [@deepbrah](https://discuss.elastic.co/u/deepbrah)\ **Replies:** 7\ **Last updated:** [July 28, 2016, 4:29pm UTC](https://discuss.elastic.co/t/help-with-storing-logs-on-e/56401 "2016-07-28T16:29:02Z")
Hey all...Network guy here...I've got the ELK stack up and running on my test environment and everything is fine... I've got the go ahead to put it in production -- where I have a 100GB HD for the OS and the services, … --- ## [How do I match a newline in grok/logstash](https://discuss.elastic.co/t/how-do-i-match-a-newline-in-grok-logstash/56339)
**Author:** [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\ **Replies:** 12\ **Last updated:** [July 28, 2016, 2:49pm UTC](https://discuss.elastic.co/t/how-do-i-match-a-newline-in-grok-logstash/56339 "2016-07-28T14:49:18Z")
I am using Logstash to parse and filter the data. The input data looks something like: \> Tue Apr 05 01:33:13 EDT 2016 r/s w/s cache free\_mem used\_mem swap\_mem page faults id wa 0 0 0 7535996 72612 232184… --- ## [@timestamp field not reflected after resetting server time!](https://discuss.elastic.co/t/timestamp-field-not-reflected-after-resetting-server-time/56179)
**Author:** [@gugansankar](https://discuss.elastic.co/u/gugansankar)\ **Replies:** 4\ **Last updated:** [July 28, 2016, 1:24pm UTC](https://discuss.elastic.co/t/timestamp-field-not-reflected-after-resetting-server-time/56179 "2016-07-28T13:24:46Z")
I'm running both Logstash server and elasticsearch in same machine. After parsing the logs into ELS , I checked it through kibana where the data shown with 5 minutes delay. Then I found that the server time \[ NTP\] is … --- ## [Filebeat config file error](https://discuss.elastic.co/t/filebeat-config-file-error/56573)
**Author:** [@yomesky20000](https://discuss.elastic.co/u/yomesky20000)\ **Replies:** 2\ **Last updated:** [July 28, 2016, 12:09pm UTC](https://discuss.elastic.co/t/filebeat-config-file-error/56573 "2016-07-28T12:09:34Z")
I am trying to start Filebeats on a clients server to ship log to my ELK server and im getting the below error service filebeat start Starting filebeat: Loading config file error: YAML config parsing failed on /etc/f… --- ## [How to compare double type filed in logstash output](https://discuss.elastic.co/t/how-to-compare-double-type-filed-in-logstash-output/56519)
**Author:** [@vandana\_sethi](https://discuss.elastic.co/u/vandana_sethi)\ **Replies:** 7\ **Last updated:** [July 28, 2016, 8:44am UTC](https://discuss.elastic.co/t/how-to-compare-double-type-filed-in-logstash-output/56519 "2016-07-28T08:44:33Z")
I want to compare two fields both are double type which I checked using this url http://localhost:9200/logstash-\*/\_mapping?pretty and I wrote like this in my logstash config file if \[field1\] \> \[field2\] { email { … --- ## [Translate filter](https://discuss.elastic.co/t/translate-filter/56359)
**Author:** [@Kishore](https://discuss.elastic.co/u/Kishore)\ **Replies:** 6\ **Last updated:** [July 28, 2016, 4:59am UTC](https://discuss.elastic.co/t/translate-filter/56359 "2016-07-28T04:59:59Z")
how to match multiple fields using translate filter? --- ## [Output pretty formatted json of all event fields in mail](https://discuss.elastic.co/t/output-pretty-formatted-json-of-all-event-fields-in-mail/51161)
**Author:** [@MartinMystikJonas](https://discuss.elastic.co/u/MartinMystikJonas)\ **Replies:** 1\ **Last updated:** [July 27, 2016, 9:42pm UTC](https://discuss.elastic.co/t/output-pretty-formatted-json-of-all-event-fields-in-mail/51161 "2016-07-27T21:42:59Z")
I'm want to send alert e-mails for certain log events and I want to print all event fields in e-mail body. Bud I did not found way how to output them in pretty formatted json. Currently i use this: filter { if … --- ## [Netflow V9 : message=\>"No matching template for flow id 279"](https://discuss.elastic.co/t/netflow-v9-message-no-matching-template-for-flow-id-279/56269)
**Author:** [@solidz](https://discuss.elastic.co/u/solidz)\ **Replies:** 1\ **Last updated:** [July 27, 2016, 8:47pm UTC](https://discuss.elastic.co/t/netflow-v9-message-no-matching-template-for-flow-id-279/56269 "2016-07-27T20:47:00Z")
Hi, I have configured Netflow V9 on my Cisco Router and I have installed Logstash 2.3.4 on my CentOS Server. I know that my Netflow configuration on my router is correct because I started a Wireshark capture and result… --- ## [Losing messages during high traffic rate](https://discuss.elastic.co/t/losing-messages-during-high-traffic-rate/56449)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 4\ **Last updated:** [July 27, 2016, 7:12pm UTC](https://discuss.elastic.co/t/losing-messages-during-high-traffic-rate/56449 "2016-07-27T19:12:08Z")
I am doing log analysis using Filebeat (1.2) -\> logstash(2.3) -\> Elasticsearch (2.3) I have 4 filebeat instances, 4 logstash instances (6 cores each) , and Elasticsearch cluster (8 cores, 64G RAM) of 2 nodes In Filebea… --- ## [Worker setting of elasticsearch output plugin](https://discuss.elastic.co/t/worker-setting-of-elasticsearch-output-plugin/56020)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 1\ **Last updated:** [July 27, 2016, 6:32pm UTC](https://discuss.elastic.co/t/worker-setting-of-elasticsearch-output-plugin/56020 "2016-07-27T18:32:55Z")
I have a question about the number of pipeline workers and elasticsearch output workers. By default, logstash pipeline will initiate workers based on the number of cores of the server. If I have 6 cores in the server,… --- ## [Please help. I'm not getting how to input IIS logs](https://discuss.elastic.co/t/please-help-im-not-getting-how-to-input-iis-logs/56197)
**Author:** [@George\_Nussbaum](https://discuss.elastic.co/u/George_Nussbaum)\ **Replies:** 9\ **Last updated:** [July 27, 2016, 5:59pm UTC](https://discuss.elastic.co/t/please-help-im-not-getting-how-to-input-iis-logs/56197 "2016-07-27T17:59:25Z")
Hello, I installed ELK a couple weeks ago and I'm pulling what I have left of my haIr out. I'm having trouble with getting IIS log data to show up. I have filebeat installed on the clients and am using IIS Advanced l… --- ## [Logstash --configtest & stdout](https://discuss.elastic.co/t/logstash-configtest-stdout/56548)
**Author:** [@franck.lefebure](https://discuss.elastic.co/u/franck.lefebure)\ **Replies:** 3\ **Last updated:** [July 27, 2016, 5:49pm UTC](https://discuss.elastic.co/t/logstash-configtest-stdout/56548 "2016-07-27T17:49:24Z")
Hi, I have an annoying little problem on a logstash 2.3.4 installed (from the repo) on a Linux Mint development box May be this problem is more linux oriented. I would like to catch the result of a configtest command … --- ## [Incremental Grok Works, but logstash grok filter fails to parse simple literal string](https://discuss.elastic.co/t/incremental-grok-works-but-logstash-grok-filter-fails-to-parse-simple-literal-string/56503)
**Author:** [@darioc](https://discuss.elastic.co/u/darioc)\ **Replies:** 2\ **Last updated:** [July 27, 2016, 3:34pm UTC](https://discuss.elastic.co/t/incremental-grok-works-but-logstash-grok-filter-fails-to-parse-simple-literal-string/56503 "2016-07-27T15:34:19Z")
Hello, Trying to extract as many fields from my log messages, but running into some basic issues. I have a log message like the following: INFO 2016-07-21 13:17:48,139 \[http-bio-8080-exec-5\] com.vendor.recserver.c… --- ## [Need Help Creating Logstash Filter to use timestamp in log message in Elastic/Kibana](https://discuss.elastic.co/t/need-help-creating-logstash-filter-to-use-timestamp-in-log-message-in-elastic-kibana/56502)
**Author:** [@darioc](https://discuss.elastic.co/u/darioc)\ **Replies:** 2\ **Last updated:** [July 27, 2016, 3:30pm UTC](https://discuss.elastic.co/t/need-help-creating-logstash-filter-to-use-timestamp-in-log-message-in-elastic-kibana/56502 "2016-07-27T15:30:16Z")
Hello, Okay, I'm probably missing something pretty obvious. I have a log message like the following: INFO 2016-07-21 13:17:48,139 \[http-bio-8080-exec-5\] com.vendor.recserver.controller.RestController - site:compan… --- ## [Logstash setup for syslog udp input](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810)
**Author:** [@panopt1k](https://discuss.elastic.co/u/panopt1k)\ **Replies:** 2\ **Last updated:** [July 27, 2016, 2:52pm UTC](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810 "2016-07-27T14:52:51Z")
Hi guys! I need your help in advanced setting up for ELK server. I have installed ELK stack into Ubuntu 14.04 Basically I setup logstash server with filebeats and successfully configured logstash filter for parsing lo… --- ## [Filebeat is not sending logs to logstash](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash/56249)
**Author:** [@Saurabh\_Jambhule](https://discuss.elastic.co/u/Saurabh_Jambhule)\ **Replies:** 10\ **Last updated:** [July 27, 2016, 1:54pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-logs-to-logstash/56249 "2016-07-27T13:54:33Z")
There are some sample apache logs in folder listed in paths, but still they not going to logstash Here is my configuration, filebeat: filebeat: prospectors: - paths: - /home/saurabh/sample-log/\*.l… --- ## [Logstash sending the complete file when new data is added to the log file](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629)
**Author:** [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\ **Replies:** 6\ **Last updated:** [July 27, 2016, 1:48pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629 "2016-07-27T13:48:57Z")
Hi, Given the log file (apache.log) as input to Logstash, after applying some filter the output data is stored in Elasticsearch. Here, whenever the new data is added to the log file, logstash is sending the complete fi… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1121) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1123)