# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=461 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 462 --- ## [Consuming 3x 2GB text logs](https://discuss.elastic.co/t/consuming-3x-2gb-text-logs/203739)
**Author:** [@joaociocca](https://discuss.elastic.co/u/joaociocca)\ **Replies:** 7\ **Last updated:** [October 16, 2019, 11:58pm UTC](https://discuss.elastic.co/t/consuming-3x-2gb-text-logs/203739 "2019-10-16T23:58:33Z")
so, I have this huge files. I tried using read mode file input, but nothing happened, I tried piping the content (like I've learned from @dadoonet over here https://www.elastic.co/pt/blog/enriching-your-postal-addresses-… --- ## [Logstash only starts on 127.0.0.1](https://discuss.elastic.co/t/logstash-only-starts-on-127-0-0-1/203716)
**Author:** [@fabricioraphael](https://discuss.elastic.co/u/fabricioraphael)\ **Replies:** 7\ **Last updated:** [October 16, 2019, 9:45pm UTC](https://discuss.elastic.co/t/logstash-only-starts-on-127-0-0-1/203716 "2019-10-16T21:45:32Z")
I'am using Logstash 7.3 installed locally on server with: sudo apt-get update && sudo apt-get install logstash The goal is use filebeat to send logs to Elastic from Logstash but when I run Logstash, they starts on 127.… --- ## [Syslog to filebeat or logstash](https://discuss.elastic.co/t/syslog-to-filebeat-or-logstash/203923)
**Author:** [@arxo](https://discuss.elastic.co/u/arxo)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 7:55pm UTC](https://discuss.elastic.co/t/syslog-to-filebeat-or-logstash/203923 "2019-10-16T19:55:45Z")
Hi everyone, this might be a really obvious Q. If I have data coming in from syslog, if I push it via a filebeat with system plugins (receiving syslog), will it do the same thing as manually parsing values in logstash (g… --- ## [Persistent Queue Monitoring](https://discuss.elastic.co/t/persistent-queue-monitoring/203909)
**Author:** [@rajuleo](https://discuss.elastic.co/u/rajuleo)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 6:31pm UTC](https://discuss.elastic.co/t/persistent-queue-monitoring/203909 "2019-10-16T18:31:05Z")
Hi team, How do we setup monitoring for Persistent queues to monitor stuck messages, Persistent queue status? Also, Is there anyway to automatically ingest the old data in the persistent queue to make sure the old data… --- ## [Loading snort alerts into logstash (GROK PATTERN)](https://discuss.elastic.co/t/loading-snort-alerts-into-logstash-grok-pattern/203889)
**Author:** [@cottoncandyy](https://discuss.elastic.co/u/cottoncandyy)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 4:58pm UTC](https://discuss.elastic.co/t/loading-snort-alerts-into-logstash-grok-pattern/203889 "2019-10-16T16:58:36Z")
hi I am new to the elk stack and im trying to load snort event log into logstash for analysis but i cant seem to figure out the grok pattern. This is a sample data of how the snort alert log looks like, can someone help … --- ## [Datas from Database](https://discuss.elastic.co/t/datas-from-database/203885)
**Author:** [@Rael\_Rodrigues](https://discuss.elastic.co/u/Rael_Rodrigues)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 4:39pm UTC](https://discuss.elastic.co/t/datas-from-database/203885 "2019-10-16T16:39:38Z")
how to get datas from some database? how to configure the jdbc? how to create the config file? --- ## [How to aggregate many events?](https://discuss.elastic.co/t/how-to-aggregate-many-events/203836)
**Author:** [@RickT](https://discuss.elastic.co/u/RickT)\ **Replies:** 4\ **Last updated:** [October 16, 2019, 3:43pm UTC](https://discuss.elastic.co/t/how-to-aggregate-many-events/203836 "2019-10-16T15:43:50Z")
hi, i'm searching how to aggregate 2 or more events issued from a log file. The events have an common id. This is the criteria to aggregate events. I has tried the example of logstash 6.8 documentation, but it doesn't … --- ## [Error while starting logstash netflow module](https://discuss.elastic.co/t/error-while-starting-logstash-netflow-module/202802)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 2\ **Last updated:** [October 16, 2019, 3:17pm UTC](https://discuss.elastic.co/t/error-while-starting-logstash-netflow-module/202802 "2019-10-16T15:17:30Z")
Somehow after updating the logstash-7.3 to 7.4 I am getting error while starting the logstash. I have checked the logs and the error is: \[ERROR\] 2019-10-09 15:04:46.938 \[\ **Author:** [@amclaughlin](https://discuss.elastic.co/u/amclaughlin)\ **Replies:** 2\ **Last updated:** [October 16, 2019, 2:47pm UTC](https://discuss.elastic.co/t/using-kafka-as-a-message-queue-for-logstash/203712 "2019-10-16T14:47:30Z") Hey all, I am currently setting up an environment with Elastic and Logstash 7.3.1 to serve as a baseline SIEM environment. We have Logstash running on the app server as a pivot point and receiver for our beats and syslo… --- ## [Logstash parse Array of objects](https://discuss.elastic.co/t/logstash-parse-array-of-objects/203854)
**Author:** [@johnSph](https://discuss.elastic.co/u/johnSph)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 2:02pm UTC](https://discuss.elastic.co/t/logstash-parse-array-of-objects/203854 "2019-10-16T14:02:53Z")
Hello, I'm having some issues with creating a proper logstash config that could parse an array of objects. I have the example data source as example: \[ { "id": 666, "time": 1561938911000, "view": { "current": { … --- ## [Calculate time difference between two log lines in a file](https://discuss.elastic.co/t/calculate-time-difference-between-two-log-lines-in-a-file/203606)
**Author:** [@basavyr](https://discuss.elastic.co/u/basavyr)\ **Replies:** 3\ **Last updated:** [October 16, 2019, 1:33pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-two-log-lines-in-a-file/203606 "2019-10-16T13:33:06Z")
Hello there, I have a log file (given by the SLURM workload manager) that looks something like this: 2019-07-15 13:26:40 Started - job id: V3gNDm1MD7unkvVVEmSjiJLoABFKDmABFKDmtdGKDmABFKDmN8Erlm, unix user: 45001:45000,… --- ## [Logstash http output not working](https://discuss.elastic.co/t/logstash-http-output-not-working/203822)
**Author:** [@sariel1212](https://discuss.elastic.co/u/sariel1212)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 11:29am UTC](https://discuss.elastic.co/t/logstash-http-output-not-working/203822 "2019-10-16T11:29:41Z")
Hi, I'm trying to forward logs from logstash to Splunk using http, and this is my logstash.conf file: input { file { path =\> "${TEST\_INPUT\_PATH}" start\_position =\> "beginning" } } output { http { http\_method =\>… --- ## [Logstash degraded when jvm heap size is near its limit](https://discuss.elastic.co/t/logstash-degraded-when-jvm-heap-size-is-near-its-limit/203337)
**Author:** [@toosaman](https://discuss.elastic.co/u/toosaman)\ **Replies:** 3\ **Last updated:** [October 16, 2019, 8:52am UTC](https://discuss.elastic.co/t/logstash-degraded-when-jvm-heap-size-is-near-its-limit/203337 "2019-10-16T08:52:49Z")
Hello! I have a logstash performance degradation problem. It happens when jvm heap memory size is near its upper level. Events received/emitted rate drops sharply and latency increases dramatically. Only after service r… --- ## [Tags in logstash](https://discuss.elastic.co/t/tags-in-logstash/203782)
**Author:** [@beren](https://discuss.elastic.co/u/beren)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 8:37am UTC](https://discuss.elastic.co/t/tags-in-logstash/203782 "2019-10-16T08:37:06Z")
Hi How to direct postfix logs to index postfix ? In logstash config input { beats { port =\> 5044 } } filter { grok { } } output { if "postfix" in \[tags\]{ elasticsearch { hosts … --- ## [Logstash using wrong version of installed plugin](https://discuss.elastic.co/t/logstash-using-wrong-version-of-installed-plugin/203774)
**Author:** [@Pheebzer](https://discuss.elastic.co/u/Pheebzer)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 8:15am UTC](https://discuss.elastic.co/t/logstash-using-wrong-version-of-installed-plugin/203774 "2019-10-16T08:15:11Z")
Hello. I have been trying to move data from DocumentDB \> Elasticsearch. I have multiple pipelines, and the Logstash is being ran in a docker container. All of these components are being hosted on AWS. My problem is wit… --- ## [Logstash Error on Pipelines settings](https://discuss.elastic.co/t/logstash-error-on-pipelines-settings/203767)
**Author:** [@skyluke.1987](https://discuss.elastic.co/u/skyluke.1987)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 7:46am UTC](https://discuss.elastic.co/t/logstash-error-on-pipelines-settings/203767 "2019-10-16T07:46:16Z")
Version: Logstash 7.3.1 Issue: \[2019-10-16T15:42:09,070\]\[ERROR\]\[org.logstash.Logstash \] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit \[2019-10-16T15:42:34,593\]\[… --- ## [How to split one hit in kibana ( single field data) to diiferent hits for a json data?](https://discuss.elastic.co/t/how-to-split-one-hit-in-kibana-single-field-data-to-diiferent-hits-for-a-json-data/203562)
**Author:** [@Muhammed\_Danish](https://discuss.elastic.co/u/Muhammed_Danish)\ **Replies:** 2\ **Last updated:** [October 16, 2019, 6:01am UTC](https://discuss.elastic.co/t/how-to-split-one-hit-in-kibana-single-field-data-to-diiferent-hits-for-a-json-data/203562 "2019-10-16T06:01:21Z")
Hi all, I have imported the json data in postgresql to elasticsearch using jdbc driver in logstash. But it is viewed as a single hit in kibana (as one filed data) and I can't create visualization using that. So, I need t… --- ## [Apache Pulsar input output plugin](https://discuss.elastic.co/t/apache-pulsar-input-output-plugin/201058)
**Author:** [@Lucas\_Dehandschutter](https://discuss.elastic.co/u/Lucas_Dehandschutter)\ **Replies:** 1\ **Last updated:** [October 16, 2019, 5:00am UTC](https://discuss.elastic.co/t/apache-pulsar-input-output-plugin/201058 "2019-10-16T05:00:16Z")
Hello, Are there any plan for a apache pulsar Input / output plugin? Best regards --- ## [How to create multiple index logstash](https://discuss.elastic.co/t/how-to-create-multiple-index-logstash/203691)
**Author:** [@TheSun](https://discuss.elastic.co/u/TheSun)\ **Replies:** 5\ **Last updated:** [October 16, 2019, 4:49am UTC](https://discuss.elastic.co/t/how-to-create-multiple-index-logstash/203691 "2019-10-16T04:49:09Z")
Hi! I am handling pfsense data with logstash but having problems with indexing. This is my configuration. I have 2 pfsense. 01-inputpf1.conf input { tcp { type =\> "syslog1" port =\> 5140 } } input { udp … --- ## [LogStash encoding Issue from Filebeat IIS Access Logs 7.4.0 Stack](https://discuss.elastic.co/t/logstash-encoding-issue-from-filebeat-iis-access-logs-7-4-0-stack/203721)
**Author:** [@Nicholas\_Penning](https://discuss.elastic.co/u/Nicholas_Penning)\ **Replies:** 5\ **Last updated:** [October 16, 2019, 3:24am UTC](https://discuss.elastic.co/t/logstash-encoding-issue-from-filebeat-iis-access-logs-7-4-0-stack/203721 "2019-10-16T03:24:05Z")
I have tried many ways to get this warning message to go away in my LogStash pipeline but am failing to do so: \[2019-10-15T17:42:15,175\]\[WARN \]\[logstash.filters.urldecode\] Received an event that has a different characte… --- ## [Multiple csv files to multiple auto field in kibana](https://discuss.elastic.co/t/multiple-csv-files-to-multiple-auto-field-in-kibana/203731)
**Author:** [@Pandikannan\_Jeyaprak](https://discuss.elastic.co/u/Pandikannan_Jeyaprak)\ **Replies:** 0\ **Last updated:** [October 16, 2019, 2:32am UTC](https://discuss.elastic.co/t/multiple-csv-files-to-multiple-auto-field-in-kibana/203731 "2019-10-16T02:32:03Z")
i am having multiple csv files with different number of column. I have used csv filter on logstash. it is created autodetect column for first csv file only. For other csv files, column names are not detecting. is there a… --- ## [How to create multiple indexes from fields of JSON file from logstach](https://discuss.elastic.co/t/how-to-create-multiple-indexes-from-fields-of-json-file-from-logstach/203558)
**Author:** [@jaytran](https://discuss.elastic.co/u/jaytran)\ **Replies:** 3\ **Last updated:** [October 16, 2019, 2:02am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indexes-from-fields-of-json-file-from-logstach/203558 "2019-10-16T02:02:10Z")
hi all, I'm a newone, I don't know how to create multiple index from fields in json. I have JSON File like this { "id": "135569", "name" : "jay", "year":2016 , "companies": \["IBM", "Google"\] } { "id": "135570", "name… --- ## [Aggregate Plugin push\_previous\_map\_as\_event](https://discuss.elastic.co/t/aggregate-plugin-push-previous-map-as-event/203701)
**Author:** [@Guanda\_Li](https://discuss.elastic.co/u/Guanda_Li)\ **Replies:** 4\ **Last updated:** [October 16, 2019, 1:18am UTC](https://discuss.elastic.co/t/aggregate-plugin-push-previous-map-as-event/203701 "2019-10-16T01:18:28Z")
I'm aggregating streaming connection log from nginx server. Usually there are several similar events coming with same id, then there are another bunch of similar events with another id coming. I'm trying to get the numbe… --- ## [Parsing array of json objects with logstash and injesting to elastic](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197)
**Author:** [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\ **Replies:** 8\ **Last updated:** [October 15, 2019, 11:49pm UTC](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197 "2019-10-15T23:49:24Z")
Hi, I am trying to injest data from logstash to elastic, and have this array of json objects, such that each element in the array is a doc in elasticsearch, with the key name as the keys in the json. \[ {"name"… --- ## [Logstash - Filtering multiple date formats - date\_time\_parse\_exception](https://discuss.elastic.co/t/logstash-filtering-multiple-date-formats-date-time-parse-exception/203725)
**Author:** [@Sidharth\_Sinha](https://discuss.elastic.co/u/Sidharth_Sinha)\ **Replies:** 0\ **Last updated:** [October 15, 2019, 11:48pm UTC](https://discuss.elastic.co/t/logstash-filtering-multiple-date-formats-date-time-parse-exception/203725 "2019-10-15T23:48:14Z")
I have a stream of json logs being sent into elastic via logstash. The input stream has inconsistent date formats, in one of the either formats: "YYYY-MM-dd'T'HH:mm:ssZ", or "YYYY-MM-dd HH:mm:ss" Example, some of them… --- ## [Parsing multiline unstructured log..Please give grok for this below log\\](https://discuss.elastic.co/t/parsing-multiline-unstructured-log-please-give-grok-for-this-below-log/203224)
**Author:** [@smr](https://discuss.elastic.co/u/smr)\ **Replies:** 5\ **Last updated:** [October 15, 2019, 9:58pm UTC](https://discuss.elastic.co/t/parsing-multiline-unstructured-log-please-give-grok-for-this-below-log/203224 "2019-10-15T21:58:00Z")
INFO 2019-04-23 01:57:00,258 \[AsyncHttpClient-Callback\] org.mule.runtime.core.internal.util.queue.QueueXaResourceManager: Starting ResourceManager INFO 2019-04-23 01:57:00,258 \[AsyncHttpClient-Callback\] org.mule.runti… --- ## [Elasticsearch input plugin creates more documents than what is in the originating index](https://discuss.elastic.co/t/elasticsearch-input-plugin-creates-more-documents-than-what-is-in-the-originating-index/203374)
**Author:** [@snowboardcto](https://discuss.elastic.co/u/snowboardcto)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 9:41pm UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin-creates-more-documents-than-what-is-in-the-originating-index/203374 "2019-10-15T21:41:28Z")
The elasticsearch Logstash input filter seems to output more documents than the index it is set to input from. Below is the filter I'm using. What do I need to do to ensure it only inputs each record in the originating… --- ## [Ingesting a separate text file in addition to the log files that are already being ingested](https://discuss.elastic.co/t/ingesting-a-separate-text-file-in-addition-to-the-log-files-that-are-already-being-ingested/203710)
**Author:** [@druffner](https://discuss.elastic.co/u/druffner)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 9:30pm UTC](https://discuss.elastic.co/t/ingesting-a-separate-text-file-in-addition-to-the-log-files-that-are-already-being-ingested/203710 "2019-10-15T21:30:39Z")
Hey Everyone, Please forgive me as I am still very new to the ELK stack. I currently have an on-premises ELK setup that is ingesting snort logs. Running on the same server, I have a separate script that outputs results … --- ## [Parse customer name](https://discuss.elastic.co/t/parse-customer-name/203556)
**Author:** [@sud0](https://discuss.elastic.co/u/sud0)\ **Replies:** 2\ **Last updated:** [October 15, 2019, 9:26pm UTC](https://discuss.elastic.co/t/parse-customer-name/203556 "2019-10-15T21:26:59Z")
Hi. I need to parse the customer name, but there are cases when the customer name has one, two or even three words. Example 1: Customer name with 3 words INFO 2019-10-15 16:34:41,964 transaction.TransactionDocumentPer… --- ## [Logstash RabbitMQ Input - Send ack after event processed](https://discuss.elastic.co/t/logstash-rabbitmq-input-send-ack-after-event-processed/203709)
**Author:** [@ThiagoBarradas](https://discuss.elastic.co/u/ThiagoBarradas)\ **Replies:** 1\ **Last updated:** [October 15, 2019, 8:03pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-send-ack-after-event-processed/203709 "2019-10-15T20:03:14Z")
Hello guys! I need to send rabbitmq ack after event processed. I tried set ack =\> true but acknowledges are sent before then event processing. How can I do this? input { rabbit { # my rabbit config … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=460) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=462)