# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=653 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 654 --- ## [Syslog output plugin to slow syslog server is causing delays for our whole pipeline. Better way to set this up?](https://discuss.elastic.co/t/syslog-output-plugin-to-slow-syslog-server-is-causing-delays-for-our-whole-pipeline-better-way-to-set-this-up/155629)
**Author:** [@dfinn](https://discuss.elastic.co/u/dfinn)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 10:44pm UTC](https://discuss.elastic.co/t/syslog-output-plugin-to-slow-syslog-server-is-causing-delays-for-our-whole-pipeline-better-way-to-set-this-up/155629 "2018-11-06T22:44:56Z")
Our pipeline is pretty straightforward. We have filebeats on all our servers which ship logs to a single server running ELK. Logstash processes all the logs, outputs them to Elasticsearch and then right at the end of t… --- ## [Logstash Dissect Filter - Multiple Delimiters?](https://discuss.elastic.co/t/logstash-dissect-filter-multiple-delimiters/155626)
**Author:** [@boss](https://discuss.elastic.co/u/boss)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 9:38pm UTC](https://discuss.elastic.co/t/logstash-dissect-filter-multiple-delimiters/155626 "2018-11-06T21:38:46Z")
Hi all, what is the the best way to support multiple delimiters with the Logstash Dissect Filter? As an example, suppose one file has logs set up as follows: 2018-11-06\_05:30:21 Log message1 And another file has logs … --- ## [Multiline flat json log file to nested json in Elasticsearch](https://discuss.elastic.co/t/multiline-flat-json-log-file-to-nested-json-in-elasticsearch/155625)
**Author:** [@psk](https://discuss.elastic.co/u/psk)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 8:59pm UTC](https://discuss.elastic.co/t/multiline-flat-json-log-file-to-nested-json-in-elasticsearch/155625 "2018-11-06T20:59:46Z")
I have a multiline json log file , which flat .but my template in the Elastic search has nested fields . i am using logstash to send to elastic search . ex my log file {"appname":"test1","level":"1","version":"v1"} {… --- ## [Issues with NMap Codec/NMAP File Ingest](https://discuss.elastic.co/t/issues-with-nmap-codec-nmap-file-ingest/155623)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 8:50pm UTC](https://discuss.elastic.co/t/issues-with-nmap-codec-nmap-file-ingest/155623 "2018-11-06T20:50:03Z")
I have some saved NMAP scans that I am trying to ingest. When using the nmap codec ingesting .nmap files, the below error is generated in the logstash log. Question 1: Can someone explain what the error indicates or w… --- ## [Logstash rsyslogd](https://discuss.elastic.co/t/logstash-rsyslogd/155617)
**Author:** [@David\_Moreno](https://discuss.elastic.co/u/David_Moreno)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 8:39pm UTC](https://discuss.elastic.co/t/logstash-rsyslogd/155617 "2018-11-06T20:39:08Z")
Im trying to set up a log server using the ELK stack. Im working on 3 devices, (A) test server, (B)Rsyslogd server, and (C) ELK server. Im trying to get my logs from my test server(A) logs in my elk server(C), im only g… --- ## [Custom userAgent regexes.yaml](https://discuss.elastic.co/t/custom-useragent-regexes-yaml/155610)
**Author:** [@he11oworld](https://discuss.elastic.co/u/he11oworld)\ **Replies:** 1\ **Last updated:** [November 6, 2018, 8:15pm UTC](https://discuss.elastic.co/t/custom-useragent-regexes-yaml/155610 "2018-11-06T20:15:01Z")
I want to add a small snippet to the default regexes.yaml, but that would mean adding the whole regexes.yaml to configmap, is there a way for userAgent plugin to parse from two different sources?(my custom regex and addi… --- ## [Aggregation feature in logstash](https://discuss.elastic.co/t/aggregation-feature-in-logstash/155613)
**Author:** [@sfunk](https://discuss.elastic.co/u/sfunk)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 8:01pm UTC](https://discuss.elastic.co/t/aggregation-feature-in-logstash/155613 "2018-11-06T20:01:57Z")
Hi, I want to use the aggregation feature in logstash to pull data from a table in Oracle which contains time-series data. I will be querying by row id, and the data in the table is aggregated in one minute chunks alrea… --- ## [Unable to view logs on kibana](https://discuss.elastic.co/t/unable-to-view-logs-on-kibana/155489)
**Author:** [@shruti51](https://discuss.elastic.co/u/shruti51)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 7:08am UTC](https://discuss.elastic.co/t/unable-to-view-logs-on-kibana/155489 "2018-11-06T07:08:18Z")
I am able to parse logs on logstash but the logs are not visible on kibana dashboard. error thrown is: \[2018-11-06T12:29:41,731\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>4… --- ## [ADD new filed to log from previous log](https://discuss.elastic.co/t/add-new-filed-to-log-from-previous-log/155467)
**Author:** [@rpendela](https://discuss.elastic.co/u/rpendela)\ **Replies:** 3\ **Last updated:** [November 6, 2018, 7:35pm UTC](https://discuss.elastic.co/t/add-new-filed-to-log-from-previous-log/155467 "2018-11-06T19:35:48Z")
Hello Community, I have a log file in CEF format and log start and end times as meta data in separate lines, startTime:1541145880148 endTime:1541146222463 CEF:0|Incapsula|SIEMintegration|1|1|Normal|0| sourceServiceName… --- ## [New fields are not added while grok seems fine](https://discuss.elastic.co/t/new-fields-are-not-added-while-grok-seems-fine/155576)
**Author:** [@Yonatan\_Brand](https://discuss.elastic.co/u/Yonatan_Brand)\ **Replies:** 2\ **Last updated:** [November 6, 2018, 7:30pm UTC](https://discuss.elastic.co/t/new-fields-are-not-added-while-grok-seems-fine/155576 "2018-11-06T19:30:21Z")
Hi, I am not able to see new fields I created with grok. I have tried to refresh the indexes, however, I am still not able to see them. Below is my logstash configuration: input { beats { port =\> 5044 host =\>… --- ## [Different index based on match keyword](https://discuss.elastic.co/t/different-index-based-on-match-keyword/155597)
**Author:** [@rrs](https://discuss.elastic.co/u/rrs)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 6:52pm UTC](https://discuss.elastic.co/t/different-index-based-on-match-keyword/155597 "2018-11-06T18:52:27Z")
Hi, As per our architecture our application logs go into /custom\_path/app.log and some into /var/log/messages as well. Our ELK version is 5.6.3. We want ship the application logs store on above both files and segregate… --- ## [Kv filter value is not always correctly extracted using value\_split\_pattern](https://discuss.elastic.co/t/kv-filter-value-is-not-always-correctly-extracted-using-value-split-pattern/154598)
**Author:** [@CaptainAye](https://discuss.elastic.co/u/CaptainAye)\ **Replies:** 4\ **Last updated:** [November 6, 2018, 5:57pm UTC](https://discuss.elastic.co/t/kv-filter-value-is-not-always-correctly-extracted-using-value-split-pattern/154598 "2018-11-06T17:57:02Z")
Hello, I have a problem extracting values using kv filter. I would like to extract the values which are seperated from key by the spaces or by the equal signs. My configuration looks like that: kv { include\_keys =\>… --- ## [Kv filter issue](https://discuss.elastic.co/t/kv-filter-issue/155566)
**Author:** [@lemon](https://discuss.elastic.co/u/lemon)\ **Replies:** 2\ **Last updated:** [November 6, 2018, 4:37pm UTC](https://discuss.elastic.co/t/kv-filter-issue/155566 "2018-11-06T16:37:16Z")
Hi, i'm a beginner with logstash. I use ELK via docker on windows. I parse a csv file which contain a field Metadata. I use kv filter to split values and i want to convert the field kv.Geolocation create by the kv filter… --- ## [Lumberjack certificate verify failed](https://discuss.elastic.co/t/lumberjack-certificate-verify-failed/155546)
**Author:** [@logger](https://discuss.elastic.co/u/logger)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 12:25pm UTC](https://discuss.elastic.co/t/lumberjack-certificate-verify-failed/155546 "2018-11-06T12:25:02Z")
Hi there, I do not have a question, but maybe an answer to other threads. We had issues with lumberjack. We wanted to connect Logstash-to-Logstash and we had issues with our certificates. Everything worked fine with o… --- ## [GeoIP filter - database file updated - restart required?](https://discuss.elastic.co/t/geoip-filter-database-file-updated-restart-required/155200)
**Author:** [@YaoChim](https://discuss.elastic.co/u/YaoChim)\ **Replies:** 1\ **Last updated:** [November 6, 2018, 12:21pm UTC](https://discuss.elastic.co/t/geoip-filter-database-file-updated-restart-required/155200 "2018-11-06T12:21:41Z")
I have the following filter in Logstash 6.4.2 if \[client\_ip\] !~ /^(10\\.|172\\.1\[6-9\]|172\\.2\[0-9\]|172\\.3\[0-1\]|192\\.168|127\\.)/ { geoip { database =\> "/opt/logstash/GeoLite2.mmdb" source =\> "client\_ip" … --- ## [Multiples field does not work](https://discuss.elastic.co/t/multiples-field-does-not-work/155400)
**Author:** [@edoliveira](https://discuss.elastic.co/u/edoliveira)\ **Replies:** 1\ **Last updated:** [November 6, 2018, 11:58am UTC](https://discuss.elastic.co/t/multiples-field-does-not-work/155400 "2018-11-06T11:58:55Z")
I send this log to logstash: 2018/10/29 16:51:14 \[info\] 4344#4344: \*556825 ModSecurity: Warning. Matched "Operator Within' with parameter .asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .co… --- ## [Extracting filename from existing field (Source)?](https://discuss.elastic.co/t/extracting-filename-from-existing-field-source/155521)
**Author:** [@roly](https://discuss.elastic.co/u/roly)\ **Replies:** 11\ **Last updated:** [November 6, 2018, 11:27am UTC](https://discuss.elastic.co/t/extracting-filename-from-existing-field-source/155521 "2018-11-06T11:27:27Z")
how can i extract directory name from the below ? source path = "/var/log/companyname/servicename/jjds/smtgxyz/smtgabc.log" from the above i have to get "servicename" as an another field. can someone please provide the… --- ## [Logstash create doc if not exist else append array](https://discuss.elastic.co/t/logstash-create-doc-if-not-exist-else-append-array/153262)
**Author:** [@idunnomuch](https://discuss.elastic.co/u/idunnomuch)\ **Replies:** 2\ **Last updated:** [November 2, 2018, 9:38am UTC](https://discuss.elastic.co/t/logstash-create-doc-if-not-exist-else-append-array/153262 "2018-11-02T09:38:01Z")
Elasticsearch: 6.4.2 Logstash: 6.4.2 Goal: Index the data if it hasn't been indexed before, otherwise, if the document does exist, then append an array. Input data round 1: { "item": \["1"\] } Input data round 2: { "… --- ## [How to check persistent queue draining?](https://discuss.elastic.co/t/how-to-check-persistent-queue-draining/153794)
**Author:** [@mw\_jko](https://discuss.elastic.co/u/mw_jko)\ **Replies:** 1\ **Last updated:** [November 6, 2018, 8:40am UTC](https://discuss.elastic.co/t/how-to-check-persistent-queue-draining/153794 "2018-11-06T08:40:36Z")
Hi there, we are (still) using ELK stack 5.4 in a very simple singlenode Docker-Compose setup with persistent queue enabled: queue.type: persisted path.queue: /usr/share/logstash/queue queue.max\_bytes: 5gb After readi… --- ## [Logstash not indexing logs when run as service](https://discuss.elastic.co/t/logstash-not-indexing-logs-when-run-as-service/155506)
**Author:** [@Billz1026](https://discuss.elastic.co/u/Billz1026)\ **Replies:** 0\ **Last updated:** [November 6, 2018, 8:32am UTC](https://discuss.elastic.co/t/logstash-not-indexing-logs-when-run-as-service/155506 "2018-11-06T08:32:35Z")
Hi, I have configure ELK stack and ran logstash and when I checked the configuration using below command, it says "configuration OK". When I run the logstash using below command it is indexing logs to Elasticsearch and… --- ## [XML mapping of IDs with Logstash](https://discuss.elastic.co/t/xml-mapping-of-ids-with-logstash/155197)
**Author:** [@ChristianCH](https://discuss.elastic.co/u/ChristianCH)\ **Replies:** 1\ **Last updated:** [November 6, 2018, 6:59am UTC](https://discuss.elastic.co/t/xml-mapping-of-ids-with-logstash/155197 "2018-11-06T06:59:37Z")
Hi all, I want to insert chats into Elasticsearch and have a problem in converting the chat XML transcript to the proper JSON using logstash. This is the XML: \ \ \ **Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 2\ **Last updated:** [November 5, 2018, 8:44pm UTC](https://discuss.elastic.co/t/grok-every-fields-is-putted-into-message/155214 "2018-11-05T20:44:03Z") hi all, I'm sending messages from Filebeat to Logstash where I will used Grok to parse correctly and then send to Elastic. The first step, from FB to LS works correctly. unfortunately, when I try to apply Grok I got a… --- ## [Array Logstash json](https://discuss.elastic.co/t/array-logstash-json/155450)
**Author:** [@rasouzas](https://discuss.elastic.co/u/rasouzas)\ **Replies:** 0\ **Last updated:** [November 5, 2018, 7:35pm UTC](https://discuss.elastic.co/t/array-logstash-json/155450 "2018-11-05T19:35:16Z")
Hello, I have the following input in JSON file: \[\[{"traceId":"ssss","id":"sss","kind":"CLIENT","name":"post","timestamp":1541431973229000,"duration":1309000,"localEndpoint":{"serviceName":"agencias"},"remoteEndpoint":{… --- ## [How to access XML data and use GROK filter on it?](https://discuss.elastic.co/t/how-to-access-xml-data-and-use-grok-filter-on-it/154512)
**Author:** [@Samvid\_Kulkarni](https://discuss.elastic.co/u/Samvid_Kulkarni)\ **Replies:** 2\ **Last updated:** [November 5, 2018, 5:14pm UTC](https://discuss.elastic.co/t/how-to-access-xml-data-and-use-grok-filter-on-it/154512 "2018-11-05T17:14:11Z")
I am trying to parse Microsoft Active Directory which is completely in XML format. Below is the example of the logs. Here is link to log Below is my config file filter { grok { match =\> \[ "message", "%{TIMEST… --- ## [Logstash look for expressions](https://discuss.elastic.co/t/logstash-look-for-expressions/155428)
**Author:** [@Miguel\_Leite](https://discuss.elastic.co/u/Miguel_Leite)\ **Replies:** 0\ **Last updated:** [November 5, 2018, 4:10pm UTC](https://discuss.elastic.co/t/logstash-look-for-expressions/155428 "2018-11-05T16:10:56Z")
Hey there, In my logstash config file I must look for a lot of expressions in order to know if there are errors or possible errors, but trying to look for those expressions with conditionals in my logstash file doesn't … --- ## [Extract some data from an event and send it to a different index](https://discuss.elastic.co/t/extract-some-data-from-an-event-and-send-it-to-a-different-index/154701)
**Author:** [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Replies:** 2\ **Last updated:** [November 5, 2018, 4:09pm UTC](https://discuss.elastic.co/t/extract-some-data-from-an-event-and-send-it-to-a-different-index/154701 "2018-11-05T16:09:36Z")
I have events that look something like this. { "\_index": "my-index", "\_type": "doc", "\_id": "ebfc6jb3fud8gjml2jq9g", "\_version": 1, "\_score": null, "\_source": { "@timestamp": "2018-10-30T14:24:48.670Z", … --- ## [Logstash: Simplest pipeline in the history of mankind not working](https://discuss.elastic.co/t/logstash-simplest-pipeline-in-the-history-of-mankind-not-working/155235)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 3\ **Last updated:** [November 5, 2018, 2:12pm UTC](https://discuss.elastic.co/t/logstash-simplest-pipeline-in-the-history-of-mankind-not-working/155235 "2018-11-05T14:12:01Z")
Using logstash 2.4 (I have my reasons) on Ubuntu 16.04 root@logbox:/etc/logstash/conf.d# ls -al total 16 drwxrwxr-x 2 root root 4096 Nov 2 19:53 . drwxrwxr-x 3 root root 4096 Nov 2 15:46 .. -rwxrwxrwx 1 root root 277… --- ## [Logstash kafka as input](https://discuss.elastic.co/t/logstash-kafka-as-input/153917)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [November 5, 2018, 1:17pm UTC](https://discuss.elastic.co/t/logstash-kafka-as-input/153917 "2018-11-05T13:17:18Z")
Would like to know whether the logstash decompresses the log events while reading the messages from Kafka. --- ## [Should I use LogStash?](https://discuss.elastic.co/t/should-i-use-logstash/155396)
**Author:** [@gnumoksha](https://discuss.elastic.co/u/gnumoksha)\ **Replies:** 0\ **Last updated:** [November 5, 2018, 12:38pm UTC](https://discuss.elastic.co/t/should-i-use-logstash/155396 "2018-11-05T12:38:09Z")
Nowadays I send records to elasticsearch through a logging library (monolog). Should I use LogStash instead? Does it have any advantages? --- ## [How to set input filter logstash?](https://discuss.elastic.co/t/how-to-set-input-filter-logstash/154433)
**Author:** [@rohith3e](https://discuss.elastic.co/u/rohith3e)\ **Replies:** 3\ **Last updated:** [October 31, 2018, 7:08am UTC](https://discuss.elastic.co/t/how-to-set-input-filter-logstash/154433 "2018-10-31T07:08:53Z")
Im bit new to ELK, I need to monitor my application logs(Java tomcat) using ELK, how do I setup a filter in logstash? Should I need to add any this think extra in filebeat.yml other than input path and output hostname an… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=652) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=654)