# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=480 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 481 --- ## [Ruby exception with no message](https://discuss.elastic.co/t/ruby-exception-with-no-message/198887)
**Author:** [@yujin](https://discuss.elastic.co/u/yujin)\ **Replies:** 0\ **Last updated:** [September 10, 2019, 11:13am UTC](https://discuss.elastic.co/t/ruby-exception-with-no-message/198887 "2019-09-10T11:13:23Z")
I was trying to use ruby filter to calculate time difference: ruby{ code =\> "require 'date'; event.set('landing\_delay',(( DateTime.parse(event.get('landing\_t')) - DateTime.parse(event.get('t')) … --- ## [Index naming based on timestamp](https://discuss.elastic.co/t/index-naming-based-on-timestamp/198856)
**Author:** [@Bob94](https://discuss.elastic.co/u/Bob94)\ **Replies:** 0\ **Last updated:** [September 10, 2019, 9:03am UTC](https://discuss.elastic.co/t/index-naming-based-on-timestamp/198856 "2019-09-10T09:03:56Z")
Hi all, Here i can see the beats was succuesfully collect the logs with this naming format and below is how the way its showed in logstash Path =\> C:\\var\\log\\myfolder\\myrecord\_2019-09-09.log My question is how to trea… --- ## [Cloudtrail log to Logstash](https://discuss.elastic.co/t/cloudtrail-log-to-logstash/198684)
**Author:** [@Abdur\_Raqeeb2](https://discuss.elastic.co/u/Abdur_Raqeeb2)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 8:43am UTC](https://discuss.elastic.co/t/cloudtrail-log-to-logstash/198684 "2019-09-10T08:43:04Z")
Hi, I not able to get the index created in elasticsearch for the cloudtrail log parsed from S3.Also im not able to get any Error message on Logstash and Elasticsearch.Logstash gets work fine without throwing any error … --- ## [Regarding to logstash performance limit](https://discuss.elastic.co/t/regarding-to-logstash-performance-limit/197763)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 3\ **Last updated:** [September 10, 2019, 8:30am UTC](https://discuss.elastic.co/t/regarding-to-logstash-performance-limit/197763 "2019-09-10T08:30:41Z")
Hi experts, I have met a logstash performance limit issue, when I parse my log, logstash report this error log: \` \[2019-09-03T14:21:49,106\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearc… --- ## [Http\_poller](https://discuss.elastic.co/t/http-poller/197313)
**Author:** [@ancy](https://discuss.elastic.co/u/ancy)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 7:03am UTC](https://discuss.elastic.co/t/http-poller/197313 "2019-09-10T07:03:29Z")
Trying to get data to elasticsearch from appdynamics api using logstash below configuration is used input { http\_poller { urls =\> { appdynamics =\>{ method =\> "GET" url =\> "https://cognizant-nfr.saas.appdynamics.c… --- ## [Ok to define custom grok patterns in default logstash pattern file?](https://discuss.elastic.co/t/ok-to-define-custom-grok-patterns-in-default-logstash-pattern-file/198789)
**Author:** [@hueyg](https://discuss.elastic.co/u/hueyg)\ **Replies:** 4\ **Last updated:** [September 9, 2019, 9:27pm UTC](https://discuss.elastic.co/t/ok-to-define-custom-grok-patterns-in-default-logstash-pattern-file/198789 "2019-09-09T21:27:07Z")
Is there any problem with defining a custom expression in the already existing pattern file that I believe is read here in version 7.2: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/pat… --- ## [Unable to start logstash: ERROR: Unknown command '�f'](https://discuss.elastic.co/t/unable-to-start-logstash-error-unknown-command-i-f/198783)
**Author:** [@chinmay.parkar](https://discuss.elastic.co/u/chinmay.parkar)\ **Replies:** 4\ **Last updated:** [September 9, 2019, 9:12pm UTC](https://discuss.elastic.co/t/unable-to-start-logstash-error-unknown-command-i-f/198783 "2019-09-09T21:12:54Z")
\[root@localhost logstash-6.3.0\]# ./bin/logstash –f logstash.conf ERROR: Unknown command '�f' See: 'bin/logstash --help' \[ERROR\] 2019-09-09 11:27:30.255 \[main\] Logstash - java.lang.IllegalStateException: Logstash sto… --- ## [Logstash to aggregate logs](https://discuss.elastic.co/t/logstash-to-aggregate-logs/198792)
**Author:** [@rp346](https://discuss.elastic.co/u/rp346)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 9:12pm UTC](https://discuss.elastic.co/t/logstash-to-aggregate-logs/198792 "2019-09-09T21:12:50Z")
I am trying to aggregate raspberry pi logs into Logstash/ElasticSearch running in EKS. This is my manifest file apiVersion: v1 kind: ServiceAccount metadata: name: logstash namespace: kube-logging labels: app… --- ## [Using existing logstash index pattern in Kibana produces 413 Response Error](https://discuss.elastic.co/t/using-existing-logstash-index-pattern-in-kibana-produces-413-response-error/198779)
**Author:** [@swnbdy](https://discuss.elastic.co/u/swnbdy)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 8:07pm UTC](https://discuss.elastic.co/t/using-existing-logstash-index-pattern-in-kibana-produces-413-response-error/198779 "2019-09-09T20:07:22Z")
Hello team, Newbie here. Today I refreshed my index pattern to bring in new fields. After noticing that I had gone from having 18 fields to 2000+ I felt something wasn't as I only expected \>10 fields. Thinking I was r… --- ## [How to read a remote text file using logstash](https://discuss.elastic.co/t/how-to-read-a-remote-text-file-using-logstash/198773)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 1\ **Last updated:** [September 9, 2019, 7:51pm UTC](https://discuss.elastic.co/t/how-to-read-a-remote-text-file-using-logstash/198773 "2019-09-09T19:51:49Z")
hi, i want to know how to read a remote text file using logstash, i suppose that i have a text file in windows machine and my ELK in ubuntu thank you --- ## [Multiply and adding fields in logstsh](https://discuss.elastic.co/t/multiply-and-adding-fields-in-logstsh/198767)
**Author:** [@Andrew22](https://discuss.elastic.co/u/Andrew22)\ **Replies:** 2\ **Last updated:** [September 9, 2019, 7:12pm UTC](https://discuss.elastic.co/t/multiply-and-adding-fields-in-logstsh/198767 "2019-09-09T19:12:03Z")
Hello, I am trying to convert three separate fields for hour minute and second to one in seconds so I have dome some research and found if \[hour\] and \[minute\] and \[second\] { ruby { c… --- ## [Extract specific fields from Json using Logstash](https://discuss.elastic.co/t/extract-specific-fields-from-json-using-logstash/198750)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 3\ **Last updated:** [September 9, 2019, 5:50pm UTC](https://discuss.elastic.co/t/extract-specific-fields-from-json-using-logstash/198750 "2019-09-09T17:50:20Z")
I have this json file { "type": "resource", "headers": { "destination\_channel": "data" }, "body": { "type":"resource", "resourceId":"estimatorB", "resourceType":null, "resourceValue":"data.c… --- ## [Mutate\>Convert and URL Decode](https://discuss.elastic.co/t/mutate-convert-and-url-decode/198745)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 4:38pm UTC](https://discuss.elastic.co/t/mutate-convert-and-url-decode/198745 "2019-09-09T16:38:03Z")
I am ingesting a document that is encoded as UTF-8BOM. This is causing an issue with the urldecode filter, though every other filter (json, mutate, translate, etc...) seems to process it just fine without changing the d… --- ## [How to set License Server](https://discuss.elastic.co/t/how-to-set-license-server/198742)
**Author:** [@jeggy](https://discuss.elastic.co/u/jeggy)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 4:14pm UTC](https://discuss.elastic.co/t/how-to-set-license-server/198742 "2019-09-09T16:14:35Z")
I'm trying to setup a logstash service via docker-swarm. I'm running it like this via a docker-compose file: logstash: image: docker.elastic.co/logstash/logstash:7.3.1 command: logstash -e 'input{ gelf{ use\_t… --- ## [How to get the original message sent to logstash?](https://discuss.elastic.co/t/how-to-get-the-original-message-sent-to-logstash/198736)
**Author:** [@Daniel\_Ben\_Hayoun](https://discuss.elastic.co/u/Daniel_Ben_Hayoun)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 3:46pm UTC](https://discuss.elastic.co/t/how-to-get-the-original-message-sent-to-logstash/198736 "2019-09-09T15:46:34Z")
Hi, I'm dealing with winlogbeat , and want to add to the entire log that I've got to a field called "\[event\]\[original\]" usually I do this mutate { rename =\> { "message" =\> "forensicLog" } } but this time , winlogbeat… --- ## [Problem with logstash DNS resolver plugin](https://discuss.elastic.co/t/problem-with-logstash-dns-resolver-plugin/198447)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 10\ **Last updated:** [September 9, 2019, 2:42pm UTC](https://discuss.elastic.co/t/problem-with-logstash-dns-resolver-plugin/198447 "2019-09-09T14:42:26Z")
So i'm trying out ELK to see if it can compare to an existing product we're using internally and i've hit a problem i can't get around. I have a bunch of IP addresses that i've gotten from a csv file and stripped out ev… --- ## [Logstash not listening on port 5044](https://discuss.elastic.co/t/logstash-not-listening-on-port-5044/198475)
**Author:** [@fastxl](https://discuss.elastic.co/u/fastxl)\ **Replies:** 2\ **Last updated:** [September 9, 2019, 1:53pm UTC](https://discuss.elastic.co/t/logstash-not-listening-on-port-5044/198475 "2019-09-09T13:53:51Z")
I have setup a ELK node that I want to send some logs to via WinLogBeats. Using Ubuntu 18 server. I can ping the node by the ip address and the FQDN, but when I do a Test-NetConnection via powershell it fails. When I do … --- ## [Where to place ssl\_certificate\_verification?](https://discuss.elastic.co/t/where-to-place-ssl-certificate-verification/198700)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 2:03pm UTC](https://discuss.elastic.co/t/where-to-place-ssl-certificate-verification/198700 "2019-09-09T14:03:24Z")
Hi, I am using security module to encrypt transport between es-nodes and to rest api. In logstash I get the following warning: \[2019-09-09T13:56:45,737\]\[WARN \]\[logstash.outputs.elasticsearch\] \*\* WARNING \*\* Detected UN… --- ## [Logstash json-filter - Wildchar to fetch fields](https://discuss.elastic.co/t/logstash-json-filter-wildchar-to-fetch-fields/198597)
**Author:** [@nilesh1](https://discuss.elastic.co/u/nilesh1)\ **Replies:** 4\ **Last updated:** [September 9, 2019, 12:14pm UTC](https://discuss.elastic.co/t/logstash-json-filter-wildchar-to-fetch-fields/198597 "2019-09-09T12:14:08Z")
New to Logstash, I am using logstash JSON filter plugin. JSON payloads look like \[each line new log entry\] { "RequestOne": { "Header": { "MessageID": "41019e6b-e2da-4303-88df-859765d0fbe3","CreationDateTime": "2019-01-… --- ## [Deade letter queue setting create some log files which are not meaningful](https://discuss.elastic.co/t/deade-letter-queue-setting-create-some-log-files-which-are-not-meaningful/198675)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 12:09pm UTC](https://discuss.elastic.co/t/deade-letter-queue-setting-create-some-log-files-which-are-not-meaningful/198675 "2019-09-09T12:09:18Z")
Hi, I am using LS-6.3.1. I enabled dead\_letter\_queue. ###logstash.yml---- path.data: /MyHome/logstash-6.3.1/data Pipelines.yml pipeline name is: my\_pipe After I restart my logstash, it creates files as below. /M… --- ## [I was writing data to influxDB. but... happend problem... Help me please](https://discuss.elastic.co/t/i-was-writing-data-to-influxdb-but-happend-problem-help-me-please/198674)
**Author:** [@ki\_mu](https://discuss.elastic.co/u/ki_mu)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 11:52am UTC](https://discuss.elastic.co/t/i-was-writing-data-to-influxdb-but-happend-problem-help-me-please/198674 "2019-09-09T11:52:28Z")
i insert data to influxDB through logstash. logstash read the log file(json form), insert data to influxDB. i successed so far. but.. after i confirm DB, It happen problem... plz see the picture... i think time tag … --- ## [Logstash does not create logstash-\* indices in ES](https://discuss.elastic.co/t/logstash-does-not-create-logstash-indices-in-es/198667)
**Author:** [@ehsank777](https://discuss.elastic.co/u/ehsank777)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 11:07am UTC](https://discuss.elastic.co/t/logstash-does-not-create-logstash-indices-in-es/198667 "2019-09-09T11:07:41Z")
Hi, Since we upgraded ELK to the version 7.2 , we faced a problem. logstash is creating indices with other names except logstash-yyyy.mm.dd format . In other words within logstash.conf we have output { if "foo" in … --- ## [NEW to ELK: using conditionals in filter](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601)
**Author:** [@hAh0L13](https://discuss.elastic.co/u/hAh0L13)\ **Replies:** 6\ **Last updated:** [September 9, 2019, 10:52am UTC](https://discuss.elastic.co/t/new-to-elk-using-conditionals-in-filter/198601 "2019-09-09T10:52:01Z")
Hello. Stuck in configuring pipeline. My filebeat agent send apache\_access logs with "fields.type = apache\_access" option: filebeat.inputs: - type: log enabled: true paths: - D:\\log\\apache\\access-\*.log … --- ## [Should Logstash crashe due to Invalid FieldReference](https://discuss.elastic.co/t/should-logstash-crashe-due-to-invalid-fieldreference/198642)
**Author:** [@JohanRask](https://discuss.elastic.co/u/JohanRask)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 9:17am UTC](https://discuss.elastic.co/t/should-logstash-crashe-due-to-invalid-fieldreference/198642 "2019-09-09T09:17:32Z")
Hi, We are using logstash 7.1.1 in production with multiple pipelines. In one of our pipelines we use kv filter to convert headers to key value pairs and one of these keys contained an invalid value (invalid value was … --- ## [Read Nifi CSV file in logstash](https://discuss.elastic.co/t/read-nifi-csv-file-in-logstash/198637)
**Author:** [@DG13](https://discuss.elastic.co/u/DG13)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 9:05am UTC](https://discuss.elastic.co/t/read-nifi-csv-file-in-logstash/198637 "2019-09-09T09:05:52Z")
Hi, I have used Nifi to generate the csv file . When I open File generated in nifi in sublime text I could clearly see the data as in new line . But when same file gets input to logstash I cannot recognize the new lin… --- ## [Grok match and move all to subfield](https://discuss.elastic.co/t/grok-match-and-move-all-to-subfield/198633)
**Author:** [@vincentvm](https://discuss.elastic.co/u/vincentvm)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 8:45am UTC](https://discuss.elastic.co/t/grok-match-and-move-all-to-subfield/198633 "2019-09-09T08:45:14Z")
Is it possible to move all fields matched by grok to a subfield? if \[program\] == "nginx-error" { grok { match =\> \[ "message" , "%{NGINX\_ERROR}"\] add\_tag =\> \["nginx-error"\] } } I would like to move this for exam… --- ## [Logstash-plugin command fail](https://discuss.elastic.co/t/logstash-plugin-command-fail/198615)
**Author:** [@111203](https://discuss.elastic.co/u/111203)\ **Replies:** 0\ **Last updated:** [September 9, 2019, 6:49am UTC](https://discuss.elastic.co/t/logstash-plugin-command-fail/198615 "2019-09-09T06:49:51Z")
I ran the two command about logstash-plugin , but all fail, both return following errors, My logstash version is 6.8.1, OS is Win 10 . Does someone knows how to fix it? Plz help, thank you. --- ## [Grok or Dissect?](https://discuss.elastic.co/t/grok-or-dissect/198520)
**Author:** [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Replies:** 2\ **Last updated:** [September 9, 2019, 5:29am UTC](https://discuss.elastic.co/t/grok-or-dissect/198520 "2019-09-09T05:29:37Z")
In my (many, many) filters I use both grok and dissect but I wonder what's recommended? is one faster then the other? --- ## [What is the reason of this issue?](https://discuss.elastic.co/t/what-is-the-reason-of-this-issue/198495)
**Author:** [@zero2](https://discuss.elastic.co/u/zero2)\ **Replies:** 2\ **Last updated:** [September 8, 2019, 4:38am UTC](https://discuss.elastic.co/t/what-is-the-reason-of-this-issue/198495 "2019-09-08T04:38:50Z")
\[2019-09-07T02:32:30,096\]\[INFO \]\[logstash.outputs.elasticsearch\] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>http://192.168.30.31:9200/, :path=\>"/"} \[2019-09-07T02:32:30,099\]… --- ## [How to create a logstash pipeline to create json type log](https://discuss.elastic.co/t/how-to-create-a-logstash-pipeline-to-create-json-type-log/198527)
**Author:** [@Fosiul\_Alam](https://discuss.elastic.co/u/Fosiul_Alam)\ **Replies:** 0\ **Last updated:** [September 7, 2019, 9:59pm UTC](https://discuss.elastic.co/t/how-to-create-a-logstash-pipeline-to-create-json-type-log/198527 "2019-09-07T21:59:15Z")
Hi, i been told that, by using logstash pipeline i can re-create a log format(i.e JSON) when entering into elasticsearch. but not understanding how to do it . i.e the original log file is like this "message": "21:33:4… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=479) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=481)