# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=705 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 706 --- ## [Calculating time between two log formats?](https://discuss.elastic.co/t/calculating-time-between-two-log-formats/143647)
**Author:** [@med\_amine](https://discuss.elastic.co/u/med_amine)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 11:56am UTC](https://discuss.elastic.co/t/calculating-time-between-two-log-formats/143647 "2018-08-09T11:56:17Z")
ffhfghthrtjhrtjrjtrjrjrjrjrjj --- ## [Logstash not responding when import csv](https://discuss.elastic.co/t/logstash-not-responding-when-import-csv/142149)
**Author:** [@jibinoosae](https://discuss.elastic.co/u/jibinoosae)\ **Replies:** 6\ **Last updated:** [August 9, 2018, 10:37am UTC](https://discuss.elastic.co/t/logstash-not-responding-when-import-csv/142149 "2018-08-09T10:37:34Z")
i am not able to get import the CSV file with below configuration input{ file{ path =\> "C:\\change\_reque07.csv" start\_position =\> "beginning" } } filter{ csv{ separator =\> "," columns =\> \["company","number","req… --- ## [Ingesting CISCO ASA Logs](https://discuss.elastic.co/t/ingesting-cisco-asa-logs/143662)
**Author:** [@Marcos\_Felix](https://discuss.elastic.co/u/Marcos_Felix)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 10:09am UTC](https://discuss.elastic.co/t/ingesting-cisco-asa-logs/143662 "2018-08-09T10:09:29Z")
Anyone got any good updated links that explains how this can be done? How updated is this one? any takes? --- ## [Empty message after curl input exec command](https://discuss.elastic.co/t/empty-message-after-curl-input-exec-command/142405)
**Author:** [@nico2](https://discuss.elastic.co/u/nico2)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 9:24am UTC](https://discuss.elastic.co/t/empty-message-after-curl-input-exec-command/142405 "2018-08-09T09:24:12Z")
Hello, The following logstash conf file should send back the content of a sftp server. But I get an empty message in the created index. When my command is "DIR C:/Logs", I can get some informations in the message. Is t… --- ## [GROK filter throws ERROR](https://discuss.elastic.co/t/grok-filter-throws-error/143424)
**Author:** [@Raghuveer\_SJ](https://discuss.elastic.co/u/Raghuveer_SJ)\ **Replies:** 24\ **Last updated:** [August 9, 2018, 8:47am UTC](https://discuss.elastic.co/t/grok-filter-throws-error/143424 "2018-08-09T08:47:48Z")
I have written a filter to parse logs based on some condition : filter { if \[message\] =~ "\\tat" { grok { match =\> \["message", "^(\\tat)"\] add\_tag =\> \["stacktrace"\] } } grok { if(\[fields\]\[log\_ty… --- ## [Mapper\_parsing\_exception Using GelfD and Logstash](https://discuss.elastic.co/t/mapper-parsing-exception-using-gelfd-and-logstash/143450)
**Author:** [@Or\_Arnon](https://discuss.elastic.co/u/Or_Arnon)\ **Replies:** 4\ **Last updated:** [August 9, 2018, 8:28am UTC](https://discuss.elastic.co/t/mapper-parsing-exception-using-gelfd-and-logstash/143450 "2018-08-09T08:28:46Z")
We're using GelfD to ship our logs and although nothing has changed, we get these errors: \[2018-08-07T13:50:57,225\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>… --- ## [ERR Failed to publish events caused by: write tcp](https://discuss.elastic.co/t/err-failed-to-publish-events-caused-by-write-tcp/143636)
**Author:** [@Gaurav\_Rishi](https://discuss.elastic.co/u/Gaurav_Rishi)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 7:40am UTC](https://discuss.elastic.co/t/err-failed-to-publish-events-caused-by-write-tcp/143636 "2018-08-09T07:40:13Z")
Hi, I am continually getting an error in my file beat. Logs are writing at very very high speed. filebeat configuration ##########start############## filebeat: prospectors: - paths: - /logs/statistics.log docume… --- ## [The Junk log directory created in logstash is invalid](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821)
**Author:** [@swethamahesh](https://discuss.elastic.co/u/swethamahesh)\ **Replies:** 6\ **Last updated:** [August 9, 2018, 5:33am UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821 "2018-08-09T05:33:51Z")
Hi, My filebeat configuration is as below: filebeat.prospectors: - input\_type: log paths: - /var/log/oes.log encoding: utf-8 fields\_under\_root: true document\_type: log fields: service\_name: ops ap… --- ## [Multiple logstash concurrent data to the ES cluster, how to achieve data load balancing](https://discuss.elastic.co/t/multiple-logstash-concurrent-data-to-the-es-cluster-how-to-achieve-data-load-balancing/139854)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 2\ **Last updated:** [August 9, 2018, 2:00am UTC](https://discuss.elastic.co/t/multiple-logstash-concurrent-data-to-the-es-cluster-how-to-achieve-data-load-balancing/139854 "2018-08-09T02:00:38Z")
Data is forwarded by logstash. How can I achieve load balancing to the ES cluster without causing data duplication? Logstash can configure multiple output addresses, but how can we do data duplication? Is it using load … --- ## [How to set maximum retries on logstash output elastic plugin?](https://discuss.elastic.co/t/how-to-set-maximum-retries-on-logstash-output-elastic-plugin/143603)
**Author:** [@gowt](https://discuss.elastic.co/u/gowt)\ **Replies:** 0\ **Last updated:** [August 8, 2018, 9:54pm UTC](https://discuss.elastic.co/t/how-to-set-maximum-retries-on-logstash-output-elastic-plugin/143603 "2018-08-08T21:54:35Z")
I am executing logstash from command prompt utility scheduled as windows service. my service triggers logstash execution every day once. I got a situation where if my elastic host is down (or) wrong elastic host configur… --- ## [Date match not working](https://discuss.elastic.co/t/date-match-not-working/143589)
**Author:** [@Victor\_Musa](https://discuss.elastic.co/u/Victor_Musa)\ **Replies:** 3\ **Last updated:** [August 8, 2018, 8:23pm UTC](https://discuss.elastic.co/t/date-match-not-working/143589 "2018-08-08T20:23:35Z")
Hello, i have a field called "timestamp" (i already triyed with another name) the content im getting with dissect its like this: 08/08/2018 17:02:39 This would be like dd/MM/yyyy HH:mm:ss im triyed with this date +… --- ## [S3 Input not working](https://discuss.elastic.co/t/s3-input-not-working/58267)
**Author:** [@Zorlin](https://discuss.elastic.co/u/Zorlin)\ **Replies:** 5\ **Last updated:** [August 8, 2018, 7:09pm UTC](https://discuss.elastic.co/t/s3-input-not-working/58267 "2018-08-08T19:09:39Z")
Hey there, We have a Logstash setup processing a bunch of different inputs successfully. Unfortunately, the S3 input is not working. Here is the current config I'm working with: input { s3 { bucket =\> "prod… --- ## [S3 input plugin failed to stream from Glaciers](https://discuss.elastic.co/t/s3-input-plugin-failed-to-stream-from-glaciers/143391)
**Author:** [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\ **Replies:** 3\ **Last updated:** [August 8, 2018, 6:29pm UTC](https://discuss.elastic.co/t/s3-input-plugin-failed-to-stream-from-glaciers/143391 "2018-08-08T18:29:42Z")
Hello there, We just set up ELK to stream event/logs from s3 to logstash using s3 input plugin (all version 6.3.2). We can stream all standard files, but we have the lifycircle enabled to save all files older then 90 d… --- ## [Need help with grok filtering](https://discuss.elastic.co/t/need-help-with-grok-filtering/142596)
**Author:** [@Victor\_Musa](https://discuss.elastic.co/u/Victor_Musa)\ **Replies:** 15\ **Last updated:** [August 8, 2018, 6:00pm UTC](https://discuss.elastic.co/t/need-help-with-grok-filtering/142596 "2018-08-08T18:00:40Z")
Hello everyone im working for several few days on loggind the f5 Big Data with Elk but the parsing of log its not working at all on logstash with Grok (and i don't find the patterns necesary for this), how would you matc… --- ## [Secure Filebeat to Logstash](https://discuss.elastic.co/t/secure-filebeat-to-logstash/143549)
**Author:** [@Chris\_Billett](https://discuss.elastic.co/u/Chris_Billett)\ **Replies:** 1\ **Last updated:** [August 8, 2018, 3:34pm UTC](https://discuss.elastic.co/t/secure-filebeat-to-logstash/143549 "2018-08-08T15:34:03Z")
Hi, I have setup a new ELK server running 6.3. I can can get my other servers to transfer data using filebeat without SSL but when I run SSL I receive the following: Filebeat log 2018-08-08T15:35:19+01:00 INFO Error p… --- ## [Logstash: Index complex structure from jdbc input](https://discuss.elastic.co/t/logstash-index-complex-structure-from-jdbc-input/143447)
**Author:** [@Lucas\_Dehandschutter](https://discuss.elastic.co/u/Lucas_Dehandschutter)\ **Replies:** 12\ **Last updated:** [August 8, 2018, 2:38pm UTC](https://discuss.elastic.co/t/logstash-index-complex-structure-from-jdbc-input/143447 "2018-08-08T14:38:52Z")
Hi, I'd like to inject data from sql databases into Elasticsearch. Let say I have 3 tables: Company, Department and Employee. Each company has several department and each department has several employees. This exampl… --- ## [HTTP Poller authentication: encode token](https://discuss.elastic.co/t/http-poller-authentication-encode-token/143532)
**Author:** [@AurelienG](https://discuss.elastic.co/u/AurelienG)\ **Replies:** 2\ **Last updated:** [August 8, 2018, 2:35pm UTC](https://discuss.elastic.co/t/http-poller-authentication-encode-token/143532 "2018-08-08T14:35:25Z")
Hello, I am trying to use the HTTP Poller input plugin to get data from an API. Although, I need to authenticate with a basic authentication, with username and token. This would be a problem if the token didn't have to … --- ## [Logstash-input-mongodb update and delete doc](https://discuss.elastic.co/t/logstash-input-mongodb-update-and-delete-doc/143360)
**Author:** [@abilio-cs](https://discuss.elastic.co/u/abilio-cs)\ **Replies:** 2\ **Last updated:** [August 8, 2018, 1:47pm UTC](https://discuss.elastic.co/t/logstash-input-mongodb-update-and-delete-doc/143360 "2018-08-08T13:47:45Z")
Hello! I am using logstash in order to replace mongo-connector, so I need to update and delete docs that were changed in MongoDB. I set upsert and I also tried to set the action, however it seems that the input is not … --- ## [Kv filter: empty value after key yields strange behavior](https://discuss.elastic.co/t/kv-filter-empty-value-after-key-yields-strange-behavior/142965)
**Author:** [@Micah\_Hunsberger](https://discuss.elastic.co/u/Micah_Hunsberger)\ **Replies:** 6\ **Last updated:** [August 8, 2018, 1:40pm UTC](https://discuss.elastic.co/t/kv-filter-empty-value-after-key-yields-strange-behavior/142965 "2018-08-08T13:40:47Z")
Logstash version 6.3.0 I'm using the kv filter with the default settings, which means: value\_split =\> "=" field\_split =\> " " If I send a message with the format of key= without a value on the other side, it interprets… --- ## [No emails received from logstash for email output plugin](https://discuss.elastic.co/t/no-emails-received-from-logstash-for-email-output-plugin/143432)
**Author:** [@naga\_kunchala](https://discuss.elastic.co/u/naga_kunchala)\ **Replies:** 7\ **Last updated:** [August 8, 2018, 1:01pm UTC](https://discuss.elastic.co/t/no-emails-received-from-logstash-for-email-output-plugin/143432 "2018-08-08T13:01:01Z")
I do not get any emails from logstash for the below config. Could anyone please help me what is going wrong, also please let me know where can I trace the email logs. There is no any error on running the the config file. … --- ## [Persistent Queue details](https://discuss.elastic.co/t/persistent-queue-details/143133)
**Author:** [@JST](https://discuss.elastic.co/u/JST)\ **Replies:** 2\ **Last updated:** [August 8, 2018, 12:26pm UTC](https://discuss.elastic.co/t/persistent-queue-details/143133 "2018-08-08T12:26:50Z")
All, I have the following setup: \[tcp, http, file, udp\] ---\> logstash ---\> \[custom output plugin\] And a few questions on persistent queues. As per the limitions do the tcp and udp inputs enter the persistent queue or… --- ## [Send dictionary objects to logstash](https://discuss.elastic.co/t/send-dictionary-objects-to-logstash/143187)
**Author:** [@Gabriela\_Diana\_Stoic](https://discuss.elastic.co/u/Gabriela_Diana_Stoic)\ **Replies:** 2\ **Last updated:** [August 8, 2018, 12:24pm UTC](https://discuss.elastic.co/t/send-dictionary-objects-to-logstash/143187 "2018-08-08T12:24:17Z")
Hi, I have a file that is continuously populated with events from a server. The events have the following format: {"Time": "2018-08-06 16:46:01", "Severity": "Minor", "Equipment": "Server", "State": "Unacknowledged Ev… --- ## [Dynamic fields value matching](https://discuss.elastic.co/t/dynamic-fields-value-matching/143490)
**Author:** [@Raghuveer\_SJ](https://discuss.elastic.co/u/Raghuveer_SJ)\ **Replies:** 1\ **Last updated:** [August 8, 2018, 12:14pm UTC](https://discuss.elastic.co/t/dynamic-fields-value-matching/143490 "2018-08-08T12:14:20Z")
I want to perform a matching like : if \[type\] in \["\*UI"\] { #do something } I know its not going to work. I want to do a regrex matching. But is there a way ? --- ## [Logstash Date filter timezone](https://discuss.elastic.co/t/logstash-date-filter-timezone/143461)
**Author:** [@Shubhank\_Gupta](https://discuss.elastic.co/u/Shubhank_Gupta)\ **Replies:** 2\ **Last updated:** [August 8, 2018, 12:11pm UTC](https://discuss.elastic.co/t/logstash-date-filter-timezone/143461 "2018-08-08T12:11:21Z")
Hi, filebeat.yml Code - filebeat.inputs: type: log paths: logs/debug/\* fields: document\_type: debug cu\_hostname\_input: dummy user\_tag: shubh\_08\_aug\_debug\_testt sw\_version: Unspecified timeZone: EST fields… --- ## [Grok filter for any symbol include string break](https://discuss.elastic.co/t/grok-filter-for-any-symbol-include-string-break/143491)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 4\ **Last updated:** [August 8, 2018, 12:06pm UTC](https://discuss.elastic.co/t/grok-filter-for-any-symbol-include-string-break/143491 "2018-08-08T12:06:20Z")
Hello. I'm trying to parse event message to fields. There is no problem with simple fields, but i'm unable to highlight biggest field that looks like this: Details: {"Realm\_Size":68,"Realm":"https://uisnotification-sb.… --- ## [Grok pattern with forward slashes result in grokfailure](https://discuss.elastic.co/t/grok-pattern-with-forward-slashes-result-in-grokfailure/143362)
**Author:** [@jmcazaux](https://discuss.elastic.co/u/jmcazaux)\ **Replies:** 4\ **Last updated:** [August 8, 2018, 11:49am UTC](https://discuss.elastic.co/t/grok-pattern-with-forward-slashes-result-in-grokfailure/143362 "2018-08-08T11:49:20Z")
Hi there, I am trying to extract path components from a "path-like" string and it fails with grokfailure. I have been banding my head on walls for houyrs with this. It works in "Grok Debugger", but fails in Logstash. M… --- ## [Logstash not processign the data](https://discuss.elastic.co/t/logstash-not-processign-the-data/143219)
**Author:** [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Replies:** 3\ **Last updated:** [August 8, 2018, 6:31am UTC](https://discuss.elastic.co/t/logstash-not-processign-the-data/143219 "2018-08-08T06:31:55Z")
Hello, I am onboarding the logs and i couldn't see those logs in kibana. For example, if configured filebeat to onbaord /var/log/elk/stage1/execution/process.log , then i could see those logs in kibana but when i onb… --- ## [Logstash.inputs.s3 failed fetching events from S3](https://discuss.elastic.co/t/logstash-inputs-s3-failed-fetching-events-from-s3/143421)
**Author:** [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\ **Replies:** 2\ **Last updated:** [August 8, 2018, 6:57am UTC](https://discuss.elastic.co/t/logstash-inputs-s3-failed-fetching-events-from-s3/143421 "2018-08-08T06:57:01Z")
Hello there, I set up the logstash s3 input plugin to fetch event/logs from CloudTrail logs on S3. The following error filled in the logstash plain log: ========================= ... \[2018-08-08T03:40:22,523\]\[INFO \]… --- ## [ELK Stack Setup to Receive Beats from Magento](https://discuss.elastic.co/t/elk-stack-setup-to-receive-beats-from-magento/143380)
**Author:** [@imranvirk](https://discuss.elastic.co/u/imranvirk)\ **Replies:** 6\ **Last updated:** [August 8, 2018, 5:52am UTC](https://discuss.elastic.co/t/elk-stack-setup-to-receive-beats-from-magento/143380 "2018-08-08T05:52:03Z")
I have 2 droplets configured in my DO project: ELK Stack Magento eCommerce platform on a LAMP Stack Both instances are up and running, with all services setup. However, I am having an issue with my Magento server, wh… --- ## [An unexpected error occurred! {:error=\>#\](https://discuss.elastic.co/t/an-unexpected-error-occurred-error-argumenterror-setting-hasnt-been-registered/143324)
**Author:** [@Arpit\_Gulati](https://discuss.elastic.co/u/Arpit_Gulati)\ **Replies:** 3\ **Last updated:** [August 8, 2018, 5:37am UTC](https://discuss.elastic.co/t/an-unexpected-error-occurred-error-argumenterror-setting-hasnt-been-registered/143324 "2018-08-08T05:37:04Z")
I update the logstash verion from 2.4.0 to 5.4.0 using a tar file but when I start the logstash using ./bin/logstash -f conf.d/logstash.conf command . I got an error \[FATAL\]\[logstash.runner \] An unexpected erro… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=704) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=706)