# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=887 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 888 --- ## [Logstash 6.x - Cannot configur logstash-input-jdbc plug in :ERROR](https://discuss.elastic.co/t/logstash-6-x-cannot-configur-logstash-input-jdbc-plug-in-error/104458)
**Author:** [@Ricigliano](https://discuss.elastic.co/u/Ricigliano)\ **Replies:** 3\ **Last updated:** [October 24, 2017, 11:04am UTC](https://discuss.elastic.co/t/logstash-6-x-cannot-configur-logstash-input-jdbc-plug-in-error/104458 "2017-10-24T11:04:27Z")
I cannot configu logstash -input-jdbc plug in ./logstash-plugin install logstash-input-jdbc ERROR: Something went wrong when installing logstash-input-jdbc, message: Failed to open TCP connection to ar ifacts.elastic.… --- ## [Elasticsearch filter and input on different events](https://discuss.elastic.co/t/elasticsearch-filter-and-input-on-different-events/104716)
**Author:** [@falco\_rocks](https://discuss.elastic.co/u/falco_rocks)\ **Replies:** 3\ **Last updated:** [October 24, 2017, 9:07am UTC](https://discuss.elastic.co/t/elasticsearch-filter-and-input-on-different-events/104716 "2017-10-24T09:07:12Z")
Hello. I am having serious trouble understanding if this is even possible. I have an elasticsearch index containing two types of information: messages and parts. I want to unify some of the events in the index into a … --- ## [Output date in epoch\_millis format](https://discuss.elastic.co/t/output-date-in-epoch-millis-format/105022)
**Author:** [@jong99](https://discuss.elastic.co/u/jong99)\ **Replies:** 1\ **Last updated:** [October 24, 2017, 10:24am UTC](https://discuss.elastic.co/t/output-date-in-epoch-millis-format/105022 "2017-10-24T10:24:27Z")
I'm trying to use Logstash to replace a custom, overly complex system that I've inherited. It currently outputs the timestamp of the event into a field called "@ts", but this is stored in epoch\_millis format. I'm tryin… --- ## [Add data from another record](https://discuss.elastic.co/t/add-data-from-another-record/105026)
**Author:** [@jahlives](https://discuss.elastic.co/u/jahlives)\ **Replies:** 0\ **Last updated:** [October 24, 2017, 9:19am UTC](https://discuss.elastic.co/t/add-data-from-another-record/105026 "2017-10-24T09:19:11Z")
I wonder if it's possible with logstash to to add a field which is retrieved from another logline? My filter currently looks like this if ("" in \[qid\] and \[qid\] == 'proxy-reject') { mutate { replace =\> \[ "qid", 'NOQU… --- ## [How to read the date from multiline Json and covert the string to date](https://discuss.elastic.co/t/how-to-read-the-date-from-multiline-json-and-covert-the-string-to-date/104171)
**Author:** [@Vignesh\_Kumar](https://discuss.elastic.co/u/Vignesh_Kumar)\ **Replies:** 0\ **Last updated:** [October 17, 2017, 5:16am UTC](https://discuss.elastic.co/t/how-to-read-the-date-from-multiline-json-and-covert-the-string-to-date/104171 "2017-10-17T05:16:10Z")
Have the following JSON object where i need to get the element and convert the string to date and traverse through the JSON, { "Client": { "Client Name": "KM Jons", "Portfolio": "Food and Beverage", "ProcessName": "… --- ## [Beats -\> Logstash security, use of security token to authenticate clients?](https://discuss.elastic.co/t/beats-logstash-security-use-of-security-token-to-authenticate-clients/104981)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 2\ **Last updated:** [October 24, 2017, 5:45am UTC](https://discuss.elastic.co/t/beats-logstash-security-use-of-security-token-to-authenticate-clients/104981 "2017-10-24T05:45:25Z")
Hi, I run logstash in a cloud environment and due to the fact I do not control our client's systems, cannot enforce the use of client certificates with beats. What would be great is if all Beats (not just filebeat) supp… --- ## [Logstash filter file don't work](https://discuss.elastic.co/t/logstash-filter-file-dont-work/104789)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 3\ **Last updated:** [October 24, 2017, 5:13am UTC](https://discuss.elastic.co/t/logstash-filter-file-dont-work/104789 "2017-10-24T05:13:14Z")
My\_logstash\_filter.conf file filter { grok { match =\> \[ "message", "(?\[A-Z\_A-Z\_A-Z\]{10,14})", "message", "(?\[A-Z:A-Z:A-Z\]{10,14})", "message", "(?\[A-Z\]{4})" \] } } These three fields never display at a time for e… --- ## [Logstash out of memory error](https://discuss.elastic.co/t/logstash-out-of-memory-error/104638)
**Author:** [@uLan](https://discuss.elastic.co/u/uLan)\ **Replies:** 11\ **Last updated:** [October 24, 2017, 2:41am UTC](https://discuss.elastic.co/t/logstash-out-of-memory-error/104638 "2017-10-24T02:41:24Z")
I have logstash in a docker container that crashes and says out of memory error after restart. I restart it using docker-compose restart logstash. docker stats says it consumes 400MiB~ of RAM when it's running normally a… --- ## [Logstash geoIP and dissect issues](https://discuss.elastic.co/t/logstash-geoip-and-dissect-issues/104944)
**Author:** [@maddie0726](https://discuss.elastic.co/u/maddie0726)\ **Replies:** 2\ **Last updated:** [October 23, 2017, 10:02pm UTC](https://discuss.elastic.co/t/logstash-geoip-and-dissect-issues/104944 "2017-10-23T22:02:58Z")
I'm having a couple issues, not sure if its syntax or what. End result is that I want to read in a 24 hour old .txt file, mimic it being read in as live data, extract an outside IP address from the message field (greedyd… --- ## [Remotely turn on additional logging to Logstash from Kibana](https://discuss.elastic.co/t/remotely-turn-on-additional-logging-to-logstash-from-kibana/104819)
**Author:** [@gminky019](https://discuss.elastic.co/u/gminky019)\ **Replies:** 3\ **Last updated:** [October 23, 2017, 8:36pm UTC](https://discuss.elastic.co/t/remotely-turn-on-additional-logging-to-logstash-from-kibana/104819 "2017-10-23T20:36:48Z")
Hi All, Ive been tasked with researching ELK (Kibana, Elasticsearch, and logstash) stack to possibly use for our new project. I think I have a pretty good grip on it so far but need help answering one question that I ha… --- ## [Logstash Dissect Filter](https://discuss.elastic.co/t/logstash-dissect-filter/104907)
**Author:** [@maddie0726](https://discuss.elastic.co/u/maddie0726)\ **Replies:** 3\ **Last updated:** [October 23, 2017, 8:36pm UTC](https://discuss.elastic.co/t/logstash-dissect-filter/104907 "2017-10-23T20:36:30Z")
I have Logstash Kibana and Elasticsearch versions 5.6.2 installed on Windows 2012 R2. I have a grok pattern that reads in text file entries adequately. The end of each entry is a message field containing a couple differe… --- ## [Logstash single date for logs](https://discuss.elastic.co/t/logstash-single-date-for-logs/104913)
**Author:** [@maddie0726](https://discuss.elastic.co/u/maddie0726)\ **Replies:** 7\ **Last updated:** [October 23, 2017, 8:04pm UTC](https://discuss.elastic.co/t/logstash-single-date-for-logs/104913 "2017-10-23T20:04:11Z")
This might sound counter intuitive, but I am trying to get my logstash config file to ONLY read the previous day's logs. I have the last 90 days of logs saved (except for current day's, it archives at midnight every nigh… --- ## [Logstash can't send the data to elastic search](https://discuss.elastic.co/t/logstash-cant-send-the-data-to-elastic-search/104626)
**Author:** [@kenny123](https://discuss.elastic.co/u/kenny123)\ **Replies:** 3\ **Last updated:** [October 23, 2017, 8:02pm UTC](https://discuss.elastic.co/t/logstash-cant-send-the-data-to-elastic-search/104626 "2017-10-23T20:02:23Z")
Hi, Below is my logstash configuration: input { beats { port =\> 5044 } } filter { if \[type\] == "abc\_serverlog" { grok { match =\> { "message" =\> "(%{TIME:logtime}\\s+)?(%{WORD:loglevel}\\s+)?(\[%{DATA:class}\]\\s" } … --- ## [Mutate timestamp for nginx logs](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814)
**Author:** [@gotjoshua](https://discuss.elastic.co/u/gotjoshua)\ **Replies:** 6\ **Last updated:** [October 23, 2017, 6:32pm UTC](https://discuss.elastic.co/t/mutate-timestamp-for-nginx-logs/104814 "2017-10-23T18:32:34Z")
Continuing the discussion from Keep Logstash from Crashing: @dedemorton gave a workaround that prevents logstash from crashing, but i end up with a read-timestamp field with the literal string value "@timestamp" in the … --- ## [Is it possible to read yesterday's log data like its real-time?](https://discuss.elastic.co/t/is-it-possible-to-read-yesterdays-log-data-like-its-real-time/104933)
**Author:** [@maddie0726](https://discuss.elastic.co/u/maddie0726)\ **Replies:** 2\ **Last updated:** [October 23, 2017, 5:31pm UTC](https://discuss.elastic.co/t/is-it-possible-to-read-yesterdays-log-data-like-its-real-time/104933 "2017-10-23T17:31:04Z")
I am reading input from a .txt file that is 24 hours old, is there any way to have it read in as live data rather than read in all at once? Basically trying to mimic real-time data with yesterday's data. I am running log… --- ## [Parse error with large JSON structure/file](https://discuss.elastic.co/t/parse-error-with-large-json-structure-file/104149)
**Author:** [@marksarnold](https://discuss.elastic.co/u/marksarnold)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 4:41pm UTC](https://discuss.elastic.co/t/parse-error-with-large-json-structure-file/104149 "2017-10-23T16:41:23Z")
I found that logstash complains about syntax errors in even "simple" JSON files when a JSON "record" becomes large. By large, I mean a few hundred key/value pairs. The example file is too large to paste in here, so crea… --- ## [Logstash filtering for field with multiline](https://discuss.elastic.co/t/logstash-filtering-for-field-with-multiline/104928)
**Author:** [@Guillain](https://discuss.elastic.co/u/Guillain)\ **Replies:** 0\ **Last updated:** [October 23, 2017, 4:19pm UTC](https://discuss.elastic.co/t/logstash-filtering-for-field-with-multiline/104928 "2017-10-23T16:19:08Z")
Hello People, I hope you're doing well! I need your support to parse and index properly the following data coming from JSON (so it's only an extract of the desired data): "text": "Input:\\nFieldA - DataA\\nFieldB - DataB… --- ## [GROK directives in one line](https://discuss.elastic.co/t/grok-directives-in-one-line/104887)
**Author:** [@erikkn](https://discuss.elastic.co/u/erikkn)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 1:38pm UTC](https://discuss.elastic.co/t/grok-directives-in-one-line/104887 "2017-10-23T13:38:46Z")
Hey guys :slight\_smile: I am using the GROK plugin to parse through my log file and to extract some useful information. Currently I am using the following config: grok { match =\> {"message" =\> "pullrequest=%{INT:pul… --- ## [Logstash-grok-filtering problem](https://discuss.elastic.co/t/logstash-grok-filtering-problem/104888)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 1:37pm UTC](https://discuss.elastic.co/t/logstash-grok-filtering-problem/104888 "2017-10-23T13:37:29Z")
my Example logs: 2017-10-23 16:59:21,784 : \[INFO \] http-nio-8080-exec-1597 \[c.n.s.c.AbstractSmsClient:567\] SENT =\> smsStatus : \[TELCO\_SENT\] | clientSmsId/gatewaySmsId : \[13383383/GP:DAC:GET\] | cell : \[1712277206\] | nazd… --- ## [NGINX access grok pattern](https://discuss.elastic.co/t/nginx-access-grok-pattern/104885)
**Author:** [@gotjoshua](https://discuss.elastic.co/u/gotjoshua)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 1:31pm UTC](https://discuss.elastic.co/t/nginx-access-grok-pattern/104885 "2017-10-23T13:31:33Z")
Lets take this message as an example: message: sub.ourhost.org 10.10.10.10 - - \[23/Oct/2017:09:30:23 +0000\] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-runner 10.0.1 (10-0-stable; go1.8.3; linux/amd64)" If… --- ## [Logtash Pattern not working](https://discuss.elastic.co/t/logtash-pattern-not-working/104553)
**Author:** [@mbvelo](https://discuss.elastic.co/u/mbvelo)\ **Replies:** 7\ **Last updated:** [October 23, 2017, 1:30pm UTC](https://discuss.elastic.co/t/logtash-pattern-not-working/104553 "2017-10-23T13:30:10Z")
input { beats { port =\> 5044 } } filter { grok { match =\> \["mesage", "ts:(?\((\[0-9\]+)-\*)+ (\[0-9\]+-\*)+ \[A-Z\]+)\\|INFO:%{{LOGLEVEL:LEVE}\\|WebContainer:%{WORD:WebContainer}\\|request:%{WORD:r… --- ## [Logstash grokparsefailure](https://discuss.elastic.co/t/logstash-grokparsefailure/104828)
**Author:** [@xiaoxinba](https://discuss.elastic.co/u/xiaoxinba)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 1:27pm UTC](https://discuss.elastic.co/t/logstash-grokparsefailure/104828 "2017-10-23T13:27:52Z")
my serio: filebeat read log info from file, then kafka, logstash consume log from kafka and write elasticsearch. But sometimes, in logstash ,there are half-baked log (a section of a log event), so grok parse failure; … --- ## [When I run logstash, its not reading /etc/logstash/conf.d/logstash.conf file](https://discuss.elastic.co/t/when-i-run-logstash-its-not-reading-etc-logstash-conf-d-logstash-conf-file/104791)
**Author:** [@bhupathireddys](https://discuss.elastic.co/u/bhupathireddys)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 1:06pm UTC](https://discuss.elastic.co/t/when-i-run-logstash-its-not-reading-etc-logstash-conf-d-logstash-conf-file/104791 "2017-10-23T13:06:56Z")
when I run logstash, systemctl start logstash, its not reading /etc/logstash/conf.d/logstash.conf file, and I gave 777 permissions owner is root. I dont see any error but not seeing any output If I run /usr/share/logs… --- ## [What should I do with Logstash when ES is died?](https://discuss.elastic.co/t/what-should-i-do-with-logstash-when-es-is-died/104853)
**Author:** [@Khanh\_Vo](https://discuss.elastic.co/u/Khanh_Vo)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 1:06pm UTC](https://discuss.elastic.co/t/what-should-i-do-with-logstash-when-es-is-died/104853 "2017-10-23T13:06:00Z")
Hi guys, I'm using Logstash to give logs of Kafka and send to ES but I have a problem that when ES is died, Logstash can't send data to ES, those data will be lost in this time. So what should I do to make sure data isn… --- ## [Logstash Constantly Terminating WinLogBeats Connection](https://discuss.elastic.co/t/logstash-constantly-terminating-winlogbeats-connection/104103)
**Author:** [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\ **Replies:** 0\ **Last updated:** [October 16, 2017, 2:38pm UTC](https://discuss.elastic.co/t/logstash-constantly-terminating-winlogbeats-connection/104103 "2017-10-16T14:38:03Z")
I have a full ELK stack on a Ubuntu VM hosted in Azure, Logstash has the latest Beats plugin installed. I also have a Windows VM with Winlogbeat that is pointed at Logstash. 2017-10-16T15:10:40+01:00 ERR Connecting erro… --- ## [Ruby plugin remove \\ from a string](https://discuss.elastic.co/t/ruby-plugin-remove-from-a-string/104792)
**Author:** [@yoavsradware](https://discuss.elastic.co/u/yoavsradware)\ **Replies:** 2\ **Last updated:** [October 23, 2017, 8:11am UTC](https://discuss.elastic.co/t/ruby-plugin-remove-from-a-string/104792 "2017-10-23T08:11:29Z")
hey , I am using the logstash ruby plugin I am trying to create new json : ruby { init =\> "require 'json'" code =\> " bandwidth = event.get('bandwidth') puts bandwidth\[0\]\['count64'\] foo = … --- ## [Logstash Error: Unmatched first part of surrogate pair](https://discuss.elastic.co/t/logstash-error-unmatched-first-part-of-surrogate-pair/104859)
**Author:** [@AndreAga](https://discuss.elastic.co/u/AndreAga)\ **Replies:** 0\ **Last updated:** [October 23, 2017, 8:05am UTC](https://discuss.elastic.co/t/logstash-error-unmatched-first-part-of-surrogate-pair/104859 "2017-10-23T08:05:49Z")
Hi guys, Sometimes Logstash fails with this error and Logstash crashes: \[FATAL\]\[logstash.runner\] An unexpected error occurred! {:error=\>#\ **Author:** [@Danilo](https://discuss.elastic.co/u/Danilo)\ **Replies:** 2\ **Last updated:** [October 23, 2017, 6:54am UTC](https://discuss.elastic.co/t/convert-value-for-nested-json-array/104672 "2017-10-23T06:54:23Z") Hello, I have a json input like this one: { "fieldA" : "valueA", "fieldB" : \[ { "nestedstring" : "string1a", "nestedint": number1b }, { "nestedstring" : "string2a", "nestedint": number2b }, ...\] } I would like to c… --- ## [Split text field into array](https://discuss.elastic.co/t/split-text-field-into-array/104797)
**Author:** [@yosi](https://discuss.elastic.co/u/yosi)\ **Replies:** 1\ **Last updated:** [October 23, 2017, 6:14am UTC](https://discuss.elastic.co/t/split-text-field-into-array/104797 "2017-10-23T06:14:36Z")
Hey guys, i'm new in ELK and i'm trying to figure how to split my field into an array . json input looks like this : { { "ID":"1", "FilesByProcess":"C:\\Windows\\system32\\taskhost.exe:100;C:\\Windows\\system32\\cmd.exe:1… --- ## [How does codec works in logstash](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630)
**Author:** [@kenny123](https://discuss.elastic.co/u/kenny123)\ **Replies:** 3\ **Last updated:** [October 23, 2017, 6:10am UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630 "2017-10-23T06:10:00Z")
Hi, I am learning ELK and implemented in test environment. logstash is reading the data from Kafka topic which is not in JSON format , but when I specified in logstash configuration that codec =\> json , I see that all … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=886) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=888)