# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1169 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1170 --- ## [Filter aggregate and timeout](https://discuss.elastic.co/t/filter-aggregate-and-timeout/46717)
**Author:** [@Gnosis](https://discuss.elastic.co/u/Gnosis)\ **Replies:** 0\ **Last updated:** [April 7, 2016, 4:16pm UTC](https://discuss.elastic.co/t/filter-aggregate-and-timeout/46717 "2016-04-07T16:16:43Z")
Hi all, Imagine several lines with a key and a letter like this : key1 - Q key2 - Q key1 - A key1 - C with Q =\> Qestion ; A =\> Answer ; C =\> Close My aim is to know when i don't have "A" for a "Q". So i use the… --- ## [\[SOLVED\]Slow pipeline](https://discuss.elastic.co/t/solved-slow-pipeline/46647)
**Author:** [@mikula](https://discuss.elastic.co/u/mikula)\ **Replies:** 1\ **Last updated:** [April 7, 2016, 4:04pm UTC](https://discuss.elastic.co/t/solved-slow-pipeline/46647 "2016-04-07T16:04:53Z")
Recently (after upgrade to LS 2.3.0 and ES 2.3.0 my piplene slowed down from thousnads of entries indexed to around 50 per second... I also started to use fingerprint plugin (twice per event) and one of figerprints is … --- ## [Parsing problem?](https://discuss.elastic.co/t/parsing-problem/46713)
**Author:** [@Cross](https://discuss.elastic.co/u/Cross)\ **Replies:** 0\ **Last updated:** [April 7, 2016, 3:49pm UTC](https://discuss.elastic.co/t/parsing-problem/46713 "2016-04-07T15:49:53Z")
sorry about the bump, but I haven't been able to figure out the problem so far. so once again any help much appreciated --- ## [Need more information on logstash](https://discuss.elastic.co/t/need-more-information-on-logstash/46694)
**Author:** [@oth1113](https://discuss.elastic.co/u/oth1113)\ **Replies:** 3\ **Last updated:** [April 7, 2016, 3:24pm UTC](https://discuss.elastic.co/t/need-more-information-on-logstash/46694 "2016-04-07T15:24:39Z")
I need easy explanation please \[:slight\_smile:\] I am a beginner in logstash i have some questions \[:slight\_smile:\] 1- What is the tag "\_gorkparsefailure" and it used for? if I delete this tag what will happen ? 2- ho… --- ## ["mutate gsub" on each element of an array](https://discuss.elastic.co/t/mutate-gsub-on-each-element-of-an-array/46512)
**Author:** [@thorsten.niederkrome](https://discuss.elastic.co/u/thorsten.niederkrome)\ **Replies:** 1\ **Last updated:** [April 7, 2016, 2:35pm UTC](https://discuss.elastic.co/t/mutate-gsub-on-each-element-of-an-array/46512 "2016-04-07T14:35:54Z")
Hi, I'm pretty new to logstash and have a problem I'm not able to solve by trying or searching the web (didn't find any similar questions/solutions yet \[:frowning:\] ) I'm using logstash to extract MAC addresses from lo… --- ## [CA Signed Certificate Not Working](https://discuss.elastic.co/t/ca-signed-certificate-not-working/46523)
**Author:** [@ltwarner](https://discuss.elastic.co/u/ltwarner)\ **Replies:** 5\ **Last updated:** [April 7, 2016, 2:07pm UTC](https://discuss.elastic.co/t/ca-signed-certificate-not-working/46523 "2016-04-07T14:07:04Z")
Hey there, I am working to setup a Logstash and Beats (file and winlog). I have everything working for a self-signed certificate on LogStash and Beats , but if I try to create a CA signed Certificate I keep getting on… --- ## [\[SOLVED\] Logstash: No output, no logs](https://discuss.elastic.co/t/solved-logstash-no-output-no-logs/46641)
**Author:** [@axelfelix](https://discuss.elastic.co/u/axelfelix)\ **Replies:** 2\ **Last updated:** [April 7, 2016, 12:42pm UTC](https://discuss.elastic.co/t/solved-logstash-no-output-no-logs/46641 "2016-04-07T12:42:44Z")
Hi all, My architecture: Equipments --\> syslog-ng --\> redis \<-- logstash --\> elasticsearch \<--kibana I made a lot of tests and I'm sure that syslog-ng received correctly logs, that redis put the logs in a list (key: t… --- ## [Logstash always output input event type as log](https://discuss.elastic.co/t/logstash-always-output-input-event-type-as-log/46554)
**Author:** [@blnprasad](https://discuss.elastic.co/u/blnprasad)\ **Replies:** 3\ **Last updated:** [April 7, 2016, 9:00am UTC](https://discuss.elastic.co/t/logstash-always-output-input-event-type-as-log/46554 "2016-04-07T09:00:06Z")
Hi, I've configured logstash input for different type. but when it ouputs elastic search, value of type filed is coming as "log". No matter what i configure type value in input section, it's being outputted as "log". my… --- ## [Unknow this error](https://discuss.elastic.co/t/unknow-this-error/46632)
**Author:** [@Inbeo\_Beo](https://discuss.elastic.co/u/Inbeo_Beo)\ **Replies:** 3\ **Last updated:** [April 7, 2016, 7:54am UTC](https://discuss.elastic.co/t/unknow-this-error/46632 "2016-04-07T07:54:09Z")
I have a file logstash.conf input { beats { port =\> 5044 } } output { elasticsearch { hosts =\> "localhost:9200" index =\> "%{\[@metadata\]\[beat\]}-%{+YYYY.MM.dd}" document\_type =\> "%{\[@metadata\]\[ty… --- ## [Logstash log parsing in logstash](https://discuss.elastic.co/t/logstash-log-parsing-in-logstash/46514)
**Author:** [@mikula](https://discuss.elastic.co/u/mikula)\ **Replies:** 2\ **Last updated:** [April 7, 2016, 6:32am UTC](https://discuss.elastic.co/t/logstash-log-parsing-in-logstash/46514 "2016-04-07T06:32:27Z")
Hello, is there way to parse logstash logs in logstash? json filter doesn't seem to be working on this... --- ## [Read logs stored using Enterprise Semantic Logging format](https://discuss.elastic.co/t/read-logs-stored-using-enterprise-semantic-logging-format/46449)
**Author:** [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Replies:** 9\ **Last updated:** [April 7, 2016, 5:45am UTC](https://discuss.elastic.co/t/read-logs-stored-using-enterprise-semantic-logging-format/46449 "2016-04-07T05:45:14Z")
We are using Semantic Logging mechanism to log our service request and response in physical files. Is there a existing GROK Pattern to pull my logs in a good way ? OR Any other efficient way to read these logs? Event… --- ## [Settings: Default pipeline workers: 2](https://discuss.elastic.co/t/settings-default-pipeline-workers-2/46128)
**Author:** [@Amin1](https://discuss.elastic.co/u/Amin1)\ **Replies:** 4\ **Last updated:** [April 7, 2016, 5:33am UTC](https://discuss.elastic.co/t/settings-default-pipeline-workers-2/46128 "2016-04-07T05:33:17Z")
Hi My sample data is like follow TransactionsPK,InsertTime,RetrivalRefNo,MsgType,TimeLocal,DateLocal,DateCaptured,TerminalID,CardAcqID,SystemTraceNo,Amount,TransType,PrimaryAccNo,PrimaryAccNoEnc,IssCode,ProcessingCod… --- ## [Logstash Heap Size Increasing](https://discuss.elastic.co/t/logstash-heap-size-increasing/46533)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 1\ **Last updated:** [April 7, 2016, 1:53am UTC](https://discuss.elastic.co/t/logstash-heap-size-increasing/46533 "2016-04-07T01:53:41Z")
HI, I'm using logstash version 2.2.1 and facing following issue while running the logstash. Anyone can help me to resolve it. Error: Your application used more memory than the safety cap of 1G. Specify -J-Xmx####m to i… --- ## [Index .txt format log file](https://discuss.elastic.co/t/index-txt-format-log-file/46338)
**Author:** [@Prateek\_Kshtriya](https://discuss.elastic.co/u/Prateek_Kshtriya)\ **Replies:** 1\ **Last updated:** [April 7, 2016, 1:09am UTC](https://discuss.elastic.co/t/index-txt-format-log-file/46338 "2016-04-07T01:09:14Z")
Hi All, Please suggest how to create a config file to index .txt format log file. Tried to index simply by adding path in the config with output to elasticsearch. The data in the log(.txt) comes like below- 02 00 10:… --- ## [Flushing buffer at interval error](https://discuss.elastic.co/t/flushing-buffer-at-interval-error/46254)
**Author:** [@mc1392](https://discuss.elastic.co/u/mc1392)\ **Replies:** 4\ **Last updated:** [April 6, 2016, 6:37pm UTC](https://discuss.elastic.co/t/flushing-buffer-at-interval-error/46254 "2016-04-06T18:37:06Z")
I am using logstash 2.1.0 and elasticsearch 2.0.0. Here is my config: input{ file{ path =\> "/home/ubuntu/torched\_products\_addresses.csv" type =\> "ats\_locations" … --- ## [Oauth2 authentication for rest services logstash output plugin](https://discuss.elastic.co/t/oauth2-authentication-for-rest-services-logstash-output-plugin/46578)
**Author:** [@kasi](https://discuss.elastic.co/u/kasi)\ **Replies:** 0\ **Last updated:** [April 6, 2016, 5:26pm UTC](https://discuss.elastic.co/t/oauth2-authentication-for-rest-services-logstash-output-plugin/46578 "2016-04-06T17:26:09Z")
We have a requirement where we have to forward messages from logstash to a rest service endpoint that uses oauth2 authentication, from what I have seen there is no output plugin that can facilitate this other than writin… --- ## [How I can modify timestamp in log message to UTC](https://discuss.elastic.co/t/how-i-can-modify-timestamp-in-log-message-to-utc/45830)
**Author:** [@nagesh](https://discuss.elastic.co/u/nagesh)\ **Replies:** 15\ **Last updated:** [April 6, 2016, 4:37pm UTC](https://discuss.elastic.co/t/how-i-can-modify-timestamp-in-log-message-to-utc/45830 "2016-04-06T16:37:33Z")
Hi, Our log messages contain timestamp in EST format. How I can modify the timestamp to UTC before displaying logs into Kibana. Example message: 2016-03-26T01:55:47.78-0500 \[App/0\] OUT \[AUDIT \] CWWKE0001I: T… --- ## [Logstash upload to Elastic search works in Command Line but hangs when run through shell script](https://discuss.elastic.co/t/logstash-upload-to-elastic-search-works-in-command-line-but-hangs-when-run-through-shell-script/45590)
**Author:** [@shanbalasay](https://discuss.elastic.co/u/shanbalasay)\ **Replies:** 2\ **Last updated:** [March 28, 2016, 9:18pm UTC](https://discuss.elastic.co/t/logstash-upload-to-elastic-search-works-in-command-line-but-hangs-when-run-through-shell-script/45590 "2016-03-28T21:18:04Z")
Executing logstash in command line worked successfully but putting the same command with the JAVA\_HOME & PATH variables set & exported in a shell script doesnt work. While running the shell script, logstash comes to the… --- ## [In: Redis, Out: Redis very slow since upgrade vom LS from 1.5 to 2.3.0](https://discuss.elastic.co/t/in-redis-out-redis-very-slow-since-upgrade-vom-ls-from-1-5-to-2-3-0/46539)
**Author:** [@Robert\_Walter](https://discuss.elastic.co/u/Robert_Walter)\ **Replies:** 0\ **Last updated:** [April 6, 2016, 2:02pm UTC](https://discuss.elastic.co/t/in-redis-out-redis-very-slow-since-upgrade-vom-ls-from-1-5-to-2-3-0/46539 "2016-04-06T14:02:51Z")
Hi, i'm using logstash to transport logs from redis to redis. \`input { redis { host =\> "redis1.int.local" port =\> "6381" data\_type =\> "list" threads =\> 50 key =\> "logs" … --- ## [One time batch processing big number of files](https://discuss.elastic.co/t/one-time-batch-processing-big-number-of-files/46461)
**Author:** [@Alexander\_Popov](https://discuss.elastic.co/u/Alexander_Popov)\ **Replies:** 12\ **Last updated:** [April 6, 2016, 1:37pm UTC](https://discuss.elastic.co/t/one-time-batch-processing-big-number-of-files/46461 "2016-04-06T13:37:52Z")
Have ~ 50K files in single directory( ~200Gb ) of logs. trying to process to parse and add them to elasticsearch my config: input { file { path =\> "/mnt/storage/\*.txt" sincedb\_path =\> "/dev/null" codec … --- ## [Code review: masking credit card numbers](https://discuss.elastic.co/t/code-review-masking-credit-card-numbers/46507)
**Author:** [@Alex\_6](https://discuss.elastic.co/u/Alex_6)\ **Replies:** 6\ **Last updated:** [April 6, 2016, 12:47pm UTC](https://discuss.elastic.co/t/code-review-masking-credit-card-numbers/46507 "2016-04-06T12:47:27Z")
We have a failsafe to mask credit card numbers just in case they appear in a log (they shouldn't but...). We have a Grok pattern for credit card numbers and want to replace any occurrences of that with 16 X's. # Check … --- ## [Create index from tags (basic question ?)](https://discuss.elastic.co/t/create-index-from-tags-basic-question/46482)
**Author:** [@smutel](https://discuss.elastic.co/u/smutel)\ **Replies:** 6\ **Last updated:** [April 6, 2016, 12:46pm UTC](https://discuss.elastic.co/t/create-index-from-tags-basic-question/46482 "2016-04-06T12:46:05Z")
Hello, I would like to create index to split data regarding the application for example. Is-it a good idea to use tags for that ? If yes, how can I convert tags from array to string ? Because when I want to use tags, l… --- ## [\_grokparsefailure Tag when parsing logs via logstash](https://discuss.elastic.co/t/-grokparsefailure-tag-when-parsing-logs-via-logstash/46224)
**Author:** [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\ **Replies:** 5\ **Last updated:** [April 6, 2016, 11:03am UTC](https://discuss.elastic.co/t/-grokparsefailure-tag-when-parsing-logs-via-logstash/46224 "2016-04-06T11:03:12Z")
I have the following log:- 2016-03-31 12:40:40 INFO SmartAppUtils:981 - Organization: AppsTwo 2016-03-31 12:40:40 INFO SmartAppUtils:988 - Brand organization: AppsTwo 2016-03-31 12:40:51 INFO SmartAppUtils:981 - Org… --- ## [Some messages are jsonparsed when coming from redis](https://discuss.elastic.co/t/some-messages-are-jsonparsed-when-coming-from-redis/46358)
**Author:** [@widhalmt](https://discuss.elastic.co/u/widhalmt)\ **Replies:** 2\ **Last updated:** [April 6, 2016, 10:11am UTC](https://discuss.elastic.co/t/some-messages-are-jsonparsed-when-coming-from-redis/46358 "2016-04-06T10:11:23Z")
Hi, I use Redis to connect a Logstash Shipper and Indexer instance. All my filter rules are in the Indexer save for one that adds a field called "shipper" with a value representing the hostname of the node this shipper… --- ## [Logstash 2.3.0 Startup Error](https://discuss.elastic.co/t/logstash-2-3-0-startup-error/46225)
**Author:** [@Sameer\_Dharur](https://discuss.elastic.co/u/Sameer_Dharur)\ **Replies:** 13\ **Last updated:** [April 6, 2016, 9:44am UTC](https://discuss.elastic.co/t/logstash-2-3-0-startup-error/46225 "2016-04-06T09:44:58Z")
Hello! I had been using this configuration for Logstash 2.2.0 until this morning, which worked perfectly. But after an upgrade to 2.3.0, I get the following error message on the console - An unexpected error occurred… --- ## [Logstash: Error: oracle.jdbc.driver.OracleDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver\_library?](https://discuss.elastic.co/t/logstash-error-oracle-jdbc-driver-oracledriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/46501)
**Author:** [@krushnat\_khawale](https://discuss.elastic.co/u/krushnat_khawale)\ **Replies:** 0\ **Last updated:** [April 6, 2016, 9:25am UTC](https://discuss.elastic.co/t/logstash-error-oracle-jdbc-driver-oracledriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/46501 "2016-04-06T09:25:19Z")
Hi, I am using Logstash-2.1.1. I want to fetch data from Oracle database and to put into ES. I'm using following configuration. But it is saying that "Error: oracle.jdbc.driver.OracleDriver not loaded. Are you sure … --- ## [How should I parse date time for below string](https://discuss.elastic.co/t/how-should-i-parse-date-time-for-below-string/46336)
**Author:** [@macymin](https://discuss.elastic.co/u/macymin)\ **Replies:** 5\ **Last updated:** [April 6, 2016, 9:08am UTC](https://discuss.elastic.co/t/how-should-i-parse-date-time-for-below-string/46336 "2016-04-06T09:08:02Z")
In my file, the content is like this: 2016040417350700, \> '/home/prbdata/esda/prod/spool.6/esda/f12masc1d0/7882201\_001\_' \[1:2:1\] how can I parse 2016040417350700 into a field with correct date time format ? --- ## [Best practices configuring indexes to periodic logs](https://discuss.elastic.co/t/best-practices-configuring-indexes-to-periodic-logs/46392)
**Author:** [@nlhedder](https://discuss.elastic.co/u/nlhedder)\ **Replies:** 5\ **Last updated:** [April 6, 2016, 9:04am UTC](https://discuss.elastic.co/t/best-practices-configuring-indexes-to-periodic-logs/46392 "2016-04-06T09:04:37Z")
Not sure if my search on the forum is correct (many hits on searchin in Indexes, so I'll try to post my question here: We have several servers /switches/appliances which I would like to pull the logging into ELK. This … --- ## [Error with throttle: RuntimeError: can't add a new key into hash during iteration](https://discuss.elastic.co/t/error-with-throttle-runtimeerror-cant-add-a-new-key-into-hash-during-iteration/44807)
**Author:** [@marcmga](https://discuss.elastic.co/u/marcmga)\ **Replies:** 2\ **Last updated:** [April 6, 2016, 9:02am UTC](https://discuss.elastic.co/t/error-with-throttle-runtimeerror-cant-add-a-new-key-into-hash-during-iteration/44807 "2016-04-06T09:02:51Z")
Hello, Logstash is stopping due to this error: logstas.err Non US-ASCII detected and no charset defined. Defaulting to UTF-8, set your own if this is incorrect. Non US-ASCII detected and no charset defined. Default… --- ## [Equivalent of 'stream\_identity' for multiline codec](https://discuss.elastic.co/t/equivalent-of-stream-identity-for-multiline-codec/45785)
**Author:** [@jpry](https://discuss.elastic.co/u/jpry)\ **Replies:** 2\ **Last updated:** [April 6, 2016, 8:23am UTC](https://discuss.elastic.co/t/equivalent-of-stream-identity-for-multiline-codec/45785 "2016-04-06T08:23:08Z")
Hello, All is in the title. Is there an equivalent for 'stream\_identity' property of the multiline filter in multiline codec? How to merge interlaced log lines based on their dynamic identifier using multiline codec in… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1168) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1170)