# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=765 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 766 --- ## [\[solved\] Namespacing grok output](https://discuss.elastic.co/t/solved-namespacing-grok-output/130448)
**Author:** [@Sushimaster](https://discuss.elastic.co/u/Sushimaster)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 9:54pm UTC](https://discuss.elastic.co/t/solved-namespacing-grok-output/130448 "2018-05-07T21:54:45Z")
Hi, when using logstash for pushing apache logs into elasticsearch, very close to the example config in logstash docs: filter { mutate { replace =\> { type =\> "apache\_access" } } grok { match =\>… --- ## [Dateparsefailure - Mac Syslogs](https://discuss.elastic.co/t/dateparsefailure-mac-syslogs/130511)
**Author:** [@TCBeeekley](https://discuss.elastic.co/u/TCBeeekley)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 7:21pm UTC](https://discuss.elastic.co/t/dateparsefailure-mac-syslogs/130511 "2018-05-07T19:21:43Z")
Hello! Brand new to the Elasticstack, I have a question about date filtering. Here's the sample message: May 3 14:08:41 foo-hostname com.apple.xpc.launchd\[1\] (com.apple.preference.displays.MirrorDisplays): Service only… --- ## [Creating an error index](https://discuss.elastic.co/t/creating-an-error-index/130552)
**Author:** [@kabilan](https://discuss.elastic.co/u/kabilan)\ **Replies:** 1\ **Last updated:** [May 7, 2018, 7:02pm UTC](https://discuss.elastic.co/t/creating-an-error-index/130552 "2018-05-07T19:02:03Z")
Hi all, So I am in the processing of migrating configs for my technologies from the grok filter to dissect filter. I want to output the dissect tag failures into a separate error index and the parsed data in another in… --- ## [Upgrading ES and logstash](https://discuss.elastic.co/t/upgrading-es-and-logstash/130891)
**Author:** [@fastboyz](https://discuss.elastic.co/u/fastboyz)\ **Replies:** 0\ **Last updated:** [May 7, 2018, 6:26pm UTC](https://discuss.elastic.co/t/upgrading-es-and-logstash/130891 "2018-05-07T18:26:00Z")
Hi, I am new to Logstash and ES, and I have to update Logstash and ES for my company. I downloaded the newest version and I get this error. {:timestamp=\>"2018-05-07T14:08:55.874000-0400", :message=\>"Attempted to send a… --- ## [Sending an unfiltered copy](https://discuss.elastic.co/t/sending-an-unfiltered-copy/130659)
**Author:** [@topher](https://discuss.elastic.co/u/topher)\ **Replies:** 14\ **Last updated:** [May 7, 2018, 6:19pm UTC](https://discuss.elastic.co/t/sending-an-unfiltered-copy/130659 "2018-05-07T18:19:29Z")
OK, I'm relatively new to Logstash; we're looking at it as a possible replacement for what we're using now. I've been tasked with setting it up in such a way that it redirects the data along to our current system and a… --- ## [Measuring errors writing to ES](https://discuss.elastic.co/t/measuring-errors-writing-to-es/130886)
**Author:** [@cptcanuck](https://discuss.elastic.co/u/cptcanuck)\ **Replies:** 0\ **Last updated:** [May 7, 2018, 6:05pm UTC](https://discuss.elastic.co/t/measuring-errors-writing-to-es/130886 "2018-05-07T18:05:47Z")
I run a pretty open date pipeline for my company, allowing users to manage their own data pretty well. Unfortunately, this leads to devs releasing code that changes field types on me, introducing document mapping except… --- ## [Translate filter is not working](https://discuss.elastic.co/t/translate-filter-is-not-working/130526)
**Author:** [@bijay](https://discuss.elastic.co/u/bijay)\ **Replies:** 4\ **Last updated:** [May 7, 2018, 5:38pm UTC](https://discuss.elastic.co/t/translate-filter-is-not-working/130526 "2018-05-07T17:38:38Z")
Logstash Version - 6.6.2 Translate Filter Version - 3.0.4 Drop Filter Version - 3.0.5 logstash-config ============= input { stdin { codec =\> json } } filter { translate { field =\> "comp" dictionary\_path =\> "/e… --- ## [S3 output creating invalid json from CSV](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843)
**Author:** [@mememe](https://discuss.elastic.co/u/mememe)\ **Replies:** 3\ **Last updated:** [May 7, 2018, 5:37pm UTC](https://discuss.elastic.co/t/s3-output-creating-invalid-json-from-csv/130843 "2018-05-07T17:37:08Z")
Hi, using s3 output to read from a csv file and store it as json. My logstash conf looks like below. Problem: The output is not a valid json array consisting of json objects but a file having a list of json objects in i… --- ## [Logstash crashing with "exception"=\>"undefined method \`tr' for 0:Fixnum",](https://discuss.elastic.co/t/logstash-crashing-with-exception-undefined-method-tr-for-0-fixnum/129742)
**Author:** [@s\_yash](https://discuss.elastic.co/u/s_yash)\ **Replies:** 4\ **Last updated:** [May 7, 2018, 4:23pm UTC](https://discuss.elastic.co/t/logstash-crashing-with-exception-undefined-method-tr-for-0-fixnum/129742 "2018-05-07T16:23:41Z")
Logstash crashing while converting value to float. See below for config file and exception. Has anyone seen this error and is there a workaround. version: logstash 5.6.9 conf file: filter { if \[type\] == "metric" {… --- ## [Start up logstash](https://discuss.elastic.co/t/start-up-logstash/130852)
**Author:** [@MrNerd](https://discuss.elastic.co/u/MrNerd)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 3:12pm UTC](https://discuss.elastic.co/t/start-up-logstash/130852 "2018-05-07T15:12:59Z")
Hy there! I have already install logstash 6.2 version and try to start it and it shows an error root@Logstash:~# service logstash status -l ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.… --- ## [Logstash s3 input, ignore old files](https://discuss.elastic.co/t/logstash-s3-input-ignore-old-files/130853)
**Author:** [@dhume](https://discuss.elastic.co/u/dhume)\ **Replies:** 0\ **Last updated:** [May 7, 2018, 2:59pm UTC](https://discuss.elastic.co/t/logstash-s3-input-ignore-old-files/130853 "2018-05-07T14:59:37Z")
Hello, I'm setting up Logstash to pull in from an s3 bucket. Within this prefix there is a lot of files but I'm only interested in pulling data from this point forward. Is there a way to have Logstash ingest the data wi… --- ## [Code indexation mail to elasticsearch](https://discuss.elastic.co/t/code-indexation-mail-to-elasticsearch/130846)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 0\ **Last updated:** [May 7, 2018, 2:21pm UTC](https://discuss.elastic.co/t/code-indexation-mail-to-elasticsearch/130846 "2018-05-07T14:21:00Z")
hello, I want to make a filter to pull some information from each email like (sender , @IP , text , name of the company etc ... I heard about grok filter but I can not use it well but since we can work with any computer… --- ## [How to use Ruby specific modules in Logstash conf file](https://discuss.elastic.co/t/how-to-use-ruby-specific-modules-in-logstash-conf-file/130819)
**Author:** [@santublr](https://discuss.elastic.co/u/santublr)\ **Replies:** 1\ **Last updated:** [May 7, 2018, 1:13pm UTC](https://discuss.elastic.co/t/how-to-use-ruby-specific-modules-in-logstash-conf-file/130819 "2018-05-07T13:13:25Z")
Here is the sample conf file. I need to capture each event and do pre-process for that I need to use date and digest/md5 module of ruby. filter { ruby { init =\> " require 'digest/md5' require 'date' " code =\> " … --- ## [Problem indexation mails to elasticsearch with logstash](https://discuss.elastic.co/t/problem-indexation-mails-to-elasticsearch-with-logstash/130211)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 1:13pm UTC](https://discuss.elastic.co/t/problem-indexation-mails-to-elasticsearch-with-logstash/130211 "2018-05-07T13:13:09Z")
hi everyone please i need your help i got verytime this error when i run logstash i have no idea why !! error : "from" =\> "Codecademy learn@codecademy.com", "@version" =\> "1", "content-type" =\> "multipart/alternativ… --- ## [Combine multiple lines into one variable](https://discuss.elastic.co/t/combine-multiple-lines-into-one-variable/130833)
**Author:** [@Petersiemens](https://discuss.elastic.co/u/Petersiemens)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 1:12pm UTC](https://discuss.elastic.co/t/combine-multiple-lines-into-one-variable/130833 "2018-05-07T13:12:09Z")
Hi, i am searching for a good solution to prozess different form of events in the same log files. I could write if{}elseif{} for every possible event, but i hope there is a better way. Some of my logs look like: Ereign… --- ## [MySQL string field to ip using Logstash](https://discuss.elastic.co/t/mysql-string-field-to-ip-using-logstash/130816)
**Author:** [@RRSR](https://discuss.elastic.co/u/RRSR)\ **Replies:** 5\ **Last updated:** [May 7, 2018, 12:40pm UTC](https://discuss.elastic.co/t/mysql-string-field-to-ip-using-logstash/130816 "2018-05-07T12:40:28Z")
Hi, I am using Logstash to create ElasticSearch index from MySQL data using jdbc plugin. There a field named ip in my MySQL which in ES index is coming as String. I want this field to be of datatype "ip". logstash.conf … --- ## [Need help on to extact JSON data from filed and send as multiple events](https://discuss.elastic.co/t/need-help-on-to-extact-json-data-from-filed-and-send-as-multiple-events/130755)
**Author:** [@ssuccess2003](https://discuss.elastic.co/u/ssuccess2003)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 11:05am UTC](https://discuss.elastic.co/t/need-help-on-to-extact-json-data-from-filed-and-send-as-multiple-events/130755 "2018-05-07T11:05:02Z")
Hi Team, I need help on the below requirement. I have developed a custome beat which stores powershell JSON output in a string field and send it to log stash. I want to extract this stringified JSON and send it as sepe… --- ## [Can Lumberjack plugin ensure 100% delivery guarantee?](https://discuss.elastic.co/t/can-lumberjack-plugin-ensure-100-delivery-guarantee/130461)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 2\ **Last updated:** [May 7, 2018, 10:52am UTC](https://discuss.elastic.co/t/can-lumberjack-plugin-ensure-100-delivery-guarantee/130461 "2018-05-07T10:52:25Z")
Hi, I am using http input and output plugins to transfer messages from one logstash to another logstash(this logstash send messages to Elasticsearch using elasticsearch plugin). I see nearly 20% of message loss and also… --- ## [How to remove some text in json file using logstash filter plugin? Which plugin to use?](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437)
**Author:** [@omkarr](https://discuss.elastic.co/u/omkarr)\ **Replies:** 7\ **Last updated:** [May 7, 2018, 10:41am UTC](https://discuss.elastic.co/t/how-to-remove-some-text-in-json-file-using-logstash-filter-plugin-which-plugin-to-use/130437 "2018-05-07T10:41:42Z")
We are trying to load cloudtrail logs into elasticsearch using logstash. But the json is in following format. {"Records":\[{"eventVersion":"1.05","userIdentity":{"type":"AWSService","invokedBy":"config.amazonaws.com"}...… --- ## [Twitter input logstash geo:nil](https://discuss.elastic.co/t/twitter-input-logstash-geo-nil/130811)
**Author:** [@gourmax](https://discuss.elastic.co/u/gourmax)\ **Replies:** 0\ **Last updated:** [May 7, 2018, 10:19am UTC](https://discuss.elastic.co/t/twitter-input-logstash-geo-nil/130811 "2018-05-07T10:19:06Z")
I'm using the input pluggin "twitter" for logstash with these follogwing options: keywords =\> \[ "marvel","starwars","elasticsearch","logstash"\] full\_tweet =\> true ignore\_retweets =\> true There is… --- ## [Filter Grok](https://discuss.elastic.co/t/filter-grok/130788)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 6\ **Last updated:** [May 7, 2018, 9:36am UTC](https://discuss.elastic.co/t/filter-grok/130788 "2018-05-07T09:36:46Z")
Please, is there any body who has an example of filter grok or fils .conf that can be use with logs that came from firwalls, I want him to separate all the fields. --- ## [Out of memory error while installing logstash-filter-elasticsearch plugin](https://discuss.elastic.co/t/out-of-memory-error-while-installing-logstash-filter-elasticsearch-plugin/130746)
**Author:** [@nadeemaon](https://discuss.elastic.co/u/nadeemaon)\ **Replies:** 1\ **Last updated:** [May 7, 2018, 6:47am UTC](https://discuss.elastic.co/t/out-of-memory-error-while-installing-logstash-filter-elasticsearch-plugin/130746 "2018-05-07T06:47:01Z")
I am installing logstash-filter-elasticsearch plugin using the command bin/logstash-plugin install logstash-filter-elasticsearch And during the installation getting below error logstash-5.6.2\\bin\\logstash-plugin insta… --- ## [Grok in logstash.conf doesnt work!](https://discuss.elastic.co/t/grok-in-logstash-conf-doesnt-work/130619)
**Author:** [@Swantje](https://discuss.elastic.co/u/Swantje)\ **Replies:** 5\ **Last updated:** [May 7, 2018, 6:43am UTC](https://discuss.elastic.co/t/grok-in-logstash-conf-doesnt-work/130619 "2018-05-07T06:43:35Z")
Hi all together, i need your help because i have to finish a project in my company. I've got this log: Recovery Manager: Release 12.1.0.2.0 - Production on Tue Apr 24 18:30:01 2018 Copyright (c) 1982, 2014, Oracle a… --- ## [Build filter chain](https://discuss.elastic.co/t/build-filter-chain/130753)
**Author:** [@secthor](https://discuss.elastic.co/u/secthor)\ **Replies:** 1\ **Last updated:** [May 6, 2018, 5:47pm UTC](https://discuss.elastic.co/t/build-filter-chain/130753 "2018-05-06T17:47:15Z")
Hi, I'm pretty new to ELK world, and now I'm trying to parse O365 audit log. (I used to used excel, than I heard about the ELK stack) So here is some test data: CreationDate,UserIds,Operations,AuditData 3462-12-12T15:… --- ## [Conditional jdbc streaming and CIDR filter questions](https://discuss.elastic.co/t/conditional-jdbc-streaming-and-cidr-filter-questions/130757)
**Author:** [@counterf](https://discuss.elastic.co/u/counterf)\ **Replies:** 0\ **Last updated:** [May 6, 2018, 5:14pm UTC](https://discuss.elastic.co/t/conditional-jdbc-streaming-and-cidr-filter-questions/130757 "2018-05-06T17:14:11Z")
Hi, Here is my current filter to check malicious IP addresses in a MYSQL database: It is working fine, but I think it needs some tuning... filter { if \[dest\_ip\] and \[src\_ip\] { cidr { add\_tag =\> \[ "internal\_de… --- ## [Logstash http poller Cookie Authentication](https://discuss.elastic.co/t/logstash-http-poller-cookie-authentication/130741)
**Author:** [@Mario\_Morelli](https://discuss.elastic.co/u/Mario_Morelli)\ **Replies:** 0\ **Last updated:** [May 6, 2018, 10:41am UTC](https://discuss.elastic.co/t/logstash-http-poller-cookie-authentication/130741 "2018-05-06T10:41:07Z")
Hi There I would just like some pointers regarding using the http poller plugin Logstash 5 & 6 tested with same issue We have an application with a REST API, when you open up the URL it pops up with an authentication … --- ## [Logstash pauses on import and needs manual intervention](https://discuss.elastic.co/t/logstash-pauses-on-import-and-needs-manual-intervention/130529)
**Author:** [@Consolodated\_Emails](https://discuss.elastic.co/u/Consolodated_Emails)\ **Replies:** 2\ **Last updated:** [May 5, 2018, 9:49pm UTC](https://discuss.elastic.co/t/logstash-pauses-on-import-and-needs-manual-intervention/130529 "2018-05-05T21:49:02Z")
This is a bit of a weird issue. I'm using Logstash to ingest gziped CSVs from S3 into Elasticsearch. It's a known issue that our data has hidden ASCII control characters (EOF, Pause, Break, etc) in it. I'm not sure if t… --- ## [Logstash stops collecting data](https://discuss.elastic.co/t/logstash-stops-collecting-data/130714)
**Author:** [@peter1992](https://discuss.elastic.co/u/peter1992)\ **Replies:** 0\ **Last updated:** [May 5, 2018, 6:51pm UTC](https://discuss.elastic.co/t/logstash-stops-collecting-data/130714 "2018-05-05T18:51:36Z")
Has anyone experienced an issue with v6.0.0 where pushing events from a web application to logstash and about 1 hour into the processing logstash completely stops collecting and there are no internal logs to suggest why.… --- ## [Dashboard using stored procedures](https://discuss.elastic.co/t/dashboard-using-stored-procedures/130712)
**Author:** [@Eduardo\_Flores](https://discuss.elastic.co/u/Eduardo_Flores)\ **Replies:** 0\ **Last updated:** [May 5, 2018, 5:15pm UTC](https://discuss.elastic.co/t/dashboard-using-stored-procedures/130712 "2018-05-05T17:15:25Z")
I have a stored procedure that receives two parameters on sql server and I'd like to create a dashboard in kibana that shows the result that the process throws... how could I do this? The parameters would be the date ran… --- ## [Failed to parse \[date\]](https://discuss.elastic.co/t/failed-to-parse-date/130563)
**Author:** [@lcguy](https://discuss.elastic.co/u/lcguy)\ **Replies:** 5\ **Last updated:** [May 5, 2018, 10:26am UTC](https://discuss.elastic.co/t/failed-to-parse-date/130563 "2018-05-05T10:26:53Z")
I hope someone can guide me as for date parsing. Logstash config has below date format. date { match =\> \[ "@date", "yyyy-MM-dd HH:mm:ss Z" \] } But, I am getting below error... "status"=\>400, "error"=\>{"type"=\>"mapp… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=764) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=766)