# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=856 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 857 --- ## [Replacing @timestamp with custom date](https://discuss.elastic.co/t/replacing-timestamp-with-custom-date/111781)
**Author:** [@munjadd](https://discuss.elastic.co/u/munjadd)\ **Replies:** 2\ **Last updated:** [December 14, 2017, 4:05pm UTC](https://discuss.elastic.co/t/replacing-timestamp-with-custom-date/111781 "2017-12-14T16:05:18Z")
Hi, I am using filebeat to send lines from a dhcp log (from windows server dhcp), and i am trying to replace the @timestamp field. Here is an example line from the dhcp log file: 11,12/14/17,14:22:04,Renew,10.1.65.60,p… --- ## [Errors after Elastic 6](https://discuss.elastic.co/t/errors-after-elastic-6/107776)
**Author:** [@ocabj](https://discuss.elastic.co/u/ocabj)\ **Replies:** 13\ **Last updated:** [December 14, 2017, 4:03pm UTC](https://discuss.elastic.co/t/errors-after-elastic-6/107776 "2017-12-14T16:03:39Z")
I did an upgrade on my personal ELK stack to 6 yesterday and everything was fine until 1600PST / 0000 UTC. \[2017-11-14T16:00:05,887\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:statu… --- ## [AH!--Websites being setup as fields--KV plugin](https://discuss.elastic.co/t/ah-websites-being-setup-as-fields-kv-plugin/111799)
**Author:** [@Daniel\_Sanchez](https://discuss.elastic.co/u/Daniel_Sanchez)\ **Replies:** 2\ **Last updated:** [December 14, 2017, 3:45pm UTC](https://discuss.elastic.co/t/ah-websites-being-setup-as-fields-kv-plugin/111799 "2017-12-14T15:45:03Z")
Oh wow, my number of fields blew up last night after I added a new server pushing logs to our logstash server. For some reason it's taking all of websites and setting those as fields. I think it's related to the KV plu… --- ## [Date parsing failed](https://discuss.elastic.co/t/date-parsing-failed/111739)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [December 14, 2017, 2:11pm UTC](https://discuss.elastic.co/t/date-parsing-failed/111739 "2017-12-14T14:11:25Z")
Hello i have an oracle log file and i need to parse the date in a field named backup\_start the date format in this field is like this 04-12-2017 20:30 and my config is date { match =\> \["backup\_start… --- ## [Recommendation for sizing Logstash machines on AWS](https://discuss.elastic.co/t/recommendation-for-sizing-logstash-machines-on-aws/111777)
**Author:** [@johnjingle](https://discuss.elastic.co/u/johnjingle)\ **Replies:** 0\ **Last updated:** [December 14, 2017, 1:31pm UTC](https://discuss.elastic.co/t/recommendation-for-sizing-logstash-machines-on-aws/111777 "2017-12-14T13:31:11Z")
Hi, A good link for Elasticsearch on AWS https://www.elastic.co/blog/running-elasticsearch-on-aws. Looking for the same advice for Logstash please. We have lots of data to process from a database. --- ## [Parsing multiple static cloudtrail files with json codec](https://discuss.elastic.co/t/parsing-multiple-static-cloudtrail-files-with-json-codec/111403)
**Author:** [@jgwentworth](https://discuss.elastic.co/u/jgwentworth)\ **Replies:** 1\ **Last updated:** [December 14, 2017, 1:11pm UTC](https://discuss.elastic.co/t/parsing-multiple-static-cloudtrail-files-with-json-codec/111403 "2017-12-14T13:11:14Z")
Hi, I'm running into an issue where I can parse individual cloud trail files but cannot ingest multiple files with the same json structure. file { path =\> "/home/\*\*/\*.json" type =\> "cloudtrail" codec =\> json { chars… --- ## [Would multiple events have same @timestamp when processed in batch](https://discuss.elastic.co/t/would-multiple-events-have-same-timestamp-when-processed-in-batch/111724)
**Author:** [@Dimitry\_Khan](https://discuss.elastic.co/u/Dimitry_Khan)\ **Replies:** 2\ **Last updated:** [December 14, 2017, 1:06pm UTC](https://discuss.elastic.co/t/would-multiple-events-have-same-timestamp-when-processed-in-batch/111724 "2017-12-14T13:06:21Z")
How logstash creates the @timestamp value for each record? So if I configure logstash.yml to process multiple events in batch all the events that are sampled in one batch should have same @timstamp. I have tested this, a… --- ## [Message body on failure](https://discuss.elastic.co/t/message-body-on-failure/111762)
**Author:** [@heilgar](https://discuss.elastic.co/u/heilgar)\ **Replies:** 0\ **Last updated:** [December 14, 2017, 12:06pm UTC](https://discuss.elastic.co/t/message-body-on-failure/111762 "2017-12-14T12:06:29Z")
I have logstash with elasticsearch HTTP i/o plugin. And on production, I often have errors that smth cant updated by \_update\_by\_query. And seems all are good with logging to logstash-plain.log but have I possibility to … --- ## [How to try many grok matches, and take only the one that works?](https://discuss.elastic.co/t/how-to-try-many-grok-matches-and-take-only-the-one-that-works/111743)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 3\ **Last updated:** [December 14, 2017, 11:54am UTC](https://discuss.elastic.co/t/how-to-try-many-grok-matches-and-take-only-the-one-that-works/111743 "2017-12-14T11:54:30Z")
Hi everyone! I have 10 types of logs I need to process. They differ from each other but have some similarities, so I built a grok filter that matches each one, and non of the other types. How do I write it into the .con… --- ## [New to Logstash and Pugins - Can anyone help?](https://discuss.elastic.co/t/new-to-logstash-and-pugins-can-anyone-help/111726)
**Author:** [@hmpmarketing](https://discuss.elastic.co/u/hmpmarketing)\ **Replies:** 1\ **Last updated:** [December 14, 2017, 11:44am UTC](https://discuss.elastic.co/t/new-to-logstash-and-pugins-can-anyone-help/111726 "2017-12-14T11:44:04Z")
Hi everyone, Im just got started with logstash and having a hard time understanding how I should setup a pipeline. I am sending from my app a json encoded string that is getting stored in a redis list, my objetive is to… --- ## [Reindex with Logstash (Elasticsearch --\> Filter --\> Elasticsearch) looses data on the way](https://discuss.elastic.co/t/reindex-with-logstash-elasticsearch-filter-elasticsearch-looses-data-on-the-way/110164)
**Author:** [@infomaniac](https://discuss.elastic.co/u/infomaniac)\ **Replies:** 1\ **Last updated:** [December 14, 2017, 11:26am UTC](https://discuss.elastic.co/t/reindex-with-logstash-elasticsearch-filter-elasticsearch-looses-data-on-the-way/110164 "2017-12-14T11:26:01Z")
Hi I am trying to reindex data from a daily index to an daily archive index, while dropping or combining some fields. Easiest way to do this seams to be with LS and elasticsearch input and output with some filters in b… --- ## [Wat is the isju reading files with no new line?](https://discuss.elastic.co/t/wat-is-the-isju-reading-files-with-no-new-line/111515)
**Author:** [@tellus83](https://discuss.elastic.co/u/tellus83)\ **Replies:** 6\ **Last updated:** [December 14, 2017, 9:48am UTC](https://discuss.elastic.co/t/wat-is-the-isju-reading-files-with-no-new-line/111515 "2017-12-14T09:48:44Z")
I have thousands of files with just one line per file. I read these files whit filebeat and send them to logstash. But i do not get any output. If I add a new line to a file I get output form logstash. Where is the prob… --- ## [Not able to get expected output from elasticsearch through logstash pipeline](https://discuss.elastic.co/t/not-able-to-get-expected-output-from-elasticsearch-through-logstash-pipeline/111696)
**Author:** [@somkhil](https://discuss.elastic.co/u/somkhil)\ **Replies:** 3\ **Last updated:** [December 14, 2017, 9:47am UTC](https://discuss.elastic.co/t/not-able-to-get-expected-output-from-elasticsearch-through-logstash-pipeline/111696 "2017-12-14T09:47:56Z")
HI I am running logstash pipeline to pull the data from elasticsearch and output it into csv file. At the very first run it generated output properly. However during subsequent runs (after loading more data/fields into… --- ## [Multiline in logstash](https://discuss.elastic.co/t/multiline-in-logstash/111719)
**Author:** [@sureshboiina](https://discuss.elastic.co/u/sureshboiina)\ **Replies:** 0\ **Last updated:** [December 14, 2017, 9:08am UTC](https://discuss.elastic.co/t/multiline-in-logstash/111719 "2017-12-14T09:08:19Z")
HI Having a file like 2017-12-14 12:45:44.570 \[http-nio-9000-exec-9\] DEBUG o.s.w.f.CommonsRequestLoggingFilter.beforeRequest(47) - Before request \[uri=/gateway/api/v1/fitment-processes/02;client=0:0:0:0:0:0:0:1;user=tc… --- ## [Logstash not producing output although pipeline main starts](https://discuss.elastic.co/t/logstash-not-producing-output-although-pipeline-main-starts/111689)
**Author:** [@khare.garima](https://discuss.elastic.co/u/khare.garima)\ **Replies:** 4\ **Last updated:** [December 14, 2017, 8:57am UTC](https://discuss.elastic.co/t/logstash-not-producing-output-although-pipeline-main-starts/111689 "2017-12-14T08:57:26Z")
I am trying to execute logstash.bat with version 5.6 and 6.0 on a windows 10 machine. Following is the conf file: input { file { path =\> "D:/access\_log113.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/nu… --- ## [Logstash performance v Elastic Java API](https://discuss.elastic.co/t/logstash-performance-v-elastic-java-api/111711)
**Author:** [@johnjingle](https://discuss.elastic.co/u/johnjingle)\ **Replies:** 0\ **Last updated:** [December 14, 2017, 8:32am UTC](https://discuss.elastic.co/t/logstash-performance-v-elastic-java-api/111711 "2017-12-14T08:32:24Z")
Not sure where to ask this. We have 500 million json records to load to Elasticsearch. Which please is the most performant method. Rest API, Logstash, Elastic Java API. I am assuming Logstash is the easiest and best… --- ## [Pipeline cannot start with http\_poller](https://discuss.elastic.co/t/pipeline-cannot-start-with-http-poller/108405)
**Author:** [@Tonni\_Hult](https://discuss.elastic.co/u/Tonni_Hult)\ **Replies:** 11\ **Last updated:** [December 13, 2017, 3:53pm UTC](https://discuss.elastic.co/t/pipeline-cannot-start-with-http-poller/108405 "2017-12-13T15:53:30Z")
Hi I've installed v6.0 and get a problem when using input http\_poller because the pipeline crashes when starting up and the output that I get is: Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipe… --- ## [Cons of having apache kafka between two logstashes](https://discuss.elastic.co/t/cons-of-having-apache-kafka-between-two-logstashes/111628)
**Author:** [@vb4t](https://discuss.elastic.co/u/vb4t)\ **Replies:** 2\ **Last updated:** [December 14, 2017, 7:17am UTC](https://discuss.elastic.co/t/cons-of-having-apache-kafka-between-two-logstashes/111628 "2017-12-14T07:17:07Z")
Hello. Since logstash 5.4, the document https://www.elastic.co/guide/en/logstash/5.4/deploying-and-scaling.html suggests to have beats directly connected to kafka instead of having "beats -\> logstash shipper -\> kafka -\> … --- ## [Logstash is calculating itself to sudden death](https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505)
**Author:** [@Haenschen](https://discuss.elastic.co/u/Haenschen)\ **Replies:** 1\ **Last updated:** [December 14, 2017, 7:00am UTC](https://discuss.elastic.co/t/logstash-is-calculating-itself-to-sudden-death/111505 "2017-12-14T07:00:47Z")
Hi folks, I need your advice solving following problem: I am going to parse the Logfiles from Akamai CDN using Logstash. In order to do this I'm using the following Logstash config: input { file { max\_open\_files =\> 1… --- ## [Multiple conf File with similar content on different Protocol creats concatenated output](https://discuss.elastic.co/t/multiple-conf-file-with-similar-content-on-different-protocol-creats-concatenated-output/111590)
**Author:** [@sru](https://discuss.elastic.co/u/sru)\ **Replies:** 2\ **Last updated:** [December 14, 2017, 6:15am UTC](https://discuss.elastic.co/t/multiple-conf-file-with-similar-content-on-different-protocol-creats-concatenated-output/111590 "2017-12-14T06:15:18Z")
two different config files are in /etc/logstash/conf.d, one input is on tcp 1514 and the other input on udp 3514. Both files are working well while using it on the command line (one by one) /usr/share/logstash/bin/logst… --- ## [Creating global array in logstash and sharing the same](https://discuss.elastic.co/t/creating-global-array-in-logstash-and-sharing-the-same/111354)
**Author:** [@naveenk](https://discuss.elastic.co/u/naveenk)\ **Replies:** 4\ **Last updated:** [December 14, 2017, 1:55am UTC](https://discuss.elastic.co/t/creating-global-array-in-logstash-and-sharing-the-same/111354 "2017-12-14T01:55:44Z")
Hi, I would like to create global array/map based on jdbc input plugin. And share the same array with in filebeat event. Example(Sample Macro Code) input { jdbc { // contains business logic details } beats { //co… --- ## [Run netflow moudules failed](https://discuss.elastic.co/t/run-netflow-moudules-failed/111236)
**Author:** [@antony.y](https://discuss.elastic.co/u/antony.y)\ **Replies:** 15\ **Last updated:** [December 14, 2017, 1:00am UTC](https://discuss.elastic.co/t/run-netflow-moudules-failed/111236 "2017-12-14T01:00:21Z")
I have tried to using "bin/logstash --modules netlfow --setup" to start the netflow module.But it does not work.The error log is like: \[ERROR\] 2017-12-12 13:39:52.658 \[LogStash::Runner\] kibanaclient - Error when execu… --- ## [Logstash in Docker + syslog + preserving source IP's](https://discuss.elastic.co/t/logstash-in-docker-syslog-preserving-source-ips/111492)
**Author:** [@jjensen](https://discuss.elastic.co/u/jjensen)\ **Replies:** 2\ **Last updated:** [December 13, 2017, 11:32pm UTC](https://discuss.elastic.co/t/logstash-in-docker-syslog-preserving-source-ips/111492 "2017-12-13T23:32:38Z")
Hi, I'm running into a wall when it comes to receiving syslog data from various sources in my network. I think this might be an issue with Docker, but I wanted to get a feel for what others are doing in production befor… --- ## [Merging @timestamp with actual log date](https://discuss.elastic.co/t/merging-timestamp-with-actual-log-date/111578)
**Author:** [@mad\_dog](https://discuss.elastic.co/u/mad_dog)\ **Replies:** 1\ **Last updated:** [December 13, 2017, 11:17pm UTC](https://discuss.elastic.co/t/merging-timestamp-with-actual-log-date/111578 "2017-12-13T23:17:55Z")
Hi, I'm new to logstash so maybe my question will not make much sense but I have this problem. I have servers in multiple geographical locations. They send logs with log time that doesn't include year. At the moment I u… --- ## [Use only one input for one output. How to segregate/isolate pipelines definitions input/output?](https://discuss.elastic.co/t/use-only-one-input-for-one-output-how-to-segregate-isolate-pipelines-definitions-input-output/111466)
**Author:** [@bizmate](https://discuss.elastic.co/u/bizmate)\ **Replies:** 2\ **Last updated:** [December 13, 2017, 10:36pm UTC](https://discuss.elastic.co/t/use-only-one-input-for-one-output-how-to-segregate-isolate-pipelines-definitions-input-output/111466 "2017-12-13T22:36:17Z")
I have a pipeline definition to query ES and get just a specific set of data exported to files that I will process with an external application. Pipeline looks like input { elasticsearch { hosts =\> "elasticsearch:9200… --- ## [Filebeat is not sending data to logstash from windows client](https://discuss.elastic.co/t/filebeat-is-not-sending-data-to-logstash-from-windows-client/111651)
**Author:** [@Surya\_Indukuri](https://discuss.elastic.co/u/Surya_Indukuri)\ **Replies:** 1\ **Last updated:** [December 13, 2017, 10:25pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-data-to-logstash-from-windows-client/111651 "2017-12-13T22:25:57Z")
Continuing the discussion from Filebeat is not sending logs to logstash: Hi Guys , I have installed kibana elasticsearch logstash in cent os but filebeat is not sending data to logstash ###################### Filebea… --- ## [IOException: Too many open files - TCP input](https://discuss.elastic.co/t/ioexception-too-many-open-files-tcp-input/107547)
**Author:** [@flalar](https://discuss.elastic.co/u/flalar)\ **Replies:** 17\ **Last updated:** [December 13, 2017, 9:50pm UTC](https://discuss.elastic.co/t/ioexception-too-many-open-files-tcp-input/107547 "2017-12-13T21:50:59Z")
We're getting the following exception when using Logstash tcp input. Elastic stack running 5.6.4 on Centos 7.4 \[2017-11-10T23:59:58,325\]\[WARN \]\[io.netty.channel.DefaultChannelPipeline\] An exceptionCaught() event was fi… --- ## [Elasticsearch - Encountered a retryable error. Will Retry with exponential backoff](https://discuss.elastic.co/t/elasticsearch-encountered-a-retryable-error-will-retry-with-exponential-backoff/111646)
**Author:** [@kdivecha](https://discuss.elastic.co/u/kdivecha)\ **Replies:** 0\ **Last updated:** [December 13, 2017, 8:47pm UTC](https://discuss.elastic.co/t/elasticsearch-encountered-a-retryable-error-will-retry-with-exponential-backoff/111646 "2017-12-13T20:47:50Z")
I keep seeing the following error in logs, still loads all the data, just wondering what might be causing it. I am ingesting into ES via ELB on AWS, running multiple logstash threads on a dedicated instance. \[ERROR\] 201… --- ## [Error occurred! {:error=\>#\](https://discuss.elastic.co/t/error-occurred-error-nomethoderror-undefined-method-for-nil-nilclass/111609)
**Author:** [@Roberto\_Javier\_Yudic](https://discuss.elastic.co/u/Roberto_Javier_Yudic)\ **Replies:** 1\ **Last updated:** [December 13, 2017, 8:12pm UTC](https://discuss.elastic.co/t/error-occurred-error-nomethoderror-undefined-method-for-nil-nilclass/111609 "2017-12-13T20:12:05Z")
I'm getting this error using lostash 6.0.1 docker image. The pipeline configuration I'm using is: input { #udp { # port =\> 5000 # codec =\> json #} tcp { port =\> 5000 codec =\> json } } filter { … --- ## [Aggregate plugin + event.get('@timestamp')](https://discuss.elastic.co/t/aggregate-plugin-event-get-timestamp/109945)
**Author:** [@wsteven](https://discuss.elastic.co/u/wsteven)\ **Replies:** 4\ **Last updated:** [December 13, 2017, 7:17pm UTC](https://discuss.elastic.co/t/aggregate-plugin-event-get-timestamp/109945 "2017-12-13T19:17:27Z")
Hello there, I must be doing it wrong :slightly\_smiling\_face: aggregate { task\_id =\> "%{clientip}" code =\> "map\['timestamp'\] ||= event.get\['@timestamp'\].to\_i" push\_map\_as\_event\_on\_timeout =\> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=855) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=857)