# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1007 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1008 --- ## [Kafka input slow stop consuming](https://discuss.elastic.co/t/kafka-input-slow-stop-consuming/80867)
**Author:** [@danielmotaleite](https://discuss.elastic.co/u/danielmotaleite)\ **Replies:** 0\ **Last updated:** [March 31, 2017, 7:26pm UTC](https://discuss.elastic.co/t/kafka-input-slow-stop-consuming/80867 "2017-03-31T19:26:49Z")
Since a few weeks ago, my yellow topic is having problems being consumed, the logstash consumers seem to stop. If i restart logstash, it starts consuming again, but slowly decrease the speed until it stops. I setup a sin… --- ## [\*beat -\> logstash -\> logstash](https://discuss.elastic.co/t/beat-logstash-logstash/63924)
**Author:** [@tbmorris78](https://discuss.elastic.co/u/tbmorris78)\ **Replies:** 6\ **Last updated:** [March 31, 2017, 7:12pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924 "2017-03-31T19:12:11Z")
I have been playing with ELK stack for a month or so and seem to have it working quite well at a single site, but I am running into issues of how to move forward. I have several sites with logstash that I would like to… --- ## [Help need to create filter](https://discuss.elastic.co/t/help-need-to-create-filter/80851)
**Author:** [@Giri](https://discuss.elastic.co/u/Giri)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 5:25pm UTC](https://discuss.elastic.co/t/help-need-to-create-filter/80851 "2017-03-31T17:25:54Z")
HI, I am newbie to Logstash, trying out few options. I need help in creating my filter to store the values in the line #5 of log file as Output required below CONFIG USED input { file { path =\> "D:/Log.txt" … --- ## [Input stdin and snmptrap?](https://discuss.elastic.co/t/input-stdin-and-snmptrap/80533)
**Author:** [@Dark\_Light](https://discuss.elastic.co/u/Dark_Light)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 5:13pm UTC](https://discuss.elastic.co/t/input-stdin-and-snmptrap/80533 "2017-03-31T17:13:14Z")
I am using stdin for testing of the logstash .conf files and need to input some snmptrap messages received from splunk in json format. The first roadblock seems to be I cannot use snmptrap input as a sub input from std… --- ## [Error on update Logstash 5.2 =\> 5.3](https://discuss.elastic.co/t/error-on-update-logstash-5-2-5-3/80740)
**Author:** [@zokratez](https://discuss.elastic.co/u/zokratez)\ **Replies:** 2\ **Last updated:** [March 31, 2017, 5:00pm UTC](https://discuss.elastic.co/t/error-on-update-logstash-5-2-5-3/80740 "2017-03-31T17:00:42Z")
After the update I get the following error: \[2017-03-30T17:17:21,667\]\[ERROR\]\[logstash.filters.kv \] Unknown setting 'trim' for kv \[2017-03-30T17:17:21,692\]\[ERROR\]\[logstash.agent \] Cannot load an invalid … --- ## [Elapsed filter. Broken?](https://discuss.elastic.co/t/elapsed-filter-broken/80381)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 4:58pm UTC](https://discuss.elastic.co/t/elapsed-filter-broken/80381 "2017-03-31T16:58:17Z")
Hi, One of the most useful filters for my use case is 'elapsed'. We made heavy use of it in our older Logstash configurations. However in our testing with LS 2.3.x and 2.4.x the behaviour of the elapsed filter is, at… --- ## [\_grokparsefailure with date, but not without it](https://discuss.elastic.co/t/-grokparsefailure-with-date-but-not-without-it/80755)
**Author:** [@yacovm](https://discuss.elastic.co/u/yacovm)\ **Replies:** 3\ **Last updated:** [March 31, 2017, 4:28pm UTC](https://discuss.elastic.co/t/-grokparsefailure-with-date-but-not-without-it/80755 "2017-03-31T16:28:52Z")
Hi all. I have an ELK container and I send logs to it using filebeats. root@20e13712ecb2:/# cat /etc/logstash/conf.d/02-beats-input.conf input { beats { port =\> 5044 ssl =\> false ssl\_certificate =\> "/etc/pki/tls/… --- ## [Logstash jdbc query issue](https://discuss.elastic.co/t/logstash-jdbc-query-issue/48725)
**Author:** [@docjay](https://discuss.elastic.co/u/docjay)\ **Replies:** 17\ **Last updated:** [March 31, 2017, 2:20pm UTC](https://discuss.elastic.co/t/logstash-jdbc-query-issue/48725 "2017-03-31T14:20:36Z")
I have the latest ELK stack installed and all three are communicating correctly. I'm trying to get logstash to query a 'mysql' database. my logstash.conf file: input { jdbc { jdbc\_driver\_library =\> "/opt/logstash/j… --- ## [Add header row to CSV output](https://discuss.elastic.co/t/add-header-row-to-csv-output/50563)
**Author:** [@astickler](https://discuss.elastic.co/u/astickler)\ **Replies:** 4\ **Last updated:** [March 31, 2017, 2:38pm UTC](https://discuss.elastic.co/t/add-header-row-to-csv-output/50563 "2017-03-31T14:38:29Z")
Hi, I need to create date-specific CSV files with logstash, but can't see how to write a header row. Is this even possible? Thanks, Andrew --- ## [LogStash-- Parsing A log file](https://discuss.elastic.co/t/logstash-parsing-a-log-file/80552)
**Author:** [@Riaz\_Mohamed](https://discuss.elastic.co/u/Riaz_Mohamed)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 2:24pm UTC](https://discuss.elastic.co/t/logstash-parsing-a-log-file/80552 "2017-03-31T14:24:40Z")
All I am looking at parsing a log file and inserting it into a MySQL Database. The output works but there seems to be a problem with the date pattern The following is my log Feb 15, 2017 09:35:43 AM GMT, INFO - INVALID\_… --- ## [Change a log value](https://discuss.elastic.co/t/change-a-log-value/80807)
**Author:** [@rabbit](https://discuss.elastic.co/u/rabbit)\ **Replies:** 3\ **Last updated:** [March 31, 2017, 2:21pm UTC](https://discuss.elastic.co/t/change-a-log-value/80807 "2017-03-31T14:21:36Z")
Hi guys, I have a log file that have a field named "Time", this field is filled sometimes with the word "IN" and other times with numbers that correspond to the time in milliseconds. I find a way to convert the word t… --- ## [Logstash slow processing](https://discuss.elastic.co/t/logstash-slow-processing/80815)
**Author:** [@ankush\_goswami](https://discuss.elastic.co/u/ankush_goswami)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 1:45pm UTC](https://discuss.elastic.co/t/logstash-slow-processing/80815 "2017-03-31T13:45:20Z")
Hey Hi Team, Want to know the current number of queued events in logstash using an http server being introduced in logstash 5.0. Aside to this I can see my logstash processing slow. I want to know the reason behind thi… --- ## [How to resolve jar dependencies in logstash](https://discuss.elastic.co/t/how-to-resolve-jar-dependencies-in-logstash/80245)
**Author:** [@vaishnavi](https://discuss.elastic.co/u/vaishnavi)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 1:31pm UTC](https://discuss.elastic.co/t/how-to-resolve-jar-dependencies-in-logstash/80245 "2017-03-31T13:31:31Z")
Hi I am using jruby 1.7.25 and centos 7.0 I am not able to resolve jar dependency while importing custom plugins in logstash..I also included gems ruby-maven,jar-dependencies.Did I need to include any new plugins \[ER… --- ## [DNS filter not working](https://discuss.elastic.co/t/dns-filter-not-working/77315)
**Author:** [@nigdav007](https://discuss.elastic.co/u/nigdav007)\ **Replies:** 4\ **Last updated:** [March 31, 2017, 1:29pm UTC](https://discuss.elastic.co/t/dns-filter-not-working/77315 "2017-03-31T13:29:42Z")
Hay all I am an tad stuck with the DNS filter. My issue all my syslogs hosts are coming up with the ip address and not the hostname. I like to use the DNS filter to get the hostname, but for some reason its not worki… --- ## [Got error to send bulk of actions: no method 'id' for arguments](https://discuss.elastic.co/t/got-error-to-send-bulk-of-actions-no-method-id-for-arguments/79941)
**Author:** [@suneesh](https://discuss.elastic.co/u/suneesh)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 1:13pm UTC](https://discuss.elastic.co/t/got-error-to-send-bulk-of-actions-no-method-id-for-arguments/79941 "2017-03-31T13:13:57Z")
I'm trying to read from oracle DB and load to EL Search using Log Stash. Here is the output file output { elasticsearch { index =\>"planogramviews" document\_type =\> "plano… --- ## [Probleme input UDP , no index, no DATA](https://discuss.elastic.co/t/probleme-input-udp-no-index-no-data/79883)
**Author:** [@pierre1](https://discuss.elastic.co/u/pierre1)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 1:11pm UTC](https://discuss.elastic.co/t/probleme-input-udp-no-index-no-data/79883 "2017-03-31T13:11:31Z")
my configuration : input { udp { port =\> 9556 codec =\> netflow { versions =\> \[5\] } type =\> "Netflow" } } output { if ( \[type\] == "Netflow" ) { elasticsearch { hos… --- ## [S3 input "include pattern"?](https://discuss.elastic.co/t/s3-input-include-pattern/79949)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 1:03pm UTC](https://discuss.elastic.co/t/s3-input-include-pattern/79949 "2017-03-31T13:03:30Z")
I have many files in my s3 bucket, is there a way to specify the pattern I want to include? They are in several different sub-buckets, so i don't think i can use prefix.. --- ## [Connection is refused with Influxdb](https://discuss.elastic.co/t/connection-is-refused-with-influxdb/42409)
**Author:** [@arok](https://discuss.elastic.co/u/arok)\ **Replies:** 2\ **Last updated:** [March 31, 2017, 12:51pm UTC](https://discuss.elastic.co/t/connection-is-refused-with-influxdb/42409 "2017-03-31T12:51:28Z")
Hi!! Connection impossible to Influxdb when using Logstash Script. My Logstash plugin Output influxdb { host =\> "192.168.3.123" port =\> "8086" db =\> "ceticdb" pass… --- ## [Response Headers http\_poller](https://discuss.elastic.co/t/response-headers-http-poller/80739)
**Author:** [@jcannell](https://discuss.elastic.co/u/jcannell)\ **Replies:** 4\ **Last updated:** [March 31, 2017, 12:17pm UTC](https://discuss.elastic.co/t/response-headers-http-poller/80739 "2017-03-31T12:17:41Z")
Is it possible to access the response headers from events processed by the http\_poller input plugin? I'm working with the Microsoft Table Service REST API: https://docs.microsoft.com/en-us/rest/api/storageservices/filese… --- ## [This is the error im getting with $ sudo apt-get update](https://discuss.elastic.co/t/this-is-the-error-im-getting-with-sudo-apt-get-update/80634)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 4\ **Last updated:** [March 31, 2017, 12:16pm UTC](https://discuss.elastic.co/t/this-is-the-error-im-getting-with-sudo-apt-get-update/80634 "2017-03-31T12:16:29Z")
W: Duplicate sources.list entry http://packages.elastic.co/logstash/2.2/debian/ stable/main i386 Packages (/var/lib/apt/lists/packages.elastic.co\_logstash\_2.2\_debian\_dists\_stable\_main\_binary-i386\_Packages) W: Duplicate s… --- ## [Set target of xml filter to root?](https://discuss.elastic.co/t/set-target-of-xml-filter-to-root/80770)
**Author:** [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 11:42am UTC](https://discuss.elastic.co/t/set-target-of-xml-filter-to-root/80770 "2017-03-31T11:42:14Z")
I want the store\_xml setting of the xml filter to store child elements of the input XML event as top-level fields. Is this possible? Or should I create an issue in GitHub for this? For example, given this input XML e… --- ## [Logstash not getting log files properly](https://discuss.elastic.co/t/logstash-not-getting-log-files-properly/80787)
**Author:** [@DharaniKumar](https://discuss.elastic.co/u/DharaniKumar)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 11:37am UTC](https://discuss.elastic.co/t/logstash-not-getting-log-files-properly/80787 "2017-03-31T11:37:56Z")
Kibana is not showing any of my logs. Im having this in my logstash config file logstash-syslog.conf : input { file { #port =\> 5044 path =\> \[ "/var/log/\*.log", "/var/log/syslog" \] type =\> "syslog" #… --- ## [Create and add to a nested fields based on previous data in logstash](https://discuss.elastic.co/t/create-and-add-to-a-nested-fields-based-on-previous-data-in-logstash/80793)
**Author:** [@Divya\_Bhardwaj](https://discuss.elastic.co/u/Divya_Bhardwaj)\ **Replies:** 0\ **Last updated:** [March 31, 2017, 10:03am UTC](https://discuss.elastic.co/t/create-and-add-to-a-nested-fields-based-on-previous-data-in-logstash/80793 "2017-03-31T10:03:15Z")
Hi, I have a use case in which I have a data set with "Account Name" , "city", "country". My input is a .csv file. Example data: AccountName city country ABC CA GH XYZ … --- ## [Logstash can judge will be output to the elasticsearch index exists](https://discuss.elastic.co/t/logstash-can-judge-will-be-output-to-the-elasticsearch-index-exists/80778)
**Author:** [@qin](https://discuss.elastic.co/u/qin)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 9:01am UTC](https://discuss.elastic.co/t/logstash-can-judge-will-be-output-to-the-elasticsearch-index-exists/80778 "2017-03-31T09:01:03Z")
Logstash can judge will be output to the elasticsearch index exists？ --- ## [How to use script parameters of elasticsearch output plugin？](https://discuss.elastic.co/t/how-to-use-script-parameters-of-elasticsearch-output-plugin/80783)
**Author:** [@qin](https://discuss.elastic.co/u/qin)\ **Replies:** 0\ **Last updated:** [March 31, 2017, 8:04am UTC](https://discuss.elastic.co/t/how-to-use-script-parameters-of-elasticsearch-output-plugin/80783 "2017-03-31T08:04:28Z")
Lostash elasticsearch output plugin, how to use script、scripted\_upsert and script\_type parameters? Can give an example? --- ## [Date Issue in EL](https://discuss.elastic.co/t/date-issue-in-el/80736)
**Author:** [@suneesh](https://discuss.elastic.co/u/suneesh)\ **Replies:** 1\ **Last updated:** [March 31, 2017, 7:08am UTC](https://discuss.elastic.co/t/date-issue-in-el/80736 "2017-03-31T07:08:18Z")
While loading data from table in oracle to EL Search, date column in oracle is stored as string EL Search . how do I fix this. select.sql=SELECT \* FROM us\_wm\_space.planogram\_view\_log input { jdbc { … --- ## [404 Error](https://discuss.elastic.co/t/404-error/80768)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 2\ **Last updated:** [March 31, 2017, 7:03am UTC](https://discuss.elastic.co/t/404-error/80768 "2017-03-31T07:03:14Z")
I keep getting the following error but I can still see that data is going to my target elasticsearch server. Anyone know what's going on? {"status":404,"reason":"SearchContextMissingException\[No search context found fo… --- ## [Dropping logs in logstash](https://discuss.elastic.co/t/dropping-logs-in-logstash/80665)
**Author:** [@Rupesh\_Ranjan](https://discuss.elastic.co/u/Rupesh_Ranjan)\ **Replies:** 4\ **Last updated:** [March 31, 2017, 6:40am UTC](https://discuss.elastic.co/t/dropping-logs-in-logstash/80665 "2017-03-31T06:40:36Z")
Is there any way where i can discard some logs from a log file in case, they match certain query? for example, i have a log file test-sample.log which has these logs: 1. abc failed due to something 2. warning is genera… --- ## [Logstash to ElasticSearch Throughput](https://discuss.elastic.co/t/logstash-to-elasticsearch-throughput/80361)
**Author:** [@wpearson4](https://discuss.elastic.co/u/wpearson4)\ **Replies:** 5\ **Last updated:** [March 31, 2017, 5:58am UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-throughput/80361 "2017-03-31T05:58:05Z")
I have log stash setup to read my csv files and send them to ES. The csv files are ~6.5 gb each, with ~4.7 million records per csv file and 400 columns per record. I am able to load about 2 million records/hr. Current… --- ## [Log file without Year](https://discuss.elastic.co/t/log-file-without-year/80222)
**Author:** [@chic\_cat](https://discuss.elastic.co/u/chic_cat)\ **Replies:** 1\ **Last updated:** [March 28, 2017, 5:32am UTC](https://discuss.elastic.co/t/log-file-without-year/80222 "2017-03-28T05:32:11Z")
Hi guys, I have logs in format: Mar 23 11:54:35 server \[Controller\] \[Thread-530\] ERROR Connection error I tried to use next filter, but it doesn't work, filter { if \[type\] == "data" { grok { match =\> { "mes… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1006) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1008)