# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=828 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 829 --- ## [How to get grok filter from variable](https://discuss.elastic.co/t/how-to-get-grok-filter-from-variable/118511)
**Author:** [@jfur](https://discuss.elastic.co/u/jfur)\ **Replies:** 2\ **Last updated:** [February 5, 2018, 10:07pm UTC](https://discuss.elastic.co/t/how-to-get-grok-filter-from-variable/118511 "2018-02-05T22:07:13Z")
I have somewhat of a complex use case where some events have data in their firewall "msg" field that needs to be parsed out (most don't). I have a dictionary file that is parsing out just fine with the kv plugin, but am … --- ## [Logstash - csv output plugin - path - today's timestamp filename](https://discuss.elastic.co/t/logstash-csv-output-plugin-path-todays-timestamp-filename/118498)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 1\ **Last updated:** [February 5, 2018, 9:25pm UTC](https://discuss.elastic.co/t/logstash-csv-output-plugin-path-todays-timestamp-filename/118498 "2018-02-05T21:25:25Z")
I'm utilizing Csv output plugin | Logstash Reference for my output. I'd like to see today's date in my filename (i.e. file-2018-02-05), so I've tried using following: /path/file-%{+YYYY-MM-dd}.txt however, few files… --- ## [Understanding the Logstash Core--Where to Begin](https://discuss.elastic.co/t/understanding-the-logstash-core-where-to-begin/118518)
**Author:** [@stevendavis](https://discuss.elastic.co/u/stevendavis)\ **Replies:** 0\ **Last updated:** [February 5, 2018, 7:36pm UTC](https://discuss.elastic.co/t/understanding-the-logstash-core-where-to-begin/118518 "2018-02-05T19:36:43Z")
I am interested in contributing the the Logstash core. Are there any useful guides already created for understanding the general flow of the source code? --- ## [Logstash is not listening on port 5045 in centos on 'oracle VM VirtualBox](https://discuss.elastic.co/t/logstash-is-not-listening-on-port-5045-in-centos-on-oracle-vm-virtualbox/118503)
**Author:** [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Replies:** 0\ **Last updated:** [February 5, 2018, 5:26pm UTC](https://discuss.elastic.co/t/logstash-is-not-listening-on-port-5045-in-centos-on-oracle-vm-virtualbox/118503 "2018-02-05T17:26:08Z")
I am not sure this is the right forum for this question but I will ask it any way. We are running logstash in centos on 'oracle VM VirtualBox' . The Virtualbox is located on my Windows 7 pc. We configure VirtuaBox … --- ## [Grok Custom pattern file](https://discuss.elastic.co/t/grok-custom-pattern-file/118267)
**Author:** [@Gabriel\_Pellegrini\_M](https://discuss.elastic.co/u/Gabriel_Pellegrini_M)\ **Replies:** 6\ **Last updated:** [February 5, 2018, 4:00pm UTC](https://discuss.elastic.co/t/grok-custom-pattern-file/118267 "2018-02-05T16:00:21Z")
Hi guys, I'm starting in logstash's world and i'm having a problem to set more than one custom grok pattern into a "patern\_dir" I've created a file into my directory "/etc/logstash/pattern" with the follow information … --- ## [Logstash file output. Why having empty lines?](https://discuss.elastic.co/t/logstash-file-output-why-having-empty-lines/118480)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 1\ **Last updated:** [February 5, 2018, 3:50pm UTC](https://discuss.elastic.co/t/logstash-file-output-why-having-empty-lines/118480 "2018-02-05T15:50:30Z")
Hi, I am using logstash to annonymize my logfiles and output them as new logfile using logstash's file output. My output config looks like this: output { file { codec =\> line { format =\> "%{customerNameFP} %{userN… --- ## [Date parsing not working](https://discuss.elastic.co/t/date-parsing-not-working/118468)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 4\ **Last updated:** [February 5, 2018, 3:46pm UTC](https://discuss.elastic.co/t/date-parsing-not-working/118468 "2018-02-05T15:46:49Z")
Hi i have a date field that i need to parse having this format: Wed Nov 1 02:04:55 2017 my date filter would be : date { match =\> \["hpna\_time", "MMM d HH:mm:ss YYYY"\] timezone =\> "… --- ## [How to set Netflow Sampling Rate](https://discuss.elastic.co/t/how-to-set-netflow-sampling-rate/118169)
**Author:** [@MWMJ](https://discuss.elastic.co/u/MWMJ)\ **Replies:** 1\ **Last updated:** [February 5, 2018, 3:43pm UTC](https://discuss.elastic.co/t/how-to-set-netflow-sampling-rate/118169 "2018-02-05T15:43:50Z")
Is there a way to set or improve Netflow sampling rate in a Logstash filter ? When I compare my Netflow data on Kibana with other applications like PRTG, the Kibana values are way smaller. --- ## [How to avoid elasticsearch duplicate documents](https://discuss.elastic.co/t/how-to-avoid-elasticsearch-duplicate-documents/118261)
**Author:** [@databeata](https://discuss.elastic.co/u/databeata)\ **Replies:** 5\ **Last updated:** [February 5, 2018, 3:40pm UTC](https://discuss.elastic.co/t/how-to-avoid-elasticsearch-duplicate-documents/118261 "2018-02-05T15:40:37Z")
How do I avoid elasticsearch duplicate documents? The elasticsearch index docs count (20,010,253) doesn’t match with logs line count (13,411,790). documentation: File input plugin File rotation is detected and handle… --- ## [Logstash 5.1.2 -\> old configuration cached?](https://discuss.elastic.co/t/logstash-5-1-2-old-configuration-cached/118475)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 4\ **Last updated:** [February 5, 2018, 3:27pm UTC](https://discuss.elastic.co/t/logstash-5-1-2-old-configuration-cached/118475 "2018-02-05T15:27:16Z")
Hi, I have some weird issue with logstash 5.1.2 on RHEL 7. I rolled out a new logstash filter for two logfiles. There first I made an mistake which I deployed. There I casted a string to integer by accident. The string… --- ## [Logstash date filter not work](https://discuss.elastic.co/t/logstash-date-filter-not-work/118454)
**Author:** [@uvali](https://discuss.elastic.co/u/uvali)\ **Replies:** 2\ **Last updated:** [February 5, 2018, 1:56pm UTC](https://discuss.elastic.co/t/logstash-date-filter-not-work/118454 "2018-02-05T13:56:27Z")
I try to use logstash date filter but it not work. My conf is: input { stdin{} } filter { date { match =\> \["when", "YYYY-MM-dd HH:mm:ss,SSS"\] } grok { match =\> { "message" =\> "\\ATID:%{SPACE}\[%{INT:id}\]%{SPACE}\[… --- ## [Logstash outofmemory error](https://discuss.elastic.co/t/logstash-outofmemory-error/118470)
**Author:** [@tyb1222](https://discuss.elastic.co/u/tyb1222)\ **Replies:** 0\ **Last updated:** [February 5, 2018, 2:13pm UTC](https://discuss.elastic.co/t/logstash-outofmemory-error/118470 "2018-02-05T14:13:03Z")
I used kafka as my input plugin like this : \`input{ kafka{ bootstrap\_servers=\>"192.168.23.125:9092,192.168.23.126:9092,192.168.23.127:9092" topics=\>\["Apache\_log"\] group\_id=\>"test-consumer-group" consumer\_threads =\> … --- ## [Logstash stdout event](https://discuss.elastic.co/t/logstash-stdout-event/118247)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 4\ **Last updated:** [February 5, 2018, 2:03pm UTC](https://discuss.elastic.co/t/logstash-stdout-event/118247 "2018-02-05T14:03:14Z")
I want to print some events from log file before upload to elasticsearch on stdout console but I don't know why don't work this conf file for me. input{ file{ type =\> "log" codec =\> "json" path =\> "/var/log/test.log… --- ## [Required fields for http\_poller](https://discuss.elastic.co/t/required-fields-for-http-poller/118469)
**Author:** [@T3rm11](https://discuss.elastic.co/u/T3rm11)\ **Replies:** 0\ **Last updated:** [February 5, 2018, 2:01pm UTC](https://discuss.elastic.co/t/required-fields-for-http-poller/118469 "2018-02-05T14:01:42Z")
The official documentation says that only urls is required. When I start Logstash I get the following error: \[2018-02-05T14:44:05,836\]\[ERROR\]\[logstash.inputs.http\_poller\] Missing a required setting for the http\_poller i… --- ## [Logstash kafka plugin outofmemory](https://discuss.elastic.co/t/logstash-kafka-plugin-outofmemory/118466)
**Author:** [@tyb1222](https://discuss.elastic.co/u/tyb1222)\ **Replies:** 0\ **Last updated:** [February 5, 2018, 1:55pm UTC](https://discuss.elastic.co/t/logstash-kafka-plugin-outofmemory/118466 "2018-02-05T13:55:41Z")
I used kafka as my input plugin like this : \`input{ kafka{ bootstrap\_servers=\>"192.168.23.125:9092,192.168.23.126:9092,192.168.23.127:9092" topics=\>\["Apache\_log"\] group\_id=\>"test-consumer-group" consumer\_threads =\> … --- ## [Http\_poller give time out error on data response of 80 K records](https://discuss.elastic.co/t/http-poller-give-time-out-error-on-data-response-of-80-k-records/118344)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [February 5, 2018, 11:50am UTC](https://discuss.elastic.co/t/http-poller-give-time-out-error-on-data-response-of-80-k-records/118344 "2018-02-05T11:50:35Z")
Hi All I have http end point which gives data of 80 K., after running the pipeline i get the error as "name"=\>"http\_request", "error"=\>"Read timed out"}. I have tried all the options of http\_poller input plugin like … --- ## [Access pipeline id inside pipeline config](https://discuss.elastic.co/t/access-pipeline-id-inside-pipeline-config/118428)
**Author:** [@Mariusz\_Gorski](https://discuss.elastic.co/u/Mariusz_Gorski)\ **Replies:** 0\ **Last updated:** [February 5, 2018, 11:07am UTC](https://discuss.elastic.co/t/access-pipeline-id-inside-pipeline-config/118428 "2018-02-05T11:07:00Z")
Hi there elastic community ! As I stumbled upon an idea that hasn't been addressed here (or at least I haven't found such topic) - here goes my first post ! I am looking for a way to access pipeline.id option (from pip… --- ## [Logstash Conf for Mainframe Logs](https://discuss.elastic.co/t/logstash-conf-for-mainframe-logs/114696)
**Author:** [@Sajeew.Ganesh](https://discuss.elastic.co/u/Sajeew.Ganesh)\ **Replies:** 3\ **Last updated:** [February 5, 2018, 10:23am UTC](https://discuss.elastic.co/t/logstash-conf-for-mainframe-logs/114696 "2018-02-05T10:23:37Z")
Hi All, I have a mainframes systems log, How do we grok for the julian date? The sample log is mentioned below. X 0000000 T158 2017208 23:59:00.27 SYSLOG 00000000 IEE042I SYSTEM LOG DATA SET INITIALIZED --- ## [My JDBC SQL statement have both single (') and double (") quote, how do i supply it in JDBC input plugin?](https://discuss.elastic.co/t/my-jdbc-sql-statement-have-both-single-and-double-quote-how-do-i-supply-it-in-jdbc-input-plugin/118396)
**Author:** [@Somnath\_Pal](https://discuss.elastic.co/u/Somnath_Pal)\ **Replies:** 2\ **Last updated:** [February 5, 2018, 10:05am UTC](https://discuss.elastic.co/t/my-jdbc-sql-statement-have-both-single-and-double-quote-how-do-i-supply-it-in-jdbc-input-plugin/118396 "2018-02-05T10:05:26Z")
My input looks like input { jdbc { jdbc\_connection\_string =\> "jdbc:redshift://infa-iidiq-cluster-qa1.ctd4xsffelb3.us-west-2.redshift.amazonaws.com:5439/iidiqdatabase" jdbc\_user =\> "" jdbc\_password =\> "" jdbc\_valida… --- ## [Date filter not working and I cannot understand why](https://discuss.elastic.co/t/date-filter-not-working-and-i-cannot-understand-why/118416)
**Author:** [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\ **Replies:** 1\ **Last updated:** [February 5, 2018, 9:38am UTC](https://discuss.elastic.co/t/date-filter-not-working-and-i-cannot-understand-why/118416 "2018-02-05T09:38:21Z")
Hi, the date filter below just refuses to work. Please help me understand what is wrong here. From logstash output: "@timestamp" =\> 2018-01-25T17:18:14.000Z, "new\_timestamp" =\> 2018-01-25T17:17:14.000Z, date … --- ## [Logstash Drop header line while csv upload](https://discuss.elastic.co/t/logstash-drop-header-line-while-csv-upload/118352)
**Author:** [@scch](https://discuss.elastic.co/u/scch)\ **Replies:** 4\ **Last updated:** [February 5, 2018, 8:48am UTC](https://discuss.elastic.co/t/logstash-drop-header-line-while-csv-upload/118352 "2018-02-05T08:48:23Z")
Hi, How to drop first header line of csv file ? i am trying below code.. and want to drop complete document. Please advise.. filter { if \[message\] =~ /^Occurred On/ { drop {} } } --- ## [How to found problematic messages?](https://discuss.elastic.co/t/how-to-found-problematic-messages/118342)
**Author:** [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Replies:** 1\ **Last updated:** [February 4, 2018, 10:15pm UTC](https://discuss.elastic.co/t/how-to-found-problematic-messages/118342 "2018-02-04T22:15:22Z")
There are many warnings in my logstash logfile about messages with too many fields: "reason"=\>"Limit of total fields \[1000\] in index \[logstash-type-YYYY.MM\] has been exceeded" How to see problematic messages? The only… --- ## [How to fetch SQL incremental data](https://discuss.elastic.co/t/how-to-fetch-sql-incremental-data/118337)
**Author:** [@shridhar](https://discuss.elastic.co/u/shridhar)\ **Replies:** 1\ **Last updated:** [February 4, 2018, 10:07pm UTC](https://discuss.elastic.co/t/how-to-fetch-sql-incremental-data/118337 "2018-02-04T22:07:13Z")
Hi I need help to fetch SQL incremental data using log stash6.1.2.. There is no id field in SQL table and we have to achieve this combining 2 columns hostname & sampletime which will be unique all the time --- ## [Getting Nested Fields Elasticsearch Filter](https://discuss.elastic.co/t/getting-nested-fields-elasticsearch-filter/118223)
**Author:** [@kahndor](https://discuss.elastic.co/u/kahndor)\ **Replies:** 1\ **Last updated:** [February 4, 2018, 3:48pm UTC](https://discuss.elastic.co/t/getting-nested-fields-elasticsearch-filter/118223 "2018-02-04T15:48:20Z")
I using the logstash-elasticsearch-filter to correlate events from Elasticsearch. Some of the fields that I need to return from the Elasticsearch hit are nested. I have been unable to retrieve these docs and the result i… --- ## [Best practice for indexing a daily log](https://discuss.elastic.co/t/best-practice-for-indexing-a-daily-log/118333)
**Author:** [@David\_Gidony](https://discuss.elastic.co/u/David_Gidony)\ **Replies:** 1\ **Last updated:** [February 4, 2018, 9:13am UTC](https://discuss.elastic.co/t/best-practice-for-indexing-a-daily-log/118333 "2018-02-04T09:13:56Z")
hi, i have a few log files that are created daily, i wanted to know what is the best way to ingest them, should i create a new index with the current date for each log file, or just add them all to one big index. for… --- ## [Needed each row as a separate document with the columns as field](https://discuss.elastic.co/t/needed-each-row-as-a-separate-document-with-the-columns-as-field/118304)
**Author:** [@Darpan\_Kukreja](https://discuss.elastic.co/u/Darpan_Kukreja)\ **Replies:** 1\ **Last updated:** [February 3, 2018, 10:03am UTC](https://discuss.elastic.co/t/needed-each-row-as-a-separate-document-with-the-columns-as-field/118304 "2018-02-03T10:03:40Z")
Hi,we are planning to implement elasticsearch for retrieving records,we have already installed elasticsearch which is working fine abhishek@CASSANDRA02 star\_sd\_24jan\]$ curl -X GET 'http://localhost:9200' { "name" : "n… --- ## [Data delay writing to ES](https://discuss.elastic.co/t/data-delay-writing-to-es/118139)
**Author:** [@antony.y](https://discuss.elastic.co/u/antony.y)\ **Replies:** 5\ **Last updated:** [February 3, 2018, 9:59am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139 "2018-02-03T09:59:17Z")
We are currently using logstash collect the netflow from our routers, we expected a real-time data,but now we found that we can only see the data of yesterday, that means the data delayed for one day, and we check our … --- ## [Issue with connection message=\>"no address for stream.twitter.com"](https://discuss.elastic.co/t/issue-with-connection-message-no-address-for-stream-twitter-com/117983)
**Author:** [@Mir\_S](https://discuss.elastic.co/u/Mir_S)\ **Replies:** 1\ **Last updated:** [February 3, 2018, 7:50am UTC](https://discuss.elastic.co/t/issue-with-connection-message-no-address-for-stream-twitter-com/117983 "2018-02-03T07:50:12Z")
Hi I have an issue with the input twitter plugin: I'm using the official logstash docker container 6.1.3 from elastic . Created a Basic Pipelinefile as follow: input { twitter { id =\> "id1" consumer\_key =\> "" cons… --- ## [Pipelines](https://discuss.elastic.co/t/pipelines/118292)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 0\ **Last updated:** [February 3, 2018, 1:49am UTC](https://discuss.elastic.co/t/pipelines/118292 "2018-02-03T01:49:36Z")
I am working with ES/Logstash 6.1 with xpack. I am using the new pipeline capability within ES to store my pipelines in ES and view them in Kibana. Has there been any reported issues with the individual pipelines being… --- ## [Trouble with Json filtering](https://discuss.elastic.co/t/trouble-with-json-filtering/117735)
**Author:** [@gymmynzl](https://discuss.elastic.co/u/gymmynzl)\ **Replies:** 2\ **Last updated:** [February 3, 2018, 1:23am UTC](https://discuss.elastic.co/t/trouble-with-json-filtering/117735 "2018-02-03T01:23:50Z")
Hi, Im having some issues with trying to get some json into ES. Im using filebeat to pass the json data over tcp to logstash ... this is a sample of the JSON, the JSON format is correct its just not feeding in the data… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=827) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=829)