# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=604 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 605 --- ## [Logstash reading sql server data real time](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357)
**Author:** [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Replies:** 8\ **Last updated:** [February 7, 2019, 10:08pm UTC](https://discuss.elastic.co/t/logstash-reading-sql-server-data-real-time/167357 "2019-02-07T22:08:05Z")
Is there any way i can configure logstash so that it picks up delta records real time automatically. If not then is there any opensource plugin/tool available to achieve this? I am pushing log data stored in sql DB to el… --- ## [WinlogBeats not making it through to Elastic](https://discuss.elastic.co/t/winlogbeats-not-making-it-through-to-elastic/167525)
**Author:** [@koocaroo](https://discuss.elastic.co/u/koocaroo)\ **Replies:** 2\ **Last updated:** [February 7, 2019, 9:10pm UTC](https://discuss.elastic.co/t/winlogbeats-not-making-it-through-to-elastic/167525 "2019-02-07T21:10:29Z")
The current setup is My Workstation \> Logstash \> ElasticSearch I can see via tcpdump that i am getting traffic from my host to logstash although its not what i would expect to see, i also dont see any traffic leaving th… --- ## [Grok stop at the first occurrancy error](https://discuss.elastic.co/t/grok-stop-at-the-first-occurrancy-error/167497)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 1\ **Last updated:** [February 7, 2019, 7:25pm UTC](https://discuss.elastic.co/t/grok-stop-at-the-first-occurrancy-error/167497 "2019-02-07T19:25:49Z")
hi, I'm trying to get a specific username using grok plugin. this is my message: {"data":{"id":"134","projectName":"Deploy","fullProjectName":"ITIO/ITSA/SABIS/CribisITA2a/Deploy","displayName":"#134","fullDisplayName"… --- ## [RabbitMQ Plugin x-message-ttl setting](https://discuss.elastic.co/t/rabbitmq-plugin-x-message-ttl-setting/167512)
**Author:** [@shimpi](https://discuss.elastic.co/u/shimpi)\ **Replies:** 1\ **Last updated:** [February 7, 2019, 7:15pm UTC](https://discuss.elastic.co/t/rabbitmq-plugin-x-message-ttl-setting/167512 "2019-02-07T19:15:12Z")
Hello Members, Is there any way I could set x-message-ttl property on the RabbitMQ plugin in logstash. Thanks! --- ## [Retaining some data across different log events](https://discuss.elastic.co/t/retaining-some-data-across-different-log-events/167383)
**Author:** [@verygreen](https://discuss.elastic.co/u/verygreen)\ **Replies:** 3\ **Last updated:** [February 7, 2019, 6:50pm UTC](https://discuss.elastic.co/t/retaining-some-data-across-different-log-events/167383 "2019-02-07T18:50:39Z")
This seems to be a particularly popular topic and I got some useful examples from other forum posts, but they fail to live to my expectations in an unexpected way. So the task at hand is - I have a collection of syslog … --- ## [Logstash doesn't generate output file](https://discuss.elastic.co/t/logstash-doesnt-generate-output-file/167493)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 1\ **Last updated:** [February 7, 2019, 5:46pm UTC](https://discuss.elastic.co/t/logstash-doesnt-generate-output-file/167493 "2019-02-07T17:46:44Z")
Hi, Logstash doesn't generate any output. My lostash.conf file looks like that: input { file { path =\> "C:/elastic/KPI-Incidents-test.csv" start\_position =\> "beginning" ignore\_older =\> 0 sincedb\_path =\> "NUL" } … --- ## [Access nested logstash event key in ruby](https://discuss.elastic.co/t/access-nested-logstash-event-key-in-ruby/167490)
**Author:** [@tszyro](https://discuss.elastic.co/u/tszyro)\ **Replies:** 1\ **Last updated:** [February 7, 2019, 5:45pm UTC](https://discuss.elastic.co/t/access-nested-logstash-event-key-in-ruby/167490 "2019-02-07T17:45:21Z")
I am trying to create new field as a list from nested keys in the event. I thought to do it in ruby filter but can't figure out how to retrieve nested keys into the list I would like to create a field with a list of all… --- ## [Issue while starting logstash](https://discuss.elastic.co/t/issue-while-starting-logstash/167401)
**Author:** [@zaidw21](https://discuss.elastic.co/u/zaidw21)\ **Replies:** 2\ **Last updated:** [February 7, 2019, 5:22pm UTC](https://discuss.elastic.co/t/issue-while-starting-logstash/167401 "2019-02-07T17:22:23Z")
Hi I am new to the ELK framework. Recently started to setup Logstash to parse them and push to elasticsearch. But I am facing the following error everytime i try to run it with my conf file. Tried a lot of options disc… --- ## [Multiple Files CSV automatic create index with filename and column header](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235)
**Author:** [@andrewjo](https://discuss.elastic.co/u/andrewjo)\ **Replies:** 2\ **Last updated:** [February 7, 2019, 4:44pm UTC](https://discuss.elastic.co/t/multiple-files-csv-automatic-create-index-with-filename-and-column-header/167235 "2019-02-07T16:44:20Z")
Hi Guys, I have multiple CSV and I want to ingest into elasticsearch with logstash. Here is my logstash config : file { path =\> '/tmp/email/\*.csv' type =\> 'testcsv' # start\_position =\> 'beginning' sinced… --- ## [Keep one logstash conf file rather having multiple logstash conf file!](https://discuss.elastic.co/t/keep-one-logstash-conf-file-rather-having-multiple-logstash-conf-file/167436)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 4\ **Last updated:** [February 7, 2019, 4:22pm UTC](https://discuss.elastic.co/t/keep-one-logstash-conf-file-rather-having-multiple-logstash-conf-file/167436 "2019-02-07T16:22:49Z")
Suppose I have two sets of data i.e. user & products. I have created two different indexes for them on ES. Now I want to use one logstash conf file that will have some condition to differentiate the data and send them to… --- ## [Issue with Persistent Queue : not polling high volume records](https://discuss.elastic.co/t/issue-with-persistent-queue-not-polling-high-volume-records/167179)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 15\ **Last updated:** [February 7, 2019, 3:53pm UTC](https://discuss.elastic.co/t/issue-with-persistent-queue-not-polling-high-volume-records/167179 "2019-02-07T15:53:18Z")
Hi all i am using a conf file whole input flush 80K records after read writes to disk My PQ setting queue.page\_capacity: 64mb queue.max\_bytes: 1gb input { http\_poller{ urls =\> { http\_request =\> { … --- ## [Parsing XML log file with logstash](https://discuss.elastic.co/t/parsing-xml-log-file-with-logstash/167306)
**Author:** [@Skwilly](https://discuss.elastic.co/u/Skwilly)\ **Replies:** 6\ **Last updated:** [February 7, 2019, 3:27pm UTC](https://discuss.elastic.co/t/parsing-xml-log-file-with-logstash/167306 "2019-02-07T15:27:22Z")
Hello, I am completely new to ElasticSearch. I did successfully set it up and explore some test data but warning: I am a newbie. I am trying to index an XML log file through logstash and I am lost as to how to procede … --- ## [How to multi grok json with pipe](https://discuss.elastic.co/t/how-to-multi-grok-json-with-pipe/167296)
**Author:** [@BBStash](https://discuss.elastic.co/u/BBStash)\ **Replies:** 2\ **Last updated:** [February 7, 2019, 3:23pm UTC](https://discuss.elastic.co/t/how-to-multi-grok-json-with-pipe/167296 "2019-02-07T15:23:18Z")
Hi How to grok this in logstash: {"data":{"context":{"code":39921,"father\_pid":"20190204171709\_eIqUa","origin":"tWarn\_1","context":"Default","project":"DATA\_RECOVERY","pid":"8ccIjj","job":"FR\_SIUV1\_Load\_MyJob","priorit… --- ## [Delete me pls](https://discuss.elastic.co/t/delete-me-pls/167472)
**Author:** [@Necus](https://discuss.elastic.co/u/Necus)\ **Replies:** 1\ **Last updated:** [February 7, 2019, 3:07pm UTC](https://discuss.elastic.co/t/delete-me-pls/167472 "2019-02-07T15:07:07Z")
Wrong issue --- ## [Is ingest-converter tool still maintained?](https://discuss.elastic.co/t/is-ingest-converter-tool-still-maintained/167297)
**Author:** [@adrian.aneci](https://discuss.elastic.co/u/adrian.aneci)\ **Replies:** 5\ **Last updated:** [February 7, 2019, 3:03pm UTC](https://discuss.elastic.co/t/is-ingest-converter-tool-still-maintained/167297 "2019-02-07T15:03:37Z")
Hello, Is the ingest converter tool (https://www.elastic.co/guide/en/logstash/current/ingest-converter.html) still maintained? I saw that based on github it was last updated 2 years ago. Currently it is lacking a lot o… --- ## [Error when port data from mySQL](https://discuss.elastic.co/t/error-when-port-data-from-mysql/167344)
**Author:** [@vschaefer](https://discuss.elastic.co/u/vschaefer)\ **Replies:** 4\ **Last updated:** [February 7, 2019, 2:31pm UTC](https://discuss.elastic.co/t/error-when-port-data-from-mysql/167344 "2019-02-07T14:31:06Z")
I'm a little desperate. I'm trying to index data from a MySQL database. But it just won't work. Can you tell me what I'm doing wrong? Elasticsearch and Kibana is running and the MySQL database is available. Here is my… --- ## [Performance capped - low events received rate](https://discuss.elastic.co/t/performance-capped-low-events-received-rate/167287)
**Author:** [@stef97](https://discuss.elastic.co/u/stef97)\ **Replies:** 8\ **Last updated:** [February 7, 2019, 2:17pm UTC](https://discuss.elastic.co/t/performance-capped-low-events-received-rate/167287 "2019-02-07T14:17:13Z")
I have a logstash 5.6.9 that seems to have its performance capped and/or very low events received rate It is a Virtual machine so event throughing resources at it ( doubling CPU, RAM ) does not help I also tried play… --- ## [Azure\_event\_hubs proxy](https://discuss.elastic.co/t/azure-event-hubs-proxy/166917)
**Author:** [@SupremeCookie](https://discuss.elastic.co/u/SupremeCookie)\ **Replies:** 1\ **Last updated:** [February 7, 2019, 2:07pm UTC](https://discuss.elastic.co/t/azure-event-hubs-proxy/166917 "2019-02-07T14:07:37Z")
Hi all, the input plugin "azure\_event\_hubs" does not include proxy parameters for the config file. Is there a way around this? Thanks --- ## [Create index from the the grok pattern output](https://discuss.elastic.co/t/create-index-from-the-the-grok-pattern-output/167119)
**Author:** [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Replies:** 5\ **Last updated:** [February 7, 2019, 1:43pm UTC](https://discuss.elastic.co/t/create-index-from-the-the-grok-pattern-output/167119 "2019-02-07T13:43:04Z")
Hi I have a requirement of creating the index based on the filed in the message for example my grok is as below (?:%{WORD:method}|-)(?:%{NOTSPACE:Appname}|-) so in my message i have a set of Appname's, so i need to i… --- ## [JDBC Input Plugin: extracting the statement as a field in Logstash?](https://discuss.elastic.co/t/jdbc-input-plugin-extracting-the-statement-as-a-field-in-logstash/167439)
**Author:** [@rozling](https://discuss.elastic.co/u/rozling)\ **Replies:** 2\ **Last updated:** [February 7, 2019, 12:38pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-extracting-the-statement-as-a-field-in-logstash/167439 "2019-02-07T12:38:00Z")
We have our JDBC input working well, and we're trying to see if we can log our SQL statement with the query result. Here is our config: input { jdbc { jdbc\_driver\_library =\> "/usr/share/java/mysql.jar" jdbc\_d… --- ## [How to convert and store logs time in a field with date type. I'm new to it and need to show some results asap](https://discuss.elastic.co/t/how-to-convert-and-store-logs-time-in-a-field-with-date-type-im-new-to-it-and-need-to-show-some-results-asap/167227)
**Author:** [@sagarleo](https://discuss.elastic.co/u/sagarleo)\ **Replies:** 4\ **Last updated:** [February 7, 2019, 11:56am UTC](https://discuss.elastic.co/t/how-to-convert-and-store-logs-time-in-a-field-with-date-type-im-new-to-it-and-need-to-show-some-results-asap/167227 "2019-02-07T11:56:25Z")
filter { grok { match =\> \[ "message", "%{TIMESTAMP\_ISO8601:event\_timestamp} %{GREEDYDATA:message}" \] } date { match =\> \["event\_timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"\] target =\> "@timestamp" } } --- ## [Logstash writing logs to syslog even though no output configured](https://discuss.elastic.co/t/logstash-writing-logs-to-syslog-even-though-no-output-configured/167428)
**Author:** [@commentator8](https://discuss.elastic.co/u/commentator8)\ **Replies:** 0\ **Last updated:** [February 7, 2019, 11:01am UTC](https://discuss.elastic.co/t/logstash-writing-logs-to-syslog-even-though-no-output-configured/167428 "2019-02-07T11:01:17Z")
Hi, I am trying to understand how it is that my logstash instance is writing logs to /var/log/syslog. I have debug open and it fills up my 40gb hard drive within hours. I checked that the contents of /etc/rsyslog.d/50… --- ## [\*\*URGENT\*\* - Need help to extract fields](https://discuss.elastic.co/t/urgent-need-help-to-extract-fields/167361)
**Author:** [@vdely](https://discuss.elastic.co/u/vdely)\ **Replies:** 2\ **Last updated:** [February 7, 2019, 9:30am UTC](https://discuss.elastic.co/t/urgent-need-help-to-extract-fields/167361 "2019-02-07T09:30:23Z")
Hi Techs and experts, I need help to configure logstash to extract the fields from a log file which has multiple lines like the following : \<134\>Agent\_Local\_Date="2019.01.11" Agent\_Local\_Time="1/11/2019 4:07:06 AM" Ag… --- ## [Unable to take some logs in a log file](https://discuss.elastic.co/t/unable-to-take-some-logs-in-a-log-file/167399)
**Author:** [@abrejuin](https://discuss.elastic.co/u/abrejuin)\ **Replies:** 0\ **Last updated:** [February 7, 2019, 8:57am UTC](https://discuss.elastic.co/t/unable-to-take-some-logs-in-a-log-file/167399 "2019-02-07T08:57:07Z")
Hello I have something special in my Elastic stack : I setted up rsyslog, logstash, Elastic and Kibana to analyze a log file named server.log. For a few weeks ago, some logs written in server.log file are not taking by… --- ## [Sending Squid Logs with Filebeat and manipulate with logstash](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298)
**Author:** [@sdvincent](https://discuss.elastic.co/u/sdvincent)\ **Replies:** 5\ **Last updated:** [February 7, 2019, 4:40am UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298 "2019-02-07T04:40:29Z")
Hello, This might be a newbish question but I haven't been able to find what I've been looking for elsewhere online. I am trying to learn how to send logs that don't have pre-built modules to logstash and have logstash … --- ## [Json field check filter](https://discuss.elastic.co/t/json-field-check-filter/165702)
**Author:** [@paano](https://discuss.elastic.co/u/paano)\ **Replies:** 2\ **Last updated:** [February 6, 2019, 11:20pm UTC](https://discuss.elastic.co/t/json-field-check-filter/165702 "2019-02-06T23:20:05Z")
in logstash, adding a filter check based on key field to add tag does not work. sample json: { "eventSource" : { "objectName": "EVENT.Q1", "objectType" : "Queue" }, "eventType" : { "name" : "Com… --- ## [How to store @timestamp in UNIX\_MS instead of ISO date format](https://discuss.elastic.co/t/how-to-store-timestamp-in-unix-ms-instead-of-iso-date-format/165571)
**Author:** [@Krishna\_Sunil](https://discuss.elastic.co/u/Krishna_Sunil)\ **Replies:** 4\ **Last updated:** [February 6, 2019, 8:33pm UTC](https://discuss.elastic.co/t/how-to-store-timestamp-in-unix-ms-instead-of-iso-date-format/165571 "2019-02-06T20:33:33Z")
Currently, @timestamp is in ISO date format. I want to store the same date in epochmilli. Sample code : filter { ruby { code =\> "event.set('epochs', ((event.get('@timestamp').to\_f\*1000).to\_i).to\_s)" } date { remove… --- ## [Trying to setup Logstash for Syslog input but having weird problems](https://discuss.elastic.co/t/trying-to-setup-logstash-for-syslog-input-but-having-weird-problems/167207)
**Author:** [@snovak](https://discuss.elastic.co/u/snovak)\ **Replies:** 5\ **Last updated:** [February 6, 2019, 8:26pm UTC](https://discuss.elastic.co/t/trying-to-setup-logstash-for-syslog-input-but-having-weird-problems/167207 "2019-02-06T20:26:32Z")
Trying to configure logstash to accept syslog messages and then place those into Elastic Search / Kibana but having several weird issues. The first is that actual syslog messages are being rejected with this type of mes… --- ## [How to convert parsed date into long (epoch time)](https://discuss.elastic.co/t/how-to-convert-parsed-date-into-long-epoch-time/167202)
**Author:** [@shimpi](https://discuss.elastic.co/u/shimpi)\ **Replies:** 4\ **Last updated:** [February 6, 2019, 6:40pm UTC](https://discuss.elastic.co/t/how-to-convert-parsed-date-into-long-epoch-time/167202 "2019-02-06T18:40:45Z")
I'm parsing tomcat log 2019-01-31 23:25:27,421 INFO SNOW FAA31C764BD69D829C09715487B1E923 \[Thread-16\] using: grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:date} %{LOGLEVEL:loglevel} %{WORD:GW} %{WORD:GUID}… --- ## [Efficient way to determine IP version (v4 vs. v6)?](https://discuss.elastic.co/t/efficient-way-to-determine-ip-version-v4-vs-v6/167334)
**Author:** [@SpeedDaemon](https://discuss.elastic.co/u/SpeedDaemon)\ **Replies:** 2\ **Last updated:** [February 6, 2019, 6:40pm UTC](https://discuss.elastic.co/t/efficient-way-to-determine-ip-version-v4-vs-v6/167334 "2019-02-06T18:40:12Z")
Is there an efficient way in a Logstash pipeline to take a field with an IP in it, and determine whether that IP is v4 or v6? The only way I can think to do it would involve at least two groks per IP (four per doc for s… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=603) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=605)