# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=925 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 926 --- ## [Date filter problem](https://discuss.elastic.co/t/date-filter-problem/97211)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 10\ **Last updated:** [August 16, 2017, 10:25am UTC](https://discuss.elastic.co/t/date-filter-problem/97211 "2017-08-16T10:25:29Z")
1 : In kibana Time field is showing string , not as date field. 2 : this is my sample log file… {“Metrics”:“type=COUNTER, name=FacilityMgmtResource Counter, count=0,Time:2017-08-16\_01:51:14.926”} {“Metrics”:“type… --- ## [I/o timeout on clientside and overloading logstash server](https://discuss.elastic.co/t/i-o-timeout-on-clientside-and-overloading-logstash-server/97210)
**Author:** [@g4nj4wizard](https://discuss.elastic.co/u/g4nj4wizard)\ **Replies:** 1\ **Last updated:** [August 16, 2017, 10:17am UTC](https://discuss.elastic.co/t/i-o-timeout-on-clientside-and-overloading-logstash-server/97210 "2017-08-16T10:17:27Z")
Hello everyone, i previously installed logstash on a Ubuntu 16.04 LTS VM on a Hyper-V. We wanted to collect all Windows Events with winlogbeat. I tested it some days with only a few server. It works, but after sometime… --- ## [Send to multiple elasticsearch index with different fields](https://discuss.elastic.co/t/send-to-multiple-elasticsearch-index-with-different-fields/97208)
**Author:** [@WangXiangUSTC](https://discuss.elastic.co/u/WangXiangUSTC)\ **Replies:** 3\ **Last updated:** [August 16, 2017, 9:35am UTC](https://discuss.elastic.co/t/send-to-multiple-elasticsearch-index-with-different-fields/97208 "2017-08-16T09:35:24Z")
I have log like this: { “info”:“just for search”, “url”:“123.234.123.234:8080/abc”, “response”: { “code”: 200, “status”: ok, “cmd”: “login” } } and I want to send the total log into index A for text search, an… --- ## [Using Gsub to replace field values](https://discuss.elastic.co/t/using-gsub-to-replace-field-values/96613)
**Author:** [@sentinel](https://discuss.elastic.co/u/sentinel)\ **Replies:** 8\ **Last updated:** [August 16, 2017, 9:33am UTC](https://discuss.elastic.co/t/using-gsub-to-replace-field-values/96613 "2017-08-16T09:33:15Z")
Hi I’m trying to transform field values in order to get rid of some characters however I do not know the values. I tried this: gsub =\> \[“fwdPtPrecision”, “.,”,"."\] but the fwdPtPrecision=0.001 was changed to 0.00. W… --- ## [Grok match with AND condition](https://discuss.elastic.co/t/grok-match-with-and-condition/97022)
**Author:** [@rammulay](https://discuss.elastic.co/u/rammulay)\ **Replies:** 3\ **Last updated:** [August 16, 2017, 8:36am UTC](https://discuss.elastic.co/t/grok-match-with-and-condition/97022 "2017-08-16T08:36:28Z")
I have the following log lines - Receive event inbound record: ConsumerRecord(topic = wms, partition = 1, offset = 52479, NoTimestampType = -1, checksum = 2914947648, serialized key size = 36, serialized value size = 21… --- ## [Logstash File output plugin - IOError: No space left on device](https://discuss.elastic.co/t/logstash-file-output-plugin-ioerror-no-space-left-on-device/97206)
**Author:** [@turnerj](https://discuss.elastic.co/u/turnerj)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 8:16am UTC](https://discuss.elastic.co/t/logstash-file-output-plugin-ioerror-no-space-left-on-device/97206 "2017-08-16T08:16:48Z")
Hi I am running a Logstash 5.1.1 instance collecting and parsing syslogs. Output plugins are a combination of exec, sending syslog data to an executable, elasticsearch, and file, writing formatted output to a text file.… --- ## [SNMPTraps in LogStash](https://discuss.elastic.co/t/snmptraps-in-logstash/97064)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 1\ **Last updated:** [August 16, 2017, 7:16am UTC](https://discuss.elastic.co/t/snmptraps-in-logstash/97064 "2017-08-16T07:16:19Z")
Hi - Could some one please help me out on how to use the SNMP plugin in the Logstash ? I’d like to collect the traps from a Windows server and publish it to Stack. Thanks in advance. --- ## [How to do SSH key based authentication to MySQL in jdbc input plugin when using Logstash](https://discuss.elastic.co/t/how-to-do-ssh-key-based-authentication-to-mysql-in-jdbc-input-plugin-when-using-logstash/97190)
**Author:** [@Deepti\_Jain](https://discuss.elastic.co/u/Deepti_Jain)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 6:37am UTC](https://discuss.elastic.co/t/how-to-do-ssh-key-based-authentication-to-mysql-in-jdbc-input-plugin-when-using-logstash/97190 "2017-08-16T06:37:13Z")
MYSQL DB server has been set up to accept an SSH key while authenticating the user. WHen the jdbc input plugin tries to connect to the server it fails to establish the connection since there is no means to provide the ke… --- ## [I give up .. someone please help](https://discuss.elastic.co/t/i-give-up-someone-please-help/97117)
**Author:** [@aliyesami](https://discuss.elastic.co/u/aliyesami)\ **Replies:** 1\ **Last updated:** [August 16, 2017, 5:46am UTC](https://discuss.elastic.co/t/i-give-up-someone-please-help/97117 "2017-08-16T05:46:29Z")
I need to extract all the lines starting with "ORA-" from the logfile which looks like shown below. Also I need to split the line in two parts the ORA- and the following message . and the last timestamp before the… --- ## [Show Latest data from CSV issue](https://discuss.elastic.co/t/show-latest-data-from-csv-issue/96581)
**Author:** [@sabi1rosel](https://discuss.elastic.co/u/sabi1rosel)\ **Replies:** 2\ **Last updated:** [August 16, 2017, 5:45am UTC](https://discuss.elastic.co/t/show-latest-data-from-csv-issue/96581 "2017-08-16T05:45:42Z")
I am fetching data from a text file and feeding it to kibana , When i remove the old data from txt file and insert new one kibana still showing old data appended with new data. My requirement is to kibana show only lates… --- ## [Logstash Duplicate Data Output?](https://discuss.elastic.co/t/logstash-duplicate-data-output/97168)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 2:22am UTC](https://discuss.elastic.co/t/logstash-duplicate-data-output/97168 "2017-08-16T02:22:51Z")
My input is from elasticsearch. I want to combine multiple indices in one index. As I’d output the data in the elasticsearch, I can see that the data is much more data if I total from the source than the combined data. --- ## [Get Difference between two date field with ruby filter](https://discuss.elastic.co/t/get-difference-between-two-date-field-with-ruby-filter/97100)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [August 16, 2017, 1:40am UTC](https://discuss.elastic.co/t/get-difference-between-two-date-field-with-ruby-filter/97100 "2017-08-16T01:40:48Z")
Hi All I have been trying the ruby code to get the difference between 2 dates Start Date : 2017-06-13T01:17:07.000Z End Date : 2017-07-13T01:17:10.000Z I need difference between these 2 dates. I have dried this but i… --- ## [Logstash configuration to achieve 300k eps](https://discuss.elastic.co/t/logstash-configuration-to-achieve-300k-eps/97162)
**Author:** [@Supernomad](https://discuss.elastic.co/u/Supernomad)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 1:23am UTC](https://discuss.elastic.co/t/logstash-configuration-to-achieve-300k-eps/97162 "2017-08-16T01:23:15Z")
So I have gone through the majority of this forum and what I can find on the web, and have come up short of how exactly to push logstash to get to this kind of scale. A bit of back story, I am currently running a 5 clie… --- ## [Http input response headers](https://discuss.elastic.co/t/http-input-response-headers/97030)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 2\ **Last updated:** [August 15, 2017, 8:32pm UTC](https://discuss.elastic.co/t/http-input-response-headers/97030 "2017-08-15T20:32:00Z")
I’m using Logstash to receive an http input from a plugin on Kibana and then it pushes it to an http output to Jira to make a ticket. I’m doing this to avoid CORS restrictions on Chrome because I can’t make POSTs to Jira… --- ## [Print directory in Logstash configuration file](https://discuss.elastic.co/t/print-directory-in-logstash-configuration-file/97106)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 1\ **Last updated:** [August 15, 2017, 3:34pm UTC](https://discuss.elastic.co/t/print-directory-in-logstash-configuration-file/97106 "2017-08-15T15:34:50Z")
I am putting the ELK stack and all of its dependencies within a zip file so that people within my company can unzip the file and run it on their local machine to analyze logs. A problem that I have is that the input in l… --- ## [Jquery error meaning](https://discuss.elastic.co/t/jquery-error-meaning/97109)
**Author:** [@GambitK](https://discuss.elastic.co/u/GambitK)\ **Replies:** 0\ **Last updated:** [August 15, 2017, 1:59pm UTC](https://discuss.elastic.co/t/jquery-error-meaning/97109 "2017-08-15T13:59:21Z")
I’m moving a logstash config from one server to another and jquery is throwing an error that I don’t know what it means: \[2017-08-15T09:57:12,980\]\[INFO \]\[logstash.pipeline \] Starting pipeline {"id"=\>"main", "pipe… --- ## [Timestamp based on lon/lat](https://discuss.elastic.co/t/timestamp-based-on-lon-lat/97101)
**Author:** [@bitesize](https://discuss.elastic.co/u/bitesize)\ **Replies:** 0\ **Last updated:** [August 15, 2017, 1:02pm UTC](https://discuss.elastic.co/t/timestamp-based-on-lon-lat/97101 "2017-08-15T13:02:50Z")
I am wondering if I could somehow add a timestamp field that would show the local time of the log based off of the lon/lat. Any help would be appreciated. --- ## [Logstash handling of field names with all caps](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017)
**Author:** [@crazyphil](https://discuss.elastic.co/u/crazyphil)\ **Replies:** 8\ **Last updated:** [August 15, 2017, 12:28pm UTC](https://discuss.elastic.co/t/logstash-handling-of-field-names-with-all-caps/97017 "2017-08-15T12:28:38Z")
Logstash version is 5.4.3. I have been attempting to perform a grok on a field, however no matter what I do, the grok never occurs (it is wrapped in an if statement). My field name is “EVENT\_CLICK” and no matter what I… --- ## [How to rename fields](https://discuss.elastic.co/t/how-to-rename-fields/97077)
**Author:** [@mikygee](https://discuss.elastic.co/u/mikygee)\ **Replies:** 3\ **Last updated:** [August 15, 2017, 12:20pm UTC](https://discuss.elastic.co/t/how-to-rename-fields/97077 "2017-08-15T12:20:31Z")
Hello, I have grok rules that work fine to match the OS and user agent filter { if \[program\] == “nginx” { grok { break\_on\_match =\> true match =\> \[ "message", "%{IPORHOST:remote\_addr} - - \\\[%{HTTPD… --- ## [Only one line in logstash-plain.log](https://discuss.elastic.co/t/only-one-line-in-logstash-plain-log/97092)
**Author:** [@urajkovic](https://discuss.elastic.co/u/urajkovic)\ **Replies:** 0\ **Last updated:** [August 15, 2017, 12:15pm UTC](https://discuss.elastic.co/t/only-one-line-in-logstash-plain-log/97092 "2017-08-15T12:15:09Z")
On CentOS, when I start Logstash with this command: sudo systemctl start logstash It produces this line in /var/log/logstash/logstash-plain.log: \[2017-08-15T13:52:14,119\]\[INFO \]\[logstash.agent \] Successfully… --- ## [Cannot create pipeline](https://discuss.elastic.co/t/cannot-create-pipeline/97062)
**Author:** [@syashwanthk](https://discuss.elastic.co/u/syashwanthk)\ **Replies:** 8\ **Last updated:** [August 15, 2017, 11:46am UTC](https://discuss.elastic.co/t/cannot-create-pipeline/97062 "2017-08-15T11:46:49Z")
Hi, logstash -e ‘input { stdin{} } output { elasticsearch{ host =\>“localhost:9200” } }’ Please find the complete logs as below while trying to configure logstash: \[2017-08-15T12:03:17,000\]\[DEBUG\]\[logstash.plugins.regi… --- ## [Logstash Kafka input causes duplicates](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885)
**Author:** [@guyisra](https://discuss.elastic.co/u/guyisra)\ **Replies:** 2\ **Last updated:** [August 15, 2017, 10:56am UTC](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885 "2017-08-15T10:56:21Z")
I am using Logstash 5.5.1 with kafka input 5.1.8. When consuming the topic, logstash forwards many duplicates to the output, while the event it self was consumed only once. Is this a bug or a configuration issue? This… --- ## [Logstash to store information for next events](https://discuss.elastic.co/t/logstash-to-store-information-for-next-events/96918)
**Author:** [@anagaraj](https://discuss.elastic.co/u/anagaraj)\ **Replies:** 2\ **Last updated:** [August 15, 2017, 7:31am UTC](https://discuss.elastic.co/t/logstash-to-store-information-for-next-events/96918 "2017-08-15T07:31:59Z")
filter { grok { match =\> { "message" =\> \[ "I %{GREEDYDATA:cur\_ts} .\* cli.\*since", "Incoming.\*: %{NUMBER:\[dep\]\[pktper\]}/%{NUMBER:\[dep\]\[byteper\]} %{NUMBER:\[de… --- ## [Additional fields while inserting to elastic](https://discuss.elastic.co/t/additional-fields-while-inserting-to-elastic/96757)
**Author:** [@remziay](https://discuss.elastic.co/u/remziay)\ **Replies:** 2\ **Last updated:** [August 11, 2017, 12:59pm UTC](https://discuss.elastic.co/t/additional-fields-while-inserting-to-elastic/96757 "2017-08-11T12:59:09Z")
Hello, i am very new to logstash. I want to import the entries in a file to the elastic. first I created the index on elastic : { “mytype-template”: { “template”: “mytype-\*”, “settings”: { “refresh\_interval”: “60s” … --- ## [Logstash kafka topic as part of elasticsearch document\_type](https://discuss.elastic.co/t/logstash-kafka-topic-as-part-of-elasticsearch-document-type/97044)
**Author:** [@sulthonzh](https://discuss.elastic.co/u/sulthonzh)\ **Replies:** 0\ **Last updated:** [August 15, 2017, 1:50am UTC](https://discuss.elastic.co/t/logstash-kafka-topic-as-part-of-elasticsearch-document-type/97044 "2017-08-15T01:50:05Z")
I have tried this conf but not working input { kafka { ... decorate\_events =\> true } } filter { mutate { add\_field =\> {"\[@metadata\]\[topic\]" =\> "%{\[kafka\]\[topi… --- ## [Geoip Location geo\_point field missing](https://discuss.elastic.co/t/geoip-location-geo-point-field-missing/96906)
**Author:** [@Duncher](https://discuss.elastic.co/u/Duncher)\ **Replies:** 2\ **Last updated:** [August 15, 2017, 1:35am UTC](https://discuss.elastic.co/t/geoip-location-geo-point-field-missing/96906 "2017-08-15T01:35:18Z")
Hi, I’m using the geoip filter and for the most part it is working as expected. However, it is not giving me the Location geo\_point field. The data is there i.e. the lat and lon floats. But Kibana won’t recognise the ge… --- ## [Metrics filter not working on 5.5](https://discuss.elastic.co/t/metrics-filter-not-working-on-5-5/97042)
**Author:** [@Rohit\_Goel](https://discuss.elastic.co/u/Rohit_Goel)\ **Replies:** 0\ **Last updated:** [August 15, 2017, 1:33am UTC](https://discuss.elastic.co/t/metrics-filter-not-working-on-5-5/97042 "2017-08-15T01:33:58Z")
Hello, I am trying to use metrics plugin with logstash 5.5 and it does not look to be working. Below is my original code metrics { meter =\> \[ “http\_%{responsecode}” \] add\_tag =\> "globanetmetrics" flush\_interval =\> … --- ## [Logstash with queue enabled not ack http input events after jdbc input runs](https://discuss.elastic.co/t/logstash-with-queue-enabled-not-ack-http-input-events-after-jdbc-input-runs/96358)
**Author:** [@bearhead](https://discuss.elastic.co/u/bearhead)\ **Replies:** 1\ **Last updated:** [August 15, 2017, 12:49am UTC](https://discuss.elastic.co/t/logstash-with-queue-enabled-not-ack-http-input-events-after-jdbc-input-runs/96358 "2017-08-15T00:49:28Z")
Hi I'm using logstash with queuing enabled. I've setup logstash to inject rows from mysql via the mysql input plugin on startup. Currently this is injecting 1846 rows. I also have a http input. When I take down ES an… --- ## [Grok and Translate logstash](https://discuss.elastic.co/t/grok-and-translate-logstash/97040)
**Author:** [@Drift](https://discuss.elastic.co/u/Drift)\ **Replies:** 1\ **Last updated:** [August 15, 2017, 12:32am UTC](https://discuss.elastic.co/t/grok-and-translate-logstash/97040 "2017-08-15T00:32:09Z")
Is this not a valid translate? I grok the IP address out of a message and then pass it to the translate plugin. filter { grok { match =\> { "message" =\> "%{IP:des\_ip}" } } translate { field =\> "des\_ip" … --- ## [Logstash rubyjit error with filter](https://discuss.elastic.co/t/logstash-rubyjit-error-with-filter/97008)
**Author:** [@TheCompScientist](https://discuss.elastic.co/u/TheCompScientist)\ **Replies:** 2\ **Last updated:** [August 14, 2017, 10:00pm UTC](https://discuss.elastic.co/t/logstash-rubyjit-error-with-filter/97008 "2017-08-14T22:00:11Z")
Hello, I am experiencing an issue with Logstash throwing an error that causes it to stop running. I get the following error: \[2017-08-11T19:20:51,987\]\[ERROR\]\[logstash.pipeline \] allSite.java:154)", "rubyjit.Log… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=924) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=926)