# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1104 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1105 --- ## [Logstash Key Value Filter unable to parse specific text](https://discuss.elastic.co/t/logstash-key-value-filter-unable-to-parse-specific-text/60145)
**Author:** [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\ **Replies:** 3\ **Last updated:** [September 9, 2016, 10:48am UTC](https://discuss.elastic.co/t/logstash-key-value-filter-unable-to-parse-specific-text/60145 "2016-09-09T10:48:26Z")
I am trying to use Logstash's Key Value (kv) filter to parse logs. It specifically fails for a pair where key is "valid". see below example: Log message: "versionInfo=VersionInfo{valid=test}" Logstash filter config s… --- ## [Logstash quote escape bug](https://discuss.elastic.co/t/logstash-quote-escape-bug/60114)
**Author:** [@kaung](https://discuss.elastic.co/u/kaung)\ **Replies:** 2\ **Last updated:** [September 9, 2016, 10:40am UTC](https://discuss.elastic.co/t/logstash-quote-escape-bug/60114 "2016-09-09T10:40:16Z")
Hi guys, I think logstash quote escaping has a bug. I do this mutate { add\_field =\> { "foo" =\> 'string with " and ' chars' } add\_field =\> { "bar" =\> "string with " and ' chars" } } I output the result to file wi… --- ## [Sincedb doesn't updated until tail reaches the end of the file](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116)
**Author:** [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\ **Replies:** 3\ **Last updated:** [September 9, 2016, 10:30am UTC](https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116 "2016-09-09T10:30:35Z")
hello. it seems sincedb updating does not occur until TAIL reaches the end of the file. with this configuration input { file { path =\> "/home1/.logstash/json\_input/deco.json.\*" codec =\> … --- ## [No effect of "size" in 'query' while reindexing in elasticsearch](https://discuss.elastic.co/t/no-effect-of-size-in-query-while-reindexing-in-elasticsearch/59967)
**Author:** [@ultimate.duwal](https://discuss.elastic.co/u/ultimate.duwal)\ **Replies:** 1\ **Last updated:** [September 9, 2016, 9:07am UTC](https://discuss.elastic.co/t/no-effect-of-size-in-query-while-reindexing-in-elasticsearch/59967 "2016-09-09T09:07:45Z")
Hello, I have been using logstash to migrate a index to another. I have recently tried to reindex certain amount of data from large dataset in local environment. So I tried using following configuration for migration: i… --- ## [\[Resolved\] Migrating from logstash 2.1 to 2.4, grok error](https://discuss.elastic.co/t/resolved-migrating-from-logstash-2-1-to-2-4-grok-error/59991)
**Author:** [@rdesgrange](https://discuss.elastic.co/u/rdesgrange)\ **Replies:** 4\ **Last updated:** [September 9, 2016, 8:22am UTC](https://discuss.elastic.co/t/resolved-migrating-from-logstash-2-1-to-2-4-grok-error/59991 "2016-09-09T08:22:29Z")
Hi, I tried to migrate from logstash 2.1 to 2.4. I have this grok rule in a file in the pattern directory : RAILS\_TIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY}\[T \]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?( %{ISO8601\_TIMEZONE})?… --- ## [Filebeat and tile map visualization](https://discuss.elastic.co/t/filebeat-and-tile-map-visualization/58969)
**Author:** [@Diggy](https://discuss.elastic.co/u/Diggy)\ **Replies:** 14\ **Last updated:** [September 9, 2016, 5:30am UTC](https://discuss.elastic.co/t/filebeat-and-tile-map-visualization/58969 "2016-09-09T05:30:54Z")
I'm trying to use Filebeat input to create tile maps of http access. The geoip data are captured, but when I try and create a map, I get the following error: No Compatible Fields: The "filebeat-\*" index pattern does n… --- ## [Kafka Output plugin logstash](https://discuss.elastic.co/t/kafka-output-plugin-logstash/60133)
**Author:** [@soodlikesjava](https://discuss.elastic.co/u/soodlikesjava)\ **Replies:** 0\ **Last updated:** [September 9, 2016, 5:28am UTC](https://discuss.elastic.co/t/kafka-output-plugin-logstash/60133 "2016-09-09T05:28:36Z")
Hi Team, I am using kafka output plugin in logstash 2.1 , i want to know if i can mention zookeeper servers in BOOTSTRAP\_SERVERS configuration of kafka output plugin of logstash instead of giving the broker… --- ## [How to debug error while running logstash-input-dynamodb plugin](https://discuss.elastic.co/t/how-to-debug-error-while-running-logstash-input-dynamodb-plugin/60122)
**Author:** [@kalpitad](https://discuss.elastic.co/u/kalpitad)\ **Replies:** 1\ **Last updated:** [September 8, 2016, 10:07pm UTC](https://discuss.elastic.co/t/how-to-debug-error-while-running-logstash-input-dynamodb-plugin/60122 "2016-09-08T22:07:43Z")
I'm trying to pull in DynamoDB data into ElasticSearch using the logstash-input-dynamodb plugin. When I try to start Logstash after setting up my .conf file, I get a LogStash::ConfigurationError. How can I debug this fur… --- ## [Unescaping XML in Logstash](https://discuss.elastic.co/t/unescaping-xml-in-logstash/60091)
**Author:** [@prestona](https://discuss.elastic.co/u/prestona)\ **Replies:** 0\ **Last updated:** [September 8, 2016, 3:49pm UTC](https://discuss.elastic.co/t/unescaping-xml-in-logstash/60091 "2016-09-08T15:49:42Z")
Given a config file such as: input { stdin { type =\> "test" } } filter { xml { source =\> "message" store\_xml =\> "false" xpath =\> \["/MAIN/SUB/text()", "value"\] } if \[value\] { mutate { … --- ## [Is it possible to access the whole logstash event as a variable in filters/outputs?](https://discuss.elastic.co/t/is-it-possible-to-access-the-whole-logstash-event-as-a-variable-in-filters-outputs/60090)
**Author:** [@vangap](https://discuss.elastic.co/u/vangap)\ **Replies:** 0\ **Last updated:** [September 8, 2016, 3:46pm UTC](https://discuss.elastic.co/t/is-it-possible-to-access-the-whole-logstash-event-as-a-variable-in-filters-outputs/60090 "2016-09-08T15:46:27Z")
I wan to change the structure of the event that gets used in the output. I want to generate soem think like this newevent = {"custom\_field" =\> 123, "data": event} I can achieve this by using the ruby filter, but I wa… --- ## [Logstash/elasticsearch failed to parse date field tried both date format \[dateOptionalTime\], and timestamp number with locale](https://discuss.elastic.co/t/logstash-elasticsearch-failed-to-parse-date-field-tried-both-date-format-dateoptionaltime-and-timestamp-number-with-locale/48442)
**Author:** [@Dee](https://discuss.elastic.co/u/Dee)\ **Replies:** 9\ **Last updated:** [September 8, 2016, 1:10pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-failed-to-parse-date-field-tried-both-date-format-dateoptionaltime-and-timestamp-number-with-locale/48442 "2016-09-08T13:10:35Z")
Hi, for some reason, logstash (version 1.5) can't process logs with this exception: {:timestamp=\>"2016-04-26T09:20:12.141000-0400", :message=\>"Failed parsing date from field", :field=\>"time", :value=\>"2016-04-26T09:20… --- ## [Creating subfileds in logstash](https://discuss.elastic.co/t/creating-subfileds-in-logstash/60070)
**Author:** [@Mahdy\_S](https://discuss.elastic.co/u/Mahdy_S)\ **Replies:** 4\ **Last updated:** [September 8, 2016, 1:38pm UTC](https://discuss.elastic.co/t/creating-subfileds-in-logstash/60070 "2016-09-08T13:38:15Z")
Dear Experts, I am trying to parse some logs with logstahs 2.3.3 into elasticsearch. The logs have similar format up to the log message, say: time\_stamp, node, field1, field2, log\_type, log\_message. Until now I was par… --- ## [How to use fliter's json to parsing nested json](https://discuss.elastic.co/t/how-to-use-fliters-json-to-parsing-nested-json/60072)
**Author:** [@WangXiangUSTC](https://discuss.elastic.co/u/WangXiangUSTC)\ **Replies:** 1\ **Last updated:** [September 8, 2016, 1:25pm UTC](https://discuss.elastic.co/t/how-to-use-fliters-json-to-parsing-nested-json/60072 "2016-09-08T13:25:18Z")
I use the fliter's json to parse nested json, the log is like below: { "time":"2016-09-08T20:25:48.657327288+08:00", "requested\_id":"34545665", "body":"{\\"data\\":\\"fheufrgjidjweidwjfndfefm",\\"code\\":\\"200\\"}" } this … --- ## [How to close logstash using batch file command](https://discuss.elastic.co/t/how-to-close-logstash-using-batch-file-command/60053)
**Author:** [@Lucifer\_hbp](https://discuss.elastic.co/u/Lucifer_hbp)\ **Replies:** 2\ **Last updated:** [September 8, 2016, 12:25pm UTC](https://discuss.elastic.co/t/how-to-close-logstash-using-batch-file-command/60053 "2016-09-08T12:25:39Z")
Hi again :slight\_smile: i would be glad if someone could provide helpful suggestion! Logstash version : 2.3.2 I have written a batch file that start the logstash and starts reading config file. Following is the batch… --- ## [Can a particular record in index be updated by logstash conf file?](https://discuss.elastic.co/t/can-a-particular-record-in-index-be-updated-by-logstash-conf-file/59979)
**Author:** [@Varun\_Breja](https://discuss.elastic.co/u/Varun_Breja)\ **Replies:** 4\ **Last updated:** [September 8, 2016, 11:15am UTC](https://discuss.elastic.co/t/can-a-particular-record-in-index-be-updated-by-logstash-conf-file/59979 "2016-09-08T11:15:34Z")
We have a scenario like this: Server Error Count\_of\_Error D1 ER1 15 D1 ER2 15 Questions: Can this count be updated in Index in logstash configuration file? Will this maintain atomicity in case t… --- ## [How to capture regex groups with /g?](https://discuss.elastic.co/t/how-to-capture-regex-groups-with-g/55858)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 3\ **Last updated:** [September 8, 2016, 10:46am UTC](https://discuss.elastic.co/t/how-to-capture-regex-groups-with-g/55858 "2016-09-08T10:46:53Z")
Hi, I have following excerpt of a log line: ... \[NO SORT\], #0...#4000 (PageSize: 4001), , Fetched: 13, 687 ms0...END (PageSize: INFINITE), , Fetched: 147, 15 ms0...END (PageSize: INFINITE), , Fetched: 147, 16 ms0...E… --- ## [How to use the second level of a fields](https://discuss.elastic.co/t/how-to-use-the-second-level-of-a-fields/60035)
**Author:** [@WangXiangUSTC](https://discuss.elastic.co/u/WangXiangUSTC)\ **Replies:** 2\ **Last updated:** [September 8, 2016, 10:31am UTC](https://discuss.elastic.co/t/how-to-use-the-second-level-of-a-fields/60035 "2016-09-08T10:31:25Z")
I add a new field "fields.ip "in filebeat's configure file, it look like below： filebeat: prospectors: - input\_type: log paths: - /home/vagarnt/log/\* fields: ip: 192.168.33.11 type: formal encoding:… --- ## [RESOLVED Logstash does not load a big file under win 7](https://discuss.elastic.co/t/resolved-logstash-does-not-load-a-big-file-under-win-7/59915)
**Author:** [@echocharlie](https://discuss.elastic.co/u/echocharlie)\ **Replies:** 1\ **Last updated:** [September 8, 2016, 9:50am UTC](https://discuss.elastic.co/t/resolved-logstash-does-not-load-a-big-file-under-win-7/59915 "2016-09-08T09:50:54Z")
I'm trying to index a log file which size is greater than 100 mb using logstash 2.4.0 and elastic search 2.4.0 If i reduce the file size by removing log entries everything works fine. Runing ELK under Windows 7. --- ## [Why logstash.err and logstash.stdout are created with root privileges?](https://discuss.elastic.co/t/why-logstash-err-and-logstash-stdout-are-created-with-root-privileges/59890)
**Author:** [@Saravanan](https://discuss.elastic.co/u/Saravanan)\ **Replies:** 2\ **Last updated:** [September 8, 2016, 8:39am UTC](https://discuss.elastic.co/t/why-logstash-err-and-logstash-stdout-are-created-with-root-privileges/59890 "2016-09-08T08:39:00Z")
Hi, I am new logstash, i've installed logstash-2.2.4 on centos linux and placed my config file at /etc/logstash/conf.d folder and try to start logstash using "service logstash start" command as a root user. By default… --- ## [Logstash syslog output - remove headers](https://discuss.elastic.co/t/logstash-syslog-output-remove-headers/60049)
**Author:** [@dorr](https://discuss.elastic.co/u/dorr)\ **Replies:** 1\ **Last updated:** [September 8, 2016, 8:19am UTC](https://discuss.elastic.co/t/logstash-syslog-output-remove-headers/60049 "2016-09-08T08:19:38Z")
Hello, I am using the syslog output, and a customer recently asked if I could send the syslog message without the headers, to simplify the message's parsing. I do want to use the syslog transport, but I want the sent s… --- ## [JSON filter (attempt to either prevent recursion or drop unwanted objects / nested fields)](https://discuss.elastic.co/t/json-filter-attempt-to-either-prevent-recursion-or-drop-unwanted-objects-nested-fields/52903)
**Author:** [@robinpblanchard](https://discuss.elastic.co/u/robinpblanchard)\ **Replies:** 3\ **Last updated:** [September 8, 2016, 8:05am UTC](https://discuss.elastic.co/t/json-filter-attempt-to-either-prevent-recursion-or-drop-unwanted-objects-nested-fields/52903 "2016-09-08T08:05:31Z")
I have java/logback feeding a template a la: \ { "environment" : "somecrazyenvironme… --- ## [Parse not valid Json logs](https://discuss.elastic.co/t/parse-not-valid-json-logs/59904)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 1\ **Last updated:** [September 8, 2016, 8:01am UTC](https://discuss.elastic.co/t/parse-not-valid-json-logs/59904 "2016-09-08T08:01:45Z")
I have some logs coming from my S3 bucket. The json in it is not valid since it has some non-standard " and { characters like below: {“app\_id":"fo","geid":"031fc83e0fea5c1e808857bd2db007d5","etid":"generic","ev":1,"dat… --- ## [Input ElasticSearch output Redis using Logstash](https://discuss.elastic.co/t/input-elasticsearch-output-redis-using-logstash/59887)
**Author:** [@Ankit\_Pradhan](https://discuss.elastic.co/u/Ankit_Pradhan)\ **Replies:** 1\ **Last updated:** [September 8, 2016, 8:00am UTC](https://discuss.elastic.co/t/input-elasticsearch-output-redis-using-logstash/59887 "2016-09-08T08:00:48Z")
I want to ship Data from Source Elasticsearch to Target ElasticSearch via Redis using Logstash ... The problem is one field (named as eventTime) which is of type Date in Source ElasticSearch index is not getting creat… --- ## [Incorrect formatting from logstash](https://discuss.elastic.co/t/incorrect-formatting-from-logstash/59965)
**Author:** [@trenzalore](https://discuss.elastic.co/u/trenzalore)\ **Replies:** 2\ **Last updated:** [September 8, 2016, 7:31am UTC](https://discuss.elastic.co/t/incorrect-formatting-from-logstash/59965 "2016-09-08T07:31:00Z")
Hi ! I'm new in the elastic community so sorry in advance if I miss something obvious. So I'm setting up and ELK stack with filebeat on a client. My client has two prospectors: one which sends apache logs (document… --- ## [Running problem logstash\_5.0.0\_alpha5 with new configuration file](https://discuss.elastic.co/t/running-problem-logstash-5-0-0-alpha5-with-new-configuration-file/60043)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 0\ **Last updated:** [September 8, 2016, 7:16am UTC](https://discuss.elastic.co/t/running-problem-logstash-5-0-0-alpha5-with-new-configuration-file/60043 "2016-09-08T07:16:34Z")
I install Logstash deb file. MY logstash configuration file like these; vim /etc/logstash/conf.d/02-beats-input.conf input { beats { port =\> 5044 ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/log… --- ## [There is huge delay starting up logstash 5.0.0-alpha5](https://discuss.elastic.co/t/there-is-huge-delay-starting-up-logstash-5-0-0-alpha5/59935)
**Author:** [@madkoala](https://discuss.elastic.co/u/madkoala)\ **Replies:** 6\ **Last updated:** [September 8, 2016, 3:19am UTC](https://discuss.elastic.co/t/there-is-huge-delay-starting-up-logstash-5-0-0-alpha5/59935 "2016-09-08T03:19:40Z")
We're using Elastic Stack 5.0.0-alpha5. We have very big trouble starting up Logstash. Logstash won't start up after executing (as a service) over 1+ hours. Process is alive, but there's no messages or logs even with … --- ## [Logstash s3 Plugin error](https://discuss.elastic.co/t/logstash-s3-plugin-error/60025)
**Author:** [@Capt\_X](https://discuss.elastic.co/u/Capt_X)\ **Replies:** 0\ **Last updated:** [September 8, 2016, 1:06am UTC](https://discuss.elastic.co/t/logstash-s3-plugin-error/60025 "2016-09-08T01:06:54Z")
Environment: RHEL 6.8 Logstash 2.4.0 Kibana 4.5.4 Amazon ElasticSearch 2.3 I've researched the error I am getting and while many have seen similar errors, I haven't found someone with the exact same error. I've tried n… --- ## [Problem with Logstash 2.4 with elasticsearch output](https://discuss.elastic.co/t/problem-with-logstash-2-4-with-elasticsearch-output/59969)
**Author:** [@Kissaki](https://discuss.elastic.co/u/Kissaki)\ **Replies:** 5\ **Last updated:** [September 7, 2016, 9:18pm UTC](https://discuss.elastic.co/t/problem-with-logstash-2-4-with-elasticsearch-output/59969 "2016-09-07T21:18:32Z")
Hey guys, I'm just getting started with the ELK stack and have some problems testing Logstash with Elasticsearch. I run a fresh fedora 24 installation and already installed Elasticsearch 2.4 on it. The curl test on loc… --- ## [Restarting Logstash vs Manual Stopping & Starting](https://discuss.elastic.co/t/restarting-logstash-vs-manual-stopping-starting/60017)
**Author:** [@seth.yes](https://discuss.elastic.co/u/seth.yes)\ **Replies:** 0\ **Last updated:** [September 7, 2016, 9:02pm UTC](https://discuss.elastic.co/t/restarting-logstash-vs-manual-stopping-starting/60017 "2016-09-07T21:02:21Z")
I'm doing constant testing on a rather-large filter set in Logstash (~1800 lines). Restarting Logstash takes 3-5 minutes every time, which really slows down my testing efficiency. I'm considering simply running the co… --- ## [Multiple jdbc SQL Statements ( & schedules, ideally)](https://discuss.elastic.co/t/multiple-jdbc-sql-statements-schedules-ideally/59930)
**Author:** [@seth.yes](https://discuss.elastic.co/u/seth.yes)\ **Replies:** 2\ **Last updated:** [September 7, 2016, 4:04pm UTC](https://discuss.elastic.co/t/multiple-jdbc-sql-statements-schedules-ideally/59930 "2016-09-07T16:04:49Z")
I'm using the most up to date version of Logstash (1:2.2.4-1) and the associated jdbc logstash plugin. My one SQL statement is configured and operating on schedule as expected, querying a MS SQL instance every minute. Ho… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1103) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1105)