# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1192 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1193 --- ## [Logstash high availability](https://discuss.elastic.co/t/logstash-high-availability/26796)
**Author:** [@saif](https://discuss.elastic.co/u/saif)\ **Replies:** 1\ **Last updated:** [February 11, 2016, 11:37pm UTC](https://discuss.elastic.co/t/logstash-high-availability/26796 "2016-02-11T23:37:19Z")
hi im using 2 logstash servers as shipper actually im trying to set up HAporxy and Keepalived to set the LB between the server because im use rsyslog as log forworder to send logs to an VIP (haproxy) this is the best… --- ## [Logstash optimization](https://discuss.elastic.co/t/logstash-optimization/41540)
**Author:** [@aaroy428](https://discuss.elastic.co/u/aaroy428)\ **Replies:** 1\ **Last updated:** [February 11, 2016, 8:42pm UTC](https://discuss.elastic.co/t/logstash-optimization/41540 "2016-02-11T20:42:41Z")
When running on a dedicated server for logstash can we improve the logstash performance by increasing the nice value to 0 in the /etc/init.d/logstash startup script. Is there any os optimization possible to speed up logs… --- ## [Jmeter parsing by date (ELK)](https://discuss.elastic.co/t/jmeter-parsing-by-date-elk/41491)
**Author:** [@Qassim](https://discuss.elastic.co/u/Qassim)\ **Replies:** 1\ **Last updated:** [February 11, 2016, 8:32pm UTC](https://discuss.elastic.co/t/jmeter-parsing-by-date-elk/41491 "2016-02-11T20:32:17Z")
Hi All, I have been struggling with getting Jmeter logs (csv) into an ELK stack and parsing them by the timestamp in the provided file. I can get the logs in fine, but they're indexed by the time logstash has processed… --- ## [Date parsing log without date in it](https://discuss.elastic.co/t/date-parsing-log-without-date-in-it/41435)
**Author:** [@yammy](https://discuss.elastic.co/u/yammy)\ **Replies:** 3\ **Last updated:** [February 11, 2016, 7:11pm UTC](https://discuss.elastic.co/t/date-parsing-log-without-date-in-it/41435 "2016-02-11T19:11:27Z")
I have a log that I need to collect, however it only contains the timestamp at the begining with no date. Example: 145231.hostname!process1.19022.2799412992.0: msg1 "INFO: msg2" I am successfully parsing the data with… --- ## [Usage of the Metrics filter - Adding a field from the original event](https://discuss.elastic.co/t/usage-of-the-metrics-filter-adding-a-field-from-the-original-event/41518)
**Author:** [@Rumbles](https://discuss.elastic.co/u/Rumbles)\ **Replies:** 0\ **Last updated:** [February 11, 2016, 4:35pm UTC](https://discuss.elastic.co/t/usage-of-the-metrics-filter-adding-a-field-from-the-original-event/41518 "2016-02-11T16:35:23Z")
I have created a mongodb filter, which gets me a number of fields, since the only logs that go through are slow logs I want to know how many slow queries and commands are running through my cluster, so I created a metric… --- ## [Logstash JDBC input malfunction](https://discuss.elastic.co/t/logstash-jdbc-input-malfunction/38601)
**Author:** [@tilney](https://discuss.elastic.co/u/tilney)\ **Replies:** 1\ **Last updated:** [February 11, 2016, 1:32pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-malfunction/38601 "2016-02-11T13:32:48Z")
configuration below could not finish if I dont specify a limit clause in mysql statement. input { jdbc { jdbc\_driver\_library =\> "mysql-connector-java-5.1.38/mysql-connector-java-5.1.38-bin.jar" jdbc… --- ## [Logs not back filled if logstash indexer pipeline is down](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118)
**Author:** [@astro](https://discuss.elastic.co/u/astro)\ **Replies:** 9\ **Last updated:** [February 11, 2016, 1:21pm UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118 "2016-02-11T13:21:17Z")
Hi Everyone, I am not able to figure out why logs are not back filled when indexer logstash is down although logs are buffered in kafka . Regards Arvind --- ## [Add\_field with json codec](https://discuss.elastic.co/t/add-field-with-json-codec/41411)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 5\ **Last updated:** [February 11, 2016, 10:27am UTC](https://discuss.elastic.co/t/add-field-with-json-codec/41411 "2016-02-11T10:27:20Z")
Hi, I'm working with json files and using the json codec to get at the values in each document. So far so good. I now need to add some new fields to the document which will contain values taken from nested json. F… --- ## ["Logstash shutdown completed" not happening](https://discuss.elastic.co/t/logstash-shutdown-completed-not-happening/41454)
**Author:** [@Architha](https://discuss.elastic.co/u/Architha)\ **Replies:** 2\ **Last updated:** [February 11, 2016, 9:34am UTC](https://discuss.elastic.co/t/logstash-shutdown-completed-not-happening/41454 "2016-02-11T09:34:46Z")
Hi All, When i run my Logstash config i see the following output : io/console not supported; tty will not be manipulated Settings: Default filter workers: 1 Logstash startup completed The shutdown process is not h… --- ## [Data mixed between indices](https://discuss.elastic.co/t/data-mixed-between-indices/41355)
**Author:** [@LoZio](https://discuss.elastic.co/u/LoZio)\ **Replies:** 9\ **Last updated:** [February 11, 2016, 8:13am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355 "2016-02-11T08:13:31Z")
I have a pretty simple setup. ES 2.2 and logstash 2.2.0. I have several conf for logstash, each one defines a port to listen to, labels data with "type=\>xyz" and puts in an index with the same name and date time referen… --- ## [Chinese Characters in Grok](https://discuss.elastic.co/t/chinese-characters-in-grok/40314)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 6\ **Last updated:** [February 11, 2016, 7:10am UTC](https://discuss.elastic.co/t/chinese-characters-in-grok/40314 "2016-02-11T07:10:39Z")
I have a log that is in Chinese. An example will be 2016-01-20T17:27:22+08:00 Server1 日志类型:电脑操作, 用户:user12 I have created a grok pattern for parsing this log. I verified it in https://grokdebug.herokuapp.com/ The … --- ## [Grok: Pattern appears to be matching but not creating new fields](https://discuss.elastic.co/t/grok-pattern-appears-to-be-matching-but-not-creating-new-fields/41330)
**Author:** [@Wolls013](https://discuss.elastic.co/u/Wolls013)\ **Replies:** 3\ **Last updated:** [February 11, 2016, 6:45am UTC](https://discuss.elastic.co/t/grok-pattern-appears-to-be-matching-but-not-creating-new-fields/41330 "2016-02-11T06:45:49Z")
Here is my Input 2015-06-02 17:13:51.331 +00:00 Here is the filter and match pattern that i'm using } filter { grok { match =\> { "time" =\> "%{TIMESTAMP\_ISO8601: event\_timestamp}" } } } The pattern appear… --- ## [Mapper field type being changed by something](https://discuss.elastic.co/t/mapper-field-type-being-changed-by-something/41452)
**Author:** [@Wolls013](https://discuss.elastic.co/u/Wolls013)\ **Replies:** 4\ **Last updated:** [February 11, 2016, 5:19am UTC](https://discuss.elastic.co/t/mapper-field-type-being-changed-by-something/41452 "2016-02-11T05:19:31Z")
Hi Everyone, I'm pretty new to new to ELK and i've made a mapper "location" which is a geo\_point type so I can pass my lat and long data to it. This is how I've mapped it \`PUT my\_index { "mappings": { "location… --- ## [Remove unnecessary fields in ElasticSearch](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673)
**Author:** [@Hayder\_Abbass](https://discuss.elastic.co/u/Hayder_Abbass)\ **Replies:** 10\ **Last updated:** [February 11, 2016, 2:45am UTC](https://discuss.elastic.co/t/remove-unnecessary-fields-in-elasticsearch/29673 "2016-02-11T02:45:00Z")
Hello, We are populating Elasticsearch via logstash. The thing is that I see some unnecessary fields that I had like to remove like for example: @version file geoip host message offset tags Is it possible to do this… --- ## [Missing Elasticsearch Output Running Logstash in Kubernetes](https://discuss.elastic.co/t/missing-elasticsearch-output-running-logstash-in-kubernetes/40798)
**Author:** [@rutschman](https://discuss.elastic.co/u/rutschman)\ **Replies:** 5\ **Last updated:** [February 11, 2016, 12:37am UTC](https://discuss.elastic.co/t/missing-elasticsearch-output-running-logstash-in-kubernetes/40798 "2016-02-11T00:37:33Z")
Hello, I've been running logstash on virtual machines for a bit and am trying to move it to Kubernetes to help scale and better manage configuration. When running logstash in Kubernetes events bound for elasticsearch a… --- ## [Unable to get logstash guide example working](https://discuss.elastic.co/t/unable-to-get-logstash-guide-example-working/41109)
**Author:** [@samba1](https://discuss.elastic.co/u/samba1)\ **Replies:** 7\ **Last updated:** [February 10, 2016, 10:15pm UTC](https://discuss.elastic.co/t/unable-to-get-logstash-guide-example-working/41109 "2016-02-10T22:15:12Z")
I'm following along with this: https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html And was on the second part. I decided to not do twitter to avoid signing up for a dev key and giving them my phon… --- ## [Grok Regex - Captured and Non-Captured Groups](https://discuss.elastic.co/t/grok-regex-captured-and-non-captured-groups/41417)
**Author:** [@bhatch](https://discuss.elastic.co/u/bhatch)\ **Replies:** 2\ **Last updated:** [February 10, 2016, 8:15pm UTC](https://discuss.elastic.co/t/grok-regex-captured-and-non-captured-groups/41417 "2016-02-10T20:15:56Z")
I have some data coming into Logstash that has an email address. What I want to do is create a new column with just the domain. This will allow us to better look for trends and issues by domain. None of the existing G… --- ## [Switching from lumberjack input to Beats for multiline events shipped from forwarders](https://discuss.elastic.co/t/switching-from-lumberjack-input-to-beats-for-multiline-events-shipped-from-forwarders/41413)
**Author:** [@michelecappelletti](https://discuss.elastic.co/u/michelecappelletti)\ **Replies:** 1\ **Last updated:** [February 10, 2016, 7:08pm UTC](https://discuss.elastic.co/t/switching-from-lumberjack-input-to-beats-for-multiline-events-shipped-from-forwarders/41413 "2016-02-10T19:08:23Z")
Hi, since i've moved multiline management from filter layer to input layer, i've doubts on how to do it correctly using lumberjack or the new version of beats (that ive never used before) In production enviroment actuall… --- ## [Wrong post](https://discuss.elastic.co/t/wrong-post/41412)
**Author:** [@sharon.c](https://discuss.elastic.co/u/sharon.c)\ **Replies:** 0\ **Last updated:** [February 10, 2016, 5:48pm UTC](https://discuss.elastic.co/t/wrong-post/41412 "2016-02-10T17:48:11Z")
wrong post --- ## [Logstash - Beats input: The circuit breaker has detected a slowdown](https://discuss.elastic.co/t/logstash-beats-input-the-circuit-breaker-has-detected-a-slowdown/41398)
**Author:** [@MikeC](https://discuss.elastic.co/u/MikeC)\ **Replies:** 0\ **Last updated:** [February 10, 2016, 4:04pm UTC](https://discuss.elastic.co/t/logstash-beats-input-the-circuit-breaker-has-detected-a-slowdown/41398 "2016-02-10T16:04:20Z")
Hi all, im trying to ship some file logs with this setup filebeat(host1) -\> logstash(host1) --+-\> elasticsearch | filebeat(host2) -\> logstash(host2) --+ … --- ## [Add a field](https://discuss.elastic.co/t/add-a-field/41378)
**Author:** [@elastic-fan](https://discuss.elastic.co/u/elastic-fan)\ **Replies:** 2\ **Last updated:** [February 10, 2016, 2:04pm UTC](https://discuss.elastic.co/t/add-a-field/41378 "2016-02-10T14:04:05Z")
hi, i am getting logs from the server and in logstash i would like to add a field and value for it how can i do this --- ## [Force logstash-forwarder to index newest events first](https://discuss.elastic.co/t/force-logstash-forwarder-to-index-newest-events-first/41381)
**Author:** [@nkoleff](https://discuss.elastic.co/u/nkoleff)\ **Replies:** 0\ **Last updated:** [February 10, 2016, 1:19pm UTC](https://discuss.elastic.co/t/force-logstash-forwarder-to-index-newest-events-first/41381 "2016-02-10T13:19:33Z")
Hello, I'm shipping logs with logstash-forwarder and for one of my types I can see that its mainly indexing old data (mostly near the dead time), whcih results in not having events for the last four hours. Instead, it's… --- ## [Access to a element in a array](https://discuss.elastic.co/t/access-to-a-element-in-a-array/41376)
**Author:** [@m.francia](https://discuss.elastic.co/u/m.francia)\ **Replies:** 0\ **Last updated:** [February 10, 2016, 12:44pm UTC](https://discuss.elastic.co/t/access-to-a-element-in-a-array/41376 "2016-02-10T12:44:51Z")
Hi! I have a json that looks like this { "aplication": "aula3vga", "nclients": "6", "anchoBanda\[Mbs\]": 9.315848, "dataclients": \[ { "aplication": "aula3vga", "ipclien… --- ## [Index template create problem with logstash 2.1.1](https://discuss.elastic.co/t/index-template-create-problem-with-logstash-2-1-1/41368)
**Author:** [@paltryeffort](https://discuss.elastic.co/u/paltryeffort)\ **Replies:** 1\ **Last updated:** [February 10, 2016, 12:23pm UTC](https://discuss.elastic.co/t/index-template-create-problem-with-logstash-2-1-1/41368 "2016-02-10T12:23:17Z")
I am trying to upgrade from logstash 1.4.2 to logstash 2.1.1 and I have a problem when a new index gets created. I tried to reduce all configs to the minimum and now I have the following: The most important piece is my… --- ## [Installing webhdfs output plugin on Logstash 2.2.0](https://discuss.elastic.co/t/installing-webhdfs-output-plugin-on-logstash-2-2-0/41174)
**Author:** [@tjc45](https://discuss.elastic.co/u/tjc45)\ **Replies:** 1\ **Last updated:** [February 10, 2016, 12:11pm UTC](https://discuss.elastic.co/t/installing-webhdfs-output-plugin-on-logstash-2-2-0/41174 "2016-02-10T12:11:04Z")
I'm trying to install the webhdfs output plugin on a fresh installation of Logstash 2.2.0. I'm getting the error shown below - any idea how I might identify what the problem is? sudo /opt/logstash/bin/plugin install l… --- ## [Bug or my usage of Mutate filter?](https://discuss.elastic.co/t/bug-or-my-usage-of-mutate-filter/41202)
**Author:** [@LeeSyd](https://discuss.elastic.co/u/LeeSyd)\ **Replies:** 2\ **Last updated:** [February 10, 2016, 12:02pm UTC](https://discuss.elastic.co/t/bug-or-my-usage-of-mutate-filter/41202 "2016-02-10T12:02:24Z")
Not sure if this is a bug or I'm not getting the syntax right with the Mutate filter but I'm trying to configure Topbeat and mutate the field name to match others I already have in elasticsearch. I've got: … --- ## [Delete inner fields (from json) using lookup file](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 2\ **Last updated:** [February 10, 2016, 7:29am UTC](https://discuss.elastic.co/t/delete-inner-fields-from-json-using-lookup-file/41294 "2016-02-10T07:29:04Z")
Hi, I'm processing quite large files using the json input codec. It's doing a great job of splitting up the data, creating fields and passing it to ES. However... Some of the documents have a nested value a bit like… --- ## [Nessus Scanner Crashes Logstash Syslog Inputs](https://discuss.elastic.co/t/nessus-scanner-crashes-logstash-syslog-inputs/41328)
**Author:** [@arizonawayfarer](https://discuss.elastic.co/u/arizonawayfarer)\ **Replies:** 1\ **Last updated:** [February 10, 2016, 6:53am UTC](https://discuss.elastic.co/t/nessus-scanner-crashes-logstash-syslog-inputs/41328 "2016-02-10T06:53:16Z")
Hi everyone, I'm not sure where/how to file a bug report - but we have a nessus scanner running on our network that sends garbage to tcp/udp ports it discovers to find exploits. When this happens, it kills the logsta… --- ## [Logstash sends last event only on shutdown](https://discuss.elastic.co/t/logstash-sends-last-event-only-on-shutdown/41348)
**Author:** [@vsadokhin](https://discuss.elastic.co/u/vsadokhin)\ **Replies:** 5\ **Last updated:** [February 10, 2016, 5:08am UTC](https://discuss.elastic.co/t/logstash-sends-last-event-only-on-shutdown/41348 "2016-02-10T05:08:01Z")
Hello I use logstash to parse log files and send result documents to elasticsearch. It seems to me that last log/event is not sent until new one appears. The only way I managed to make logstash send last event is turnin… --- ## [Rabbitmq input plugin does not acknowledge after idle time](https://discuss.elastic.co/t/rabbitmq-input-plugin-does-not-acknowledge-after-idle-time/41323)
**Author:** [@sripri](https://discuss.elastic.co/u/sripri)\ **Replies:** 0\ **Last updated:** [February 9, 2016, 10:14pm UTC](https://discuss.elastic.co/t/rabbitmq-input-plugin-does-not-acknowledge-after-idle-time/41323 "2016-02-09T22:14:59Z")
I am running into an issue with logstash not reading from a queue after an idle time. If I load the que up with messages constantly, everything works fine. If I idle the queue for more than 10 minutes or so (varies) t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1191) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1193)