# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1263 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1264 --- ## [Re: "create action does not use \_id for request #79"](https://discuss.elastic.co/t/re-create-action-does-not-use--id-for-request-79/1501)
**Author:** [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\ **Replies:** 1\ **Last updated:** [May 29, 2015, 10:52am UTC](https://discuss.elastic.co/t/re-create-action-does-not-use--id-for-request-79/1501 "2015-05-29T10:52:10Z")
create action does not use \_id for request #79 hxxps://github.com/logstash-plugins/logstash-output-elasticsearch/issues/79 This means that I can not use my own deduplication where I manually set the document\_id to a gi… --- ## [How to get all matches for a grok pattern in a multiline message](https://discuss.elastic.co/t/how-to-get-all-matches-for-a-grok-pattern-in-a-multiline-message/181)
**Author:** [@markus](https://discuss.elastic.co/u/markus)\ **Replies:** 4\ **Last updated:** [May 29, 2015, 9:13am UTC](https://discuss.elastic.co/t/how-to-get-all-matches-for-a-grok-pattern-in-a-multiline-message/181 "2015-05-29T09:13:47Z")
Hi, I am having a multiline log format that consists of an XML document with a very deep message structure. All elements in the data structure can have a child element called Message containing a value attribute like so: … --- ## [Logstash issue grok](https://discuss.elastic.co/t/logstash-issue-grok/1251)
**Author:** [@alexmac](https://discuss.elastic.co/u/alexmac)\ **Replies:** 11\ **Last updated:** [May 29, 2015, 7:21am UTC](https://discuss.elastic.co/t/logstash-issue-grok/1251 "2015-05-29T07:21:48Z")
Hi, Im having an issue with logsasher, after sending couple of events it stop sending logs to elastic. basically Im inputing AWS ELB logs, extracting the url\_path on first match and continue extracting additional data fr… --- ## [Inconsistency: input elasticsearch does not require "cluster" but output elastic search requires it](https://discuss.elastic.co/t/inconsistency-input-elasticsearch-does-not-require-cluster-but-output-elastic-search-requires-it/1520)
**Author:** [@discuss03](https://discuss.elastic.co/u/discuss03)\ **Replies:** 3\ **Last updated:** [May 29, 2015, 7:05am UTC](https://discuss.elastic.co/t/inconsistency-input-elasticsearch-does-not-require-cluster-but-output-elastic-search-requires-it/1520 "2015-05-29T07:05:29Z")
There seems to be a inconsistency between the input and output versions for logstash elasticsearch. Input elastic search does not understand the parameter "host" even though it is documented on the website. Input el… --- ## [How to filter the logs which has the numeric value greater than specific number in particular field?](https://discuss.elastic.co/t/how-to-filter-the-logs-which-has-the-numeric-value-greater-than-specific-number-in-particular-field/1360)
**Author:** [@selvam](https://discuss.elastic.co/u/selvam)\ **Replies:** 5\ **Last updated:** [May 29, 2015, 4:39am UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-which-has-the-numeric-value-greater-than-specific-number-in-particular-field/1360 "2015-05-29T04:39:44Z")
I am running tomcat service in client. I would like to monitor tomcat access logs and filter the logs which has response time greater than specific number. e.g : I have the tomcat access log. 192.168.1.10 - - \[26/Ma… --- ## [How do we match multiple random ips?](https://discuss.elastic.co/t/how-do-we-match-multiple-random-ips/1507)
**Author:** [@Ben](https://discuss.elastic.co/u/Ben)\ **Replies:** 2\ **Last updated:** [May 28, 2015, 9:22pm UTC](https://discuss.elastic.co/t/how-do-we-match-multiple-random-ips/1507 "2015-05-28T21:22:45Z")
I am trying to match random number of ips in my access logs. Basically, the access logs looks like this 10.xx.xx.xx 107.xx.xxx.xxx 107.xx.xx.xx, 184.xx.xx.xx, 131.xx.xx.xx, 74.xx.xx.xxx, 209.xx.xx.xxx, 10.xx.xx.x, 10.… --- ## [Not able to create Index in ES by using logstash](https://discuss.elastic.co/t/not-able-to-create-index-in-es-by-using-logstash/1444)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 3\ **Last updated:** [May 28, 2015, 6:38pm UTC](https://discuss.elastic.co/t/not-able-to-create-index-in-es-by-using-logstash/1444 "2015-05-28T18:38:20Z")
Hi All, I use this config file for twitter input.But its not creating index in Elastic Search. When i run this config file for first time its created index.But next time when i want to create new index by using same c… --- ## [Real Data Centers Logs Velocity](https://discuss.elastic.co/t/real-data-centers-logs-velocity/1455)
**Author:** [@ChrisMark](https://discuss.elastic.co/u/ChrisMark)\ **Replies:** 1\ **Last updated:** [May 28, 2015, 11:03am UTC](https://discuss.elastic.co/t/real-data-centers-logs-velocity/1455 "2015-05-28T11:03:37Z")
Hello, i have a general question. I was wondering what huge amounts of data dailiy, in a high velocity, a large cluster of a cloud provider like Amazon or Rackspace produce. I am not sure if this is the right topic f… --- ## [Logstash forwarder 100s of processes](https://discuss.elastic.co/t/logstash-forwarder-100s-of-processes/1459)
**Author:** [@Wayne\_Oliver](https://discuss.elastic.co/u/Wayne_Oliver)\ **Replies:** 0\ **Last updated:** [May 28, 2015, 11:03am UTC](https://discuss.elastic.co/t/logstash-forwarder-100s-of-processes/1459 "2015-05-28T11:03:20Z")
Hi all. I am seeing the following in my process list. Any ideas of what I can look at to resolve? root 27940 0.0 0.0 320308 5920 ? Sl May26 1:59 /opt/lumberjack/bin/lumberjack -config /etc/lumberjack… --- ## [\[SOLVED\] Logstash not parsing UTM logs correctly](https://discuss.elastic.co/t/solved-logstash-not-parsing-utm-logs-correctly/955)
**Author:** [@Wurzelmann](https://discuss.elastic.co/u/Wurzelmann)\ **Replies:** 3\ **Last updated:** [May 28, 2015, 9:06am UTC](https://discuss.elastic.co/t/solved-logstash-not-parsing-utm-logs-correctly/955 "2015-05-28T09:06:06Z")
I've got some UTM logs I need to process with logstash, but it only parsing half of it. The logs are divided into four parts, so to speak, namely: 1) ulogd 2) aua 3) reverseproxy 4) wlan I've built a config file, whi… --- ## [Install specific plugin version](https://discuss.elastic.co/t/install-specific-plugin-version/1407)
**Author:** [@rectalogic](https://discuss.elastic.co/u/rectalogic)\ **Replies:** 1\ **Last updated:** [May 28, 2015, 1:16am UTC](https://discuss.elastic.co/t/install-specific-plugin-version/1407 "2015-05-28T01:16:30Z")
How do I install a specific version of a plugin? plugin install --version fails because the plugin is locked at an older version. Uninstalling and then installing also fails. I'm using logstash 1.5.0 from the deb repos… --- ## [Failed unmarshalling json: invalid character 'i' look ing for beginning of value](https://discuss.elastic.co/t/failed-unmarshalling-json-invalid-character-i-look-ing-for-beginning-of-value/1412)
**Author:** [@iinquisitivee](https://discuss.elastic.co/u/iinquisitivee)\ **Replies:** 0\ **Last updated:** [May 27, 2015, 5:46pm UTC](https://discuss.elastic.co/t/failed-unmarshalling-json-invalid-character-i-look-ing-for-beginning-of-value/1412 "2015-05-27T17:46:11Z")
my first excercise with logstash so pls pardon anything silly (runing on windows) I was trying it based on the blog post here.. http://blog.webkid.io/visualize-datasets-with-elk/ and get an error when running logstas… --- ## [Example in the 0-60 video, cannot reproduce](https://discuss.elastic.co/t/example-in-the-0-60-video-cannot-reproduce/1306)
**Author:** [@Paolo](https://discuss.elastic.co/u/Paolo)\ **Replies:** 2\ **Last updated:** [May 27, 2015, 10:56am UTC](https://discuss.elastic.co/t/example-in-the-0-60-video-cannot-reproduce/1306 "2015-05-27T10:56:03Z")
Hello All, I am a newbie, my aim is to analyze some log files created with the "liferay" environment. As a test I was trying to reproduce the log analysis of the 0-60 video. I copied both the log and the config file fro… --- ## [Logstash conf multiline management](https://discuss.elastic.co/t/logstash-conf-multiline-management/1321)
**Author:** [@Tibo](https://discuss.elastic.co/u/Tibo)\ **Replies:** 2\ **Last updated:** [May 27, 2015, 8:03am UTC](https://discuss.elastic.co/t/logstash-conf-multiline-management/1321 "2015-05-27T08:03:35Z")
Hi, I have a log file in input to logstash, something like that : \[....\] 2015-04-08 00:21:21,248 \[11 \] INFO Traitement de l'archive 20150313.tar-gz 2015-04-08 00:21:21,248 \[11 \] INFO Extraction de l'archive 20150313.… --- ## [How can i pull data from twitter to logstash](https://discuss.elastic.co/t/how-can-i-pull-data-from-twitter-to-logstash/1353)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 2\ **Last updated:** [May 27, 2015, 5:14am UTC](https://discuss.elastic.co/t/how-can-i-pull-data-from-twitter-to-logstash/1353 "2015-05-27T05:14:07Z")
Hi All, How can i pull data from twitter to logstash. Example: cocacola and its hashtag @cocacola I wan't only pull data from cocacola hashtag. Please give me step by step procedure. I can able to pull data from twit… --- ## [Problems with config date {} multiline files](https://discuss.elastic.co/t/problems-with-config-date-multiline-files/1287)
**Author:** [@Sergey](https://discuss.elastic.co/u/Sergey)\ **Replies:** 1\ **Last updated:** [May 27, 2015, 1:27am UTC](https://discuss.elastic.co/t/problems-with-config-date-multiline-files/1287 "2015-05-27T01:27:01Z")
Hello everyone! Any log file of the application. The application creates a single transaction = one log file. I use Logstash analysis of this file. The application repeatedly inserts timestamp in the log files. I make a… --- ## [Logstash file output creating wrong file?](https://discuss.elastic.co/t/logstash-file-output-creating-wrong-file/1349)
**Author:** [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Replies:** 2\ **Last updated:** [May 27, 2015, 12:57am UTC](https://discuss.elastic.co/t/logstash-file-output-creating-wrong-file/1349 "2015-05-27T00:57:14Z")
As well as outputting to ElasticSearch, I have Logstash outputting to a gzipped file with: file { path =\> "/var/data/logstash/archive-%{+YYYY-MM-dd}.log.gz" gzip =\> true } I then have a cronjob that runs … --- ## [ELK gathered data/rules documentation](https://discuss.elastic.co/t/elk-gathered-data-rules-documentation/1333)
**Author:** [@rafaltrojniak](https://discuss.elastic.co/u/rafaltrojniak)\ **Replies:** 1\ **Last updated:** [May 27, 2015, 12:19am UTC](https://discuss.elastic.co/t/elk-gathered-data-rules-documentation/1333 "2015-05-27T00:19:31Z")
Hello, I had been working with logstash for some time now. I had come to the point where I have couple of event sources, I am integrating next, and training operators to work with the data. I was wondering, if there … --- ## [File gzip output creating wrong file?](https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350)
**Author:** [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Replies:** 1\ **Last updated:** [May 27, 2015, 12:12am UTC](https://discuss.elastic.co/t/file-gzip-output-creating-wrong-file/1350 "2015-05-27T00:12:20Z")
Sorry about this - I managed to delete my previous post somehow! My issue is that I have an gzip output like: file { path =\> "/var/data/logstash/archive-%{+YYYY-MM-dd}.log.gz" gzip =\> true } The files cre… --- ## [Lumberjack input with multiple files](https://discuss.elastic.co/t/lumberjack-input-with-multiple-files/1195)
**Author:** [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Replies:** 2\ **Last updated:** [May 26, 2015, 10:30pm UTC](https://discuss.elastic.co/t/lumberjack-input-with-multiple-files/1195 "2015-05-26T22:30:28Z")
Hi All, With no stream\_identity field in multiline codec, can someone tell me how to handle multiple files coming in through lumberjack protocol? I am using Logstash 1.5 GA --- ## [Can't create logstash indexes in elasticsearch](https://discuss.elastic.co/t/cant-create-logstash-indexes-in-elasticsearch/1342)
**Author:** [@bluethundr](https://discuss.elastic.co/u/bluethundr)\ **Replies:** 1\ **Last updated:** [May 26, 2015, 8:54pm UTC](https://discuss.elastic.co/t/cant-create-logstash-indexes-in-elasticsearch/1342 "2015-05-26T20:54:31Z")
Hello, I've downloaded the binary tar of logstash 1.5.0 and I'm trying to get it working with a 3 node elasticsearch cluster running on ES 1.5.1. I have kibana 3 running under nginx. But when I go to my web browser a… --- ## [LogStash CloudTrail Integration](https://discuss.elastic.co/t/logstash-cloudtrail-integration/1340)
**Author:** [@ravi](https://discuss.elastic.co/u/ravi)\ **Replies:** 0\ **Last updated:** [May 26, 2015, 7:13pm UTC](https://discuss.elastic.co/t/logstash-cloudtrail-integration/1340 "2015-05-26T19:13:02Z")
Hi, Am looking for a working sample of integrating AWS cloudtrail logs with logstash. Regards, Ravi --- ## [Problem with analyzed fields in indexes](https://discuss.elastic.co/t/problem-with-analyzed-fields-in-indexes/1330)
**Author:** [@paul](https://discuss.elastic.co/u/paul)\ **Replies:** 2\ **Last updated:** [May 26, 2015, 7:03pm UTC](https://discuss.elastic.co/t/problem-with-analyzed-fields-in-indexes/1330 "2015-05-26T19:03:49Z")
I am trying to stop the field 'host' from being set to analyzed. Even when I can get it to say 'false' in the index description, it is still pulling apart the hostnames using '.' as a separator. I am using logstash 1.4.2… --- ## [Metrics filter out to Elasticsearch](https://discuss.elastic.co/t/metrics-filter-out-to-elasticsearch/869)
**Author:** [@jruybal](https://discuss.elastic.co/u/jruybal)\ **Replies:** 3\ **Last updated:** [May 26, 2015, 6:07pm UTC](https://discuss.elastic.co/t/metrics-filter-out-to-elasticsearch/869 "2015-05-26T18:07:27Z")
Hi, This seems simple. I wish I could say it was, but I cannot get metrics to show up in Kibana. filter{ metrics { meter =\> \[ "events" \] add\_tag =\> "metric" } } output { elasticsearch { cluster =\> … --- ## [How to Create JDBC connection using Logstash for Oracle DB?](https://discuss.elastic.co/t/how-to-create-jdbc-connection-using-logstash-for-oracle-db/1284)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [May 26, 2015, 7:24am UTC](https://discuss.elastic.co/t/how-to-create-jdbc-connection-using-logstash-for-oracle-db/1284 "2015-05-26T07:24:22Z")
How to Create JDBC connection using Logstash for Oracle DB? --- ## [Using filters with logstash-forwarder](https://discuss.elastic.co/t/using-filters-with-logstash-forwarder/1262)
**Author:** [@palmerabollo](https://discuss.elastic.co/u/palmerabollo)\ **Replies:** 1\ **Last updated:** [May 26, 2015, 4:37am UTC](https://discuss.elastic.co/t/using-filters-with-logstash-forwarder/1262 "2015-05-26T04:37:35Z")
I can use filters such as "environment" to add an environment variable to my logs with the logstash agent. This is very useful when you need to enrich your logs with metadata, for example: filter { environment { … --- ## [Not able to create Index in ES with Logstash](https://discuss.elastic.co/t/not-able-to-create-index-in-es-with-logstash/1173)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 5\ **Last updated:** [May 25, 2015, 5:33am UTC](https://discuss.elastic.co/t/not-able-to-create-index-in-es-with-logstash/1173 "2015-05-25T05:33:59Z")
Continuing the discussion from Not able to create Index with Logstash: --- ## [Gracefully Stopping Logstash & Avoiding kill-9](https://discuss.elastic.co/t/gracefully-stopping-logstash-avoiding-kill-9/1197)
**Author:** [@David\_Kleiner](https://discuss.elastic.co/u/David_Kleiner)\ **Replies:** 4\ **Last updated:** [May 25, 2015, 12:05am UTC](https://discuss.elastic.co/t/gracefully-stopping-logstash-avoiding-kill-9/1197 "2015-05-25T00:05:42Z")
Hello, Once in a while I need to restart logstash. I use SV for service management. Here's what happens: \[woot@log04 elasticsearch\]$ sudo sv stop /etc/service/backend-indexnode\* timeout: run: /etc/service/backend-ind… --- ## [Input file plugin under Windows is continuously looping through the file](https://discuss.elastic.co/t/input-file-plugin-under-windows-is-continuously-looping-through-the-file/1229)
**Author:** [@discuss03](https://discuss.elastic.co/u/discuss03)\ **Replies:** 0\ **Last updated:** [May 24, 2015, 1:15pm UTC](https://discuss.elastic.co/t/input-file-plugin-under-windows-is-continuously-looping-through-the-file/1229 "2015-05-24T13:15:02Z")
I am trying to use logstash under WIndows (don't ask)... and everything seems fine, my logstash configuration the input file definition, the grok filter, and my output to debug and elastic search... In order to test I … --- ## [How to parse multiple date fields?](https://discuss.elastic.co/t/how-to-parse-multiple-date-fields/1208)
**Author:** [@pescobar](https://discuss.elastic.co/u/pescobar)\ **Replies:** 2\ **Last updated:** [May 24, 2015, 10:00am UTC](https://discuss.elastic.co/t/how-to-parse-multiple-date-fields/1208 "2015-05-24T10:00:03Z")
Hi, I am learning logstash to insert my logfiles in elasticsearch. By now I am testing this config file: input { file { type =\> "accounting" path =\> \["/root/logstash-tests/mini-test/accounting.txt"\] sta… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1262) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1264)