# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1226 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1227 --- ## [Embedded Elasticsearch in Logstash 2.0 still available?](https://discuss.elastic.co/t/embedded-elasticsearch-in-logstash-2-0-still-available/33356)
**Author:** [@hwe](https://discuss.elastic.co/u/hwe)\ **Replies:** 2\ **Last updated:** [October 30, 2015, 10:42am UTC](https://discuss.elastic.co/t/embedded-elasticsearch-in-logstash-2-0-still-available/33356 "2015-10-30T10:42:47Z")
Is the embedded Elasticsearch in Logstash 2.0 still available? --- ## [Logstash Subqueries](https://discuss.elastic.co/t/logstash-subqueries/33327)
**Author:** [@DefensiveDepth](https://discuss.elastic.co/u/DefensiveDepth)\ **Replies:** 2\ **Last updated:** [October 30, 2015, 10:14am UTC](https://discuss.elastic.co/t/logstash-subqueries/33327 "2015-10-30T10:14:44Z")
I am working on a project that will be doing the following on a recurring basis: 1) Query an API for certain $IDs & associated metadata 2) For each $ID, hit 5 other APIs and Output each resultant dataset into ES Try… --- ## [Logstash Issue](https://discuss.elastic.co/t/logstash-issue/33259)
**Author:** [@Hoffin](https://discuss.elastic.co/u/Hoffin)\ **Replies:** 1\ **Last updated:** [October 30, 2015, 7:31am UTC](https://discuss.elastic.co/t/logstash-issue/33259 "2015-10-30T07:31:59Z")
In logstash 2.0 latest version for jdbc input plugin having the issue The error reported is: TypeError: cannot convert instance of class org.jruby.RubyObject to class java.lang.String Following is my configuration … --- ## [Timestamp format help "20151027 10:20:37"](https://discuss.elastic.co/t/timestamp-format-help-20151027-10-20-37/33318)
**Author:** [@ruchii](https://discuss.elastic.co/u/ruchii)\ **Replies:** 2\ **Last updated:** [October 29, 2015, 11:35pm UTC](https://discuss.elastic.co/t/timestamp-format-help-20151027-10-20-37/33318 "2015-10-29T23:35:00Z")
I am new to logstash would like to extract @timestamp data from "20151027 10:20:37 0 0 1 0 0 0 0 99 1 0 0 1352 1305 0 1225 2 0.61 0.53 0.52 3 0 24 22 3 16 0 5 0 0 0 0 0 0 0 0 0 0 0 0". Thanks in advance --- ## [A plugin had an unrecoverable error](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error/33176)
**Author:** [@Daniel\_Silva1](https://discuss.elastic.co/u/Daniel_Silva1)\ **Replies:** 3\ **Last updated:** [October 29, 2015, 4:00pm UTC](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error/33176 "2015-10-29T16:00:11Z")
Hi When I start logstash it crashes sometimes after one minute, sometimes take more time, but in resume crash. Did you have any suggestion? I already google it, but didn't find a good fix... Here is my input and t… --- ## [LSF Error: Skipping file (older than dead time of 24h0m0s):](https://discuss.elastic.co/t/lsf-error-skipping-file-older-than-dead-time-of-24h0m0s/32921)
**Author:** [@Fatema](https://discuss.elastic.co/u/Fatema)\ **Replies:** 10\ **Last updated:** [October 29, 2015, 2:42pm UTC](https://discuss.elastic.co/t/lsf-error-skipping-file-older-than-dead-time-of-24h0m0s/32921 "2015-10-29T14:42:07Z")
Hi, I have an ELK cluster. I have some files in a input dir which I need to forward to the ELK via LSF. My ELK and LSF are on the same machine. When I restarted ELK and LSF I am getting following msgs in LSF err logs: … --- ## [How to rotate logstash.log in ELK](https://discuss.elastic.co/t/how-to-rotate-logstash-log-in-elk/32925)
**Author:** [@kartikelastic](https://discuss.elastic.co/u/kartikelastic)\ **Replies:** 7\ **Last updated:** [October 29, 2015, 2:25pm UTC](https://discuss.elastic.co/t/how-to-rotate-logstash-log-in-elk/32925 "2015-10-29T14:25:05Z")
Hi team, How to rotate /var/log/logstash/logstash.log? It fills up the drive and then everything breaks. Simply doing a server# \> /var/log/logstash/logstash.log does not work What I have done is downloaded "curator" h… --- ## [Strange index dates](https://discuss.elastic.co/t/strange-index-dates/32714)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 2\ **Last updated:** [October 29, 2015, 12:59pm UTC](https://discuss.elastic.co/t/strange-index-dates/32714 "2015-10-29T12:59:36Z")
I am creating dated indices using logstash but there is a problem with the dates when ingesting records from Bro. I get dates from different years and different dates. I'm not sure why this is happening, when I look in… --- ## [Twitter Bounding box](https://discuss.elastic.co/t/twitter-bounding-box/33203)
**Author:** [@Gautam\_S\_Thakur](https://discuss.elastic.co/u/Gautam_S_Thakur)\ **Replies:** 2\ **Last updated:** [October 29, 2015, 11:25am UTC](https://discuss.elastic.co/t/twitter-bounding-box/33203 "2015-10-29T11:25:39Z")
I know Logstash can be used to collect tweets based on keywords. Is it possible to use Logstash to specify bounding box (south-west geo coordinates and north east geo coordinates) to collected geo-coded tweets? --- ## [Email output with mandrill](https://discuss.elastic.co/t/email-output-with-mandrill/33198)
**Author:** [@voipoclay](https://discuss.elastic.co/u/voipoclay)\ **Replies:** 1\ **Last updated:** [October 29, 2015, 2:33am UTC](https://discuss.elastic.co/t/email-output-with-mandrill/33198 "2015-10-29T02:33:41Z")
Hello, I am running logstash 2.0 and I configured email output for mandrill and emails dont send and debug info doesnt seem very helpful. My settings are basically output { email { … --- ## [File input and file output line count mismatch](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112)
**Author:** [@mihir\_ray](https://discuss.elastic.co/u/mihir_ray)\ **Replies:** 3\ **Last updated:** [October 28, 2015, 8:48pm UTC](https://discuss.elastic.co/t/file-input-and-file-output-line-count-mismatch/33112 "2015-10-28T20:48:59Z")
Input data: Input data is a static file containing 10 lines of accesslogs. If there is a way to share a file, please let me know, i can share the input file. Config: input { file { sincedb\_path =\> "/opt/analyti… --- ## [Match Logs Without a Common ID](https://discuss.elastic.co/t/match-logs-without-a-common-id/33174)
**Author:** [@chelseac313](https://discuss.elastic.co/u/chelseac313)\ **Replies:** 0\ **Last updated:** [October 28, 2015, 5:48pm UTC](https://discuss.elastic.co/t/match-logs-without-a-common-id/33174 "2015-10-28T17:48:42Z")
I am trying to match logs that do not have a common id between them. I'm trying to match a log that has the state ALARM with a log that is identical except the state is NORMAL. I am able to use grok to take apart my mess… --- ## [Convert @timestamp to string](https://discuss.elastic.co/t/convert-timestamp-to-string/33002)
**Author:** [@paltryeffort](https://discuss.elastic.co/u/paltryeffort)\ **Replies:** 2\ **Last updated:** [October 28, 2015, 3:04pm UTC](https://discuss.elastic.co/t/convert-timestamp-to-string/33002 "2015-10-28T15:04:29Z")
I'd like to upgrade logstash 1.4 to 1.5. Before my logstash server processes the actual message I copy the @timestamp to a new field in order to know when the server received the message: mutate { add\_field =\> … --- ## [How to create a filed based on a condition](https://discuss.elastic.co/t/how-to-create-a-filed-based-on-a-condition/33129)
**Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 5\ **Last updated:** [October 28, 2015, 2:49pm UTC](https://discuss.elastic.co/t/how-to-create-a-filed-based-on-a-condition/33129 "2015-10-28T14:49:36Z")
Hi Experts, I want to create a filed (pri\_caption) in logstash based on a condition i.e addfield pri\_caption { if priority between 1 to 3 then "low" else if priority between 4 to 7 then "medium" elase High } Please… --- ## [Using logstash-input-jms](https://discuss.elastic.co/t/using-logstash-input-jms/32103)
**Author:** [@cta100](https://discuss.elastic.co/u/cta100)\ **Replies:** 2\ **Last updated:** [October 28, 2015, 1:59pm UTC](https://discuss.elastic.co/t/using-logstash-input-jms/32103 "2015-10-28T13:59:18Z")
This is my first elk stack project and I need to process JMS input. Has anyone used the logstash-jms-input plugin and if so could I get some guidence on how to configure the config file? I have it built and installe… --- ## [Replace dots in field names for ES 2.0](https://discuss.elastic.co/t/replace-dots-in-field-names-for-es-2-0/32206)
**Author:** [@juerkan](https://discuss.elastic.co/u/juerkan)\ **Replies:** 3\ **Last updated:** [October 28, 2015, 1:36pm UTC](https://discuss.elastic.co/t/replace-dots-in-field-names-for-es-2-0/32206 "2015-10-28T13:36:11Z")
Dear logstash pro's We produce logs and we use logstash to bring it into elasticsearch. There hundrets of dynamic produced field names with a dot inside. (not nestet fields) And we have field names with no dots. for exam… --- ## [S3 input, execution expired \[SOLVED\]](https://discuss.elastic.co/t/s3-input-execution-expired-solved/32996)
**Author:** [@calle\_eman](https://discuss.elastic.co/u/calle_eman)\ **Replies:** 5\ **Last updated:** [October 28, 2015, 10:41am UTC](https://discuss.elastic.co/t/s3-input-execution-expired-solved/32996 "2015-10-28T10:41:41Z")
I have one logstash that's shipping files to s3, and one logstash that gets them from s3 and inserts them in elasticsearch. But the s3 input isn't working. Config: s3 { access\_key\_id =\> "XXXX" secret\_access\_key … --- ## [Indexing date from a field with two entries](https://discuss.elastic.co/t/indexing-date-from-a-field-with-two-entries/33064)
**Author:** [@aviv.ratzon](https://discuss.elastic.co/u/aviv.ratzon)\ **Replies:** 2\ **Last updated:** [October 28, 2015, 8:58am UTC](https://discuss.elastic.co/t/indexing-date-from-a-field-with-two-entries/33064 "2015-10-28T08:58:00Z")
Hello, I am parsing a log file and got to a point where I have this field: "Date":\["27/10/2015","10:09:01"\] I can't seem to preform a date filter on it in any way, I've tried different configurations and nothing. I al… --- ## [Logstash forwarding events only on log file update](https://discuss.elastic.co/t/logstash-forwarding-events-only-on-log-file-update/33050)
**Author:** [@nivetha](https://discuss.elastic.co/u/nivetha)\ **Replies:** 6\ **Last updated:** [October 28, 2015, 7:15am UTC](https://discuss.elastic.co/t/logstash-forwarding-events-only-on-log-file-update/33050 "2015-10-28T07:15:14Z")
Hi I am new to ELK. I am using logstash,Redis and Elasticsearch.When I push Events from logstash Shipper to Redis and then to Logstash Indexer and Query for events in Elasticsearch it shows Events lesser than the actual… --- ## [Buffer between logstash and elasticsearch](https://discuss.elastic.co/t/buffer-between-logstash-and-elasticsearch/33124)
**Author:** [@camarasingham](https://discuss.elastic.co/u/camarasingham)\ **Replies:** 1\ **Last updated:** [October 28, 2015, 6:40am UTC](https://discuss.elastic.co/t/buffer-between-logstash-and-elasticsearch/33124 "2015-10-28T06:40:43Z")
Hi All, We are using logstash tcp/udp json input plugin and the elasticsearch output plugin. I was wondering if there was a buffer of sorts between the two, so that, if the elasticsearch instance is restarted, message… --- ## [Send bulk inserts to dedicated elasticsearch client nodes?](https://discuss.elastic.co/t/send-bulk-inserts-to-dedicated-elasticsearch-client-nodes/33106)
**Author:** [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Replies:** 3\ **Last updated:** [October 28, 2015, 5:34am UTC](https://discuss.elastic.co/t/send-bulk-inserts-to-dedicated-elasticsearch-client-nodes/33106 "2015-10-28T05:34:38Z")
Currently my logstash instances are pointing directly to the elasticsearch data nodes. Are there much gain to deploy dedicated elasticsearch client nodes to handle the inserts? I imagine there is some benefit but dont… --- ## [How to parse a hostname](https://discuss.elastic.co/t/how-to-parse-a-hostname/33038)
**Author:** [@Mark\_Tellier](https://discuss.elastic.co/u/Mark_Tellier)\ **Replies:** 2\ **Last updated:** [October 28, 2015, 1:33am UTC](https://discuss.elastic.co/t/how-to-parse-a-hostname/33038 "2015-10-28T01:33:35Z")
I am new to Logstash and am having difficulty with a filter, I would like to replace a fully qualified host name with the short host name. The possible host name values are inconsistent: esx1.acme.com esx2.acme.com esx… --- ## [Apache log to logstash and consume content](https://discuss.elastic.co/t/apache-log-to-logstash-and-consume-content/33110)
**Author:** [@steven\_vogogo](https://discuss.elastic.co/u/steven_vogogo)\ **Replies:** 1\ **Last updated:** [October 27, 2015, 9:18pm UTC](https://discuss.elastic.co/t/apache-log-to-logstash-and-consume-content/33110 "2015-10-27T21:18:04Z")
Hello, We are using logstash to parse sensitive data from our logs and as a result we need the original logfile (that contains the sensitive data) to not retain any content. I have tried using a mkfifo and have logst… --- ## [Using the Curl Command](https://discuss.elastic.co/t/using-the-curl-command/33111)
**Author:** [@Didier](https://discuss.elastic.co/u/Didier)\ **Replies:** 1\ **Last updated:** [October 27, 2015, 9:17pm UTC](https://discuss.elastic.co/t/using-the-curl-command/33111 "2015-10-27T21:17:48Z")
Hello, In order to validate that my log entries are being parsed correctly, I am running the following command: curl http://localhost:9200/\_search? Whenever a new entry is added, I see the "Hit" increment as expec… --- ## [How logstash forwarder works?](https://discuss.elastic.co/t/how-logstash-forwarder-works/33062)
**Author:** [@sampasei](https://discuss.elastic.co/u/sampasei)\ **Replies:** 1\ **Last updated:** [October 27, 2015, 6:58pm UTC](https://discuss.elastic.co/t/how-logstash-forwarder-works/33062 "2015-10-27T18:58:08Z")
Hi all, i have a question about logstash-forwarder. I'm using a log program able to write txt files that are zipped and created again every hour: consider this case 8:59 my program is writing a file log.txt 9.00 log.tx… --- ## [Remove extra spaces between log messages using mutate & gsub](https://discuss.elastic.co/t/remove-extra-spaces-between-log-messages-using-mutate-gsub/33031)
**Author:** [@richinkansal](https://discuss.elastic.co/u/richinkansal)\ **Replies:** 2\ **Last updated:** [October 27, 2015, 5:45pm UTC](https://discuss.elastic.co/t/remove-extra-spaces-between-log-messages-using-mutate-gsub/33031 "2015-10-27T17:45:22Z")
Log file Example: INFO | jvm 1 | 2015/10/19 16:43:35 | logs logs logs text text text INFO | jvm 1 | 2015/10/19 16:43:35 | logs logs logs text text text query to execute(in millis) : 28 INFO | jvm 1 | 20… --- ## [Logstash hangs with OutOfMemoryError](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005)
**Author:** [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Replies:** 6\ **Last updated:** [October 27, 2015, 5:41pm UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005 "2015-10-27T17:41:09Z")
Today i noticed that 5/8 logstash indexers were "hung" and i see the below error in /var/log/logstash/logstash.err Running logstash 1.5.3 with Transport Node for elasticsearch output Running logstash with LS\_HEAP\_SIZE… --- ## [Getting \_jsonparsefailure in logstash](https://discuss.elastic.co/t/getting--jsonparsefailure-in-logstash/33078)
**Author:** [@philippconcur](https://discuss.elastic.co/u/philippconcur)\ **Replies:** 2\ **Last updated:** [October 27, 2015, 5:10pm UTC](https://discuss.elastic.co/t/getting--jsonparsefailure-in-logstash/33078 "2015-10-27T17:10:21Z")
Hi I am trying to parse a json file. I have tried troubleshooting with suggestions from stackoverflow (links at bottom)but none have worked for me. I am hoping someone has some insight on probably a silly mistake I am m… --- ## [Upgrade Logstash : mutate filter is not working well](https://discuss.elastic.co/t/upgrade-logstash-mutate-filter-is-not-working-well/28090)
**Author:** [@Hornov](https://discuss.elastic.co/u/Hornov)\ **Replies:** 4\ **Last updated:** [October 27, 2015, 2:53pm UTC](https://discuss.elastic.co/t/upgrade-logstash-mutate-filter-is-not-working-well/28090 "2015-10-27T14:53:34Z")
Following the logstash migrating from 1.4.2 version to 1.5.3 version when I do an uppercase in the mutate filter on a field ending with numeric value the field becomes nil. Yet, before that worked. How can I fix it ? M… --- ## [Match against blacklist](https://discuss.elastic.co/t/match-against-blacklist/32995)
**Author:** [@zappe](https://discuss.elastic.co/u/zappe)\ **Replies:** 7\ **Last updated:** [October 27, 2015, 2:29pm UTC](https://discuss.elastic.co/t/match-against-blacklist/32995 "2015-10-27T14:29:57Z")
I have DNS logs which I'd like to match against a blacklist. I've imported the dns logs via a grok filter. Can anyone point me in the right direction to match my parsed URL field with a blacklist in some way? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1225) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1227)