# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1173 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1174 --- ## [JSON log format](https://discuss.elastic.co/t/json-log-format/45567)
**Author:** [@Mykolaichenko](https://discuss.elastic.co/u/Mykolaichenko)\ **Replies:** 2\ **Last updated:** [March 30, 2016, 4:02pm UTC](https://discuss.elastic.co/t/json-log-format/45567 "2016-03-30T16:02:48Z")
Hi! I have filebeat, logstash and json log format in application. It looks like this: {"timestamp":1459152943492,"date":"Mar 28, 2016 10:15:43 AM","hostname":"hostname","username":"tomcat","level":"INFO","thread":"Si… --- ## [Issue in gem Installation on Logstash 2.2.1](https://discuss.elastic.co/t/issue-in-gem-installation-on-logstash-2-2-1/45787)
**Author:** [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)\ **Replies:** 0\ **Last updated:** [March 30, 2016, 12:34pm UTC](https://discuss.elastic.co/t/issue-in-gem-installation-on-logstash-2-2-1/45787 "2016-03-30T12:34:14Z")
Hi, I am trying to install logstash-output-riemann 2.0 in my Logstash 2.2.1. I am getting the below error, -sh-4.2$ cd /software/logstash-2.2.1/bin -sh-4.2$ ./plugin install /gems/logstash-output-riemann-0.2.0.gem Va… --- ## [Pull data from Twitter(particular hashtag)](https://discuss.elastic.co/t/pull-data-from-twitter-particular-hashtag/27271)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 2\ **Last updated:** [March 30, 2016, 12:07pm UTC](https://discuss.elastic.co/t/pull-data-from-twitter-particular-hashtag/27271 "2016-03-30T12:07:51Z")
Hi All, I am using logstash to pull data from twitter to elasticsearch. Below is my config file. input { twitter { add\_field =\> consumer\_key =\> "xxxxxxxxxxxxxxxxx" consumer\_secret =\> "xxxxxxxxxxxxxxxxxx" … --- ## [Change Status code to the name of the status?](https://discuss.elastic.co/t/change-status-code-to-the-name-of-the-status/45781)
**Author:** [@jervdv](https://discuss.elastic.co/u/jervdv)\ **Replies:** 1\ **Last updated:** [March 30, 2016, 11:21am UTC](https://discuss.elastic.co/t/change-status-code-to-the-name-of-the-status/45781 "2016-03-30T11:21:13Z")
Is it posibble to change the status code of a server (ex. 200) to the name of the status code (ex. OK) in your elasticsearch data when you send it from logstash? Just like it is possible with geoip information. --- ## [File input not picking up copied or moved files](https://discuss.elastic.co/t/file-input-not-picking-up-copied-or-moved-files/27245)
**Author:** [@karolpivo](https://discuss.elastic.co/u/karolpivo)\ **Replies:** 3\ **Last updated:** [March 30, 2016, 11:20am UTC](https://discuss.elastic.co/t/file-input-not-picking-up-copied-or-moved-files/27245 "2016-03-30T11:20:25Z")
Hi, I'm successfully using logstash to parse json formated events and send them to elasticsearch. Each event is created in a seprate file. One event per file with .json extension. Logstash is correctly picking up a … --- ## [Detect Elastic Search Server is up](https://discuss.elastic.co/t/detect-elastic-search-server-is-up/45767)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 3\ **Last updated:** [March 30, 2016, 9:18am UTC](https://discuss.elastic.co/t/detect-elastic-search-server-is-up/45767 "2016-03-30T09:18:28Z")
Is there a way to make logstash not to parse data until it detects that the Elasticsearch server is up and ready --- ## [Identify Files Already Parsed by Logstash](https://discuss.elastic.co/t/identify-files-already-parsed-by-logstash/45766)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 1\ **Last updated:** [March 30, 2016, 8:58am UTC](https://discuss.elastic.co/t/identify-files-already-parsed-by-logstash/45766 "2016-03-30T08:58:58Z")
How do I know if a file is already parsed by logstash? I want to create a setup such that once a file is copied into a folder I will parsed it and move it to another location for archiving --- ## [Logstash-forwarder not connecting with the server](https://discuss.elastic.co/t/logstash-forwarder-not-connecting-with-the-server/45688)
**Author:** [@Roshan\_r](https://discuss.elastic.co/u/Roshan_r)\ **Replies:** 9\ **Last updated:** [March 30, 2016, 7:57am UTC](https://discuss.elastic.co/t/logstash-forwarder-not-connecting-with-the-server/45688 "2016-03-30T07:57:23Z")
Hi All, I am trying to use the logstash and logstash-forwarder fir the first time. My logstash-forwarder and logstash are installed on 2 separate VMs. Consider forwarder on server A and Logstash on Server B. I generat… --- ## [Number of workers vs Memory consumption](https://discuss.elastic.co/t/number-of-workers-vs-memory-consumption/45681)
**Author:** [@billy6](https://discuss.elastic.co/u/billy6)\ **Replies:** 2\ **Last updated:** [March 30, 2016, 7:44am UTC](https://discuss.elastic.co/t/number-of-workers-vs-memory-consumption/45681 "2016-03-30T07:44:04Z")
Hello, I would like to know if there is a direct dependency between the number of workers I'm using in logstash and the memory consumption. I have changed from 1 worker to 50 workers with a little change in memory consu… --- ## [Huge Time Delay between logstash and elasticsearch](https://discuss.elastic.co/t/huge-time-delay-between-logstash-and-elasticsearch/45560)
**Author:** [@ApoorvaLad](https://discuss.elastic.co/u/ApoorvaLad)\ **Replies:** 4\ **Last updated:** [March 30, 2016, 6:52am UTC](https://discuss.elastic.co/t/huge-time-delay-between-logstash-and-elasticsearch/45560 "2016-03-30T06:52:23Z")
My current pipeline is: Rsyslog -\> Kafka -\> Logstash -\> ES (5 nodes) I see a huge time delay (Arnd 10hrs) between the logs processed by Kafka and the logs in my ES cluster. the input and output plugin in logstash look… --- ## [IMAP-plugin logstash](https://discuss.elastic.co/t/imap-plugin-logstash/43240)
**Author:** [@petar.kozic](https://discuss.elastic.co/u/petar.kozic)\ **Replies:** 5\ **Last updated:** [March 30, 2016, 6:40am UTC](https://discuss.elastic.co/t/imap-plugin-logstash/43240 "2016-03-30T06:40:05Z")
Can it imap-plugin to receive mail from multiple folder, I think on Inbox and subfolder or only from Inbox ? --- ## [How do I know/or make sure that elastic search does not add same log file twice?](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736)
**Author:** [@Dani](https://discuss.elastic.co/u/Dani)\ **Replies:** 2\ **Last updated:** [March 30, 2016, 5:37am UTC](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736 "2016-03-30T05:37:52Z")
When using a file input, with elasticsearch output how do I make sure that each of the lines in the log files I generate, and have logstash process only get added once? If I add another output, like S3 some time late… --- ## [Write a plugin to enable logstash password protection](https://discuss.elastic.co/t/write-a-plugin-to-enable-logstash-password-protection/45609)
**Author:** [@M\_Situation](https://discuss.elastic.co/u/M_Situation)\ **Replies:** 4\ **Last updated:** [March 29, 2016, 9:23pm UTC](https://discuss.elastic.co/t/write-a-plugin-to-enable-logstash-password-protection/45609 "2016-03-29T21:23:27Z")
Hi everyone, I'm actually playing around elastic technologies and I want to write a logstash plugin to enable password protection. Does something like this already exist? If not, anyone can give me skeleton or sample c… --- ## [Logstash duplicate line / send log with elasticsearch:9200 no response](https://discuss.elastic.co/t/logstash-duplicate-line-send-log-with-elasticsearch-9200-no-response/45612)
**Author:** [@pittoch](https://discuss.elastic.co/u/pittoch)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 9:01pm UTC](https://discuss.elastic.co/t/logstash-duplicate-line-send-log-with-elasticsearch-9200-no-response/45612 "2016-03-29T21:01:14Z")
Hi, I use Logstash with CSV files sources. What happens if logstash can not contact ElasticSearch on 9200 port. They sent the logs will be reissued , or the logs are lost ? Dy ou know a concept of primary key to rej… --- ## [Shipping data continously](https://discuss.elastic.co/t/shipping-data-continously/45729)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 8:44pm UTC](https://discuss.elastic.co/t/shipping-data-continously/45729 "2016-03-29T20:44:01Z")
Hi, Currently I have it set up so that a logstash agent ships data to a different server through redis. I have created a dashboard out of the data that I receive. Now I want it to automatically ship the data so that i do… --- ## [File input fails to respect shutdown](https://discuss.elastic.co/t/file-input-fails-to-respect-shutdown/44730)
**Author:** [@sarthur](https://discuss.elastic.co/u/sarthur)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 7:16pm UTC](https://discuss.elastic.co/t/file-input-fails-to-respect-shutdown/44730 "2016-03-29T19:16:54Z")
I just file a new bug against the file input plugin bundled in 2.2.2. Basically when I try to do a graceful shutdown the file input says its closed but it really isn't and events from the tailer keep arriving to the inp… --- ## [Grokparsefailure, but works in grok debugger](https://discuss.elastic.co/t/grokparsefailure-but-works-in-grok-debugger/45613)
**Author:** [@mikeface](https://discuss.elastic.co/u/mikeface)\ **Replies:** 2\ **Last updated:** [March 29, 2016, 3:30pm UTC](https://discuss.elastic.co/t/grokparsefailure-but-works-in-grok-debugger/45613 "2016-03-29T15:30:17Z")
Hey guys, need some help with grok, kind of at a loss. I've used Logstash quite a bit in the past, never run into this issue. On Logstash v2.2.2, I can't seem to get grok to parse some logs from Nginx. They have some … --- ## [Syntax for adding tags from couchdb doc array attribute](https://discuss.elastic.co/t/syntax-for-adding-tags-from-couchdb-doc-array-attribute/45619)
**Author:** [@pagameba](https://discuss.elastic.co/u/pagameba)\ **Replies:** 2\ **Last updated:** [March 29, 2016, 2:09pm UTC](https://discuss.elastic.co/t/syntax-for-adding-tags-from-couchdb-doc-array-attribute/45619 "2016-03-29T14:09:08Z")
Hi, I'm trying to figure out the logstash filter syntax for mutate to add tags (using add\_tag) to an event coming from a couchdb\_changes feed where the tags need to come from a document attribute that is, itself, an arra… --- ## [Missing fields while Indexing Tomcat access logs data in elasticsearch](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580)
**Author:** [@aviral\_srivastava](https://discuss.elastic.co/u/aviral_srivastava)\ **Replies:** 6\ **Last updated:** [March 29, 2016, 1:54pm UTC](https://discuss.elastic.co/t/missing-fields-while-indexing-tomcat-access-logs-data-in-elasticsearch/45580 "2016-03-29T13:54:37Z")
I have localhost\_access\_log.2016-03-23.txt file which I am trying to index in elasticsearch via logstash. Above file looks as follows:- 0:0:0:0:0:0:0:1 - - \[23/Mar/2016:12:20:46 +0530\] "GET / HTTP/1.1" 200 11444 0:0:0… --- ## [Resolve Field from External File (CSV?)](https://discuss.elastic.co/t/resolve-field-from-external-file-csv/45679)
**Author:** [@Chris\_Schubert](https://discuss.elastic.co/u/Chris_Schubert)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 11:53am UTC](https://discuss.elastic.co/t/resolve-field-from-external-file-csv/45679 "2016-03-29T11:53:19Z")
Is there a way to resolve an incoming field from an external source (i.e. a csv file?). I'm working with auditd and I want the syscall field to be resolved to the actual syscall name rather than the integer for easier re… --- ## [How to get alerts for logstash failures of plugins / input / filters?](https://discuss.elastic.co/t/how-to-get-alerts-for-logstash-failures-of-plugins-input-filters/27087)
**Author:** [@KrishnaPG](https://discuss.elastic.co/u/KrishnaPG)\ **Replies:** 6\ **Last updated:** [March 29, 2016, 11:47am UTC](https://discuss.elastic.co/t/how-to-get-alerts-for-logstash-failures-of-plugins-input-filters/27087 "2016-03-29T11:47:28Z")
Inside logstash, what is the official mechanism to know about any critical errors that happen with plugins / inputs / filters during the production run? For example, I am using http\_poller input and elastic output. Say… --- ## [Unrecoverable error in input { elasticsearch {} } when using "fields" in query](https://discuss.elastic.co/t/unrecoverable-error-in-input-elasticsearch-when-using-fields-in-query/39020)
**Author:** [@TheCheapestPixels](https://discuss.elastic.co/u/TheCheapestPixels)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 9:39am UTC](https://discuss.elastic.co/t/unrecoverable-error-in-input-elasticsearch-when-using-fields-in-query/39020 "2016-03-29T09:39:27Z")
This is my rather trivial logstash config: input { elasticsearch { hosts =\> "localhost" query =\> '{"query": {"match": {"tags": "pipeline-v0.6"}}}' # query =\> '{"query": {"match": {"tags": "p… --- ## [How to config Logstash to set custom type](https://discuss.elastic.co/t/how-to-config-logstash-to-set-custom-type/45616)
**Author:** [@tuananhtd](https://discuss.elastic.co/u/tuananhtd)\ **Replies:** 4\ **Last updated:** [March 29, 2016, 6:18am UTC](https://discuss.elastic.co/t/how-to-config-logstash-to-set-custom-type/45616 "2016-03-29T06:18:25Z")
Hi guys! I know that this maybe a duplicated topic but I guess that the old topic is just outdated since I couldn't make it works. In my project I use FileBeat to ship JSON logs to Logstash. I want to have different ty… --- ## [Install Plugin with Proxy and Authentication](https://discuss.elastic.co/t/install-plugin-with-proxy-and-authentication/45552)
**Author:** [@s802645](https://discuss.elastic.co/u/s802645)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 5:54am UTC](https://discuss.elastic.co/t/install-plugin-with-proxy-and-authentication/45552 "2016-03-29T05:54:12Z")
Hi, Trying to install plugin # export https\_proxy=https://myname:passw@rd@10.10.10.11:80 # export http\_proxy=http://myname:passw@rd@10.10.10.11:80 # bin/plugin install logstash-output-syslog Validating logstash-outp… --- ## [Using logstash-plugin](https://discuss.elastic.co/t/using-logstash-plugin/45572)
**Author:** [@Inbeo\_Beo](https://discuss.elastic.co/u/Inbeo_Beo)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 5:53am UTC](https://discuss.elastic.co/t/using-logstash-plugin/45572 "2016-03-29T05:53:06Z")
Hi all Follow this site: https://github.com/logstash-plugins?query=example Could you help me to use this plugin in this site in my logstash? i really don't know how can i get and install them Thanks so much --- ## [Join data in two different logs](https://discuss.elastic.co/t/join-data-in-two-different-logs/45608)
**Author:** [@vilas](https://discuss.elastic.co/u/vilas)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 5:48am UTC](https://discuss.elastic.co/t/join-data-in-two-different-logs/45608 "2016-03-29T05:48:24Z")
Hi, I have two logs files on remote server. Both those log files have a key that I can join the two log files on. I want this data to be in elasticsearch as a single document. Can we do this somehow? Thanks. --- ## [How to convert input data to parquet using logstash.?](https://discuss.elastic.co/t/how-to-convert-input-data-to-parquet-using-logstash/45628)
**Author:** [@sdaruna](https://discuss.elastic.co/u/sdaruna)\ **Replies:** 3\ **Last updated:** [March 29, 2016, 5:31am UTC](https://discuss.elastic.co/t/how-to-convert-input-data-to-parquet-using-logstash/45628 "2016-03-29T05:31:44Z")
Hi, I have json/xml data as input and i need to move the data to parquet using logstash. Is there a way to output parquet data from logstash.? --- ## [CSV plugin](https://discuss.elastic.co/t/csv-plugin/45557)
**Author:** [@priyakumar](https://discuss.elastic.co/u/priyakumar)\ **Replies:** 2\ **Last updated:** [March 29, 2016, 4:21am UTC](https://discuss.elastic.co/t/csv-plugin/45557 "2016-03-29T04:21:04Z")
Is there any way that input file path and unique field value can be read from user using any web UI like http or something like we shall write ruby filter to get input file path option from user's console. --- ## [LogStash not sending data to ElasticSearch](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/44791)
**Author:** [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Replies:** 20\ **Last updated:** [March 29, 2016, 4:20am UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elasticsearch/44791 "2016-03-29T04:20:49Z")
This is my logstash conf - input { file { path =\> "C:\\LogStash Logs\*" } } output { elasticsearch { hosts =\> \["localhost:9200"\] } stdout { codec =\> rubydebug } } ES is working fine. What ever fi… --- ## [Logstash's OutOfMemoryError](https://discuss.elastic.co/t/logstashs-outofmemoryerror/45319)
**Author:** [@lester015](https://discuss.elastic.co/u/lester015)\ **Replies:** 1\ **Last updated:** [March 29, 2016, 3:07am UTC](https://discuss.elastic.co/t/logstashs-outofmemoryerror/45319 "2016-03-29T03:07:03Z")
hi everyone, after I execute logstash orders the console show error as bellows: ""Ruby-0-Thread-6: /usr/pds/logstash-2.2.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.1-java/lib/logstash/outputs… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1172) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1174)