# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1125 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1126 --- ## [ELK Scalability Query](https://discuss.elastic.co/t/elk-scalability-query/55649)
**Author:** [@Gunjan\_Gupta](https://discuss.elastic.co/u/Gunjan_Gupta)\ **Replies:** 7\ **Last updated:** [July 21, 2016, 5:30am UTC](https://discuss.elastic.co/t/elk-scalability-query/55649 "2016-07-21T05:30:02Z")
Scenario: 1.ELK is already placed and currently getting three different log files as input from three different server . 2.Now as my system is in place already ,if I want to add log file from fourth server on the fly… --- ## [Replace the existing index with new data](https://discuss.elastic.co/t/replace-the-existing-index-with-new-data/55908)
**Author:** [@AnantPatil](https://discuss.elastic.co/u/AnantPatil)\ **Replies:** 3\ **Last updated:** [July 20, 2016, 10:12pm UTC](https://discuss.elastic.co/t/replace-the-existing-index-with-new-data/55908 "2016-07-20T22:12:23Z")
Hi, I have the index name called "pattwo" ,wanted to update the old data with new data for this index.without creating the new index. My input file is having two records which is already indexed and the records in th… --- ## [Wrong doc filter cvs convert](https://discuss.elastic.co/t/wrong-doc-filter-cvs-convert/56018)
**Author:** [@AlexisAnzieu](https://discuss.elastic.co/u/AlexisAnzieu)\ **Replies:** 1\ **Last updated:** [July 20, 2016, 10:07pm UTC](https://discuss.elastic.co/t/wrong-doc-filter-cvs-convert/56018 "2016-07-20T22:07:31Z")
Hi, There is a problem with the doc example : https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-convert filter { csv { convert =\> { "column1" =\> "integer", "colum… --- ## [Drop filter not working](https://discuss.elastic.co/t/drop-filter-not-working/55940)
**Author:** [@nishant\_goel](https://discuss.elastic.co/u/nishant_goel)\ **Replies:** 3\ **Last updated:** [July 20, 2016, 9:53pm UTC](https://discuss.elastic.co/t/drop-filter-not-working/55940 "2016-07-20T21:53:05Z")
I have multiple log messages in a file which I am processing using logstash filter plugins. Then, the filtered logs are getting sent to elasticsearch. There is one field called addID in a log message. I want to drop al… --- ## [Grok filter for mysql-slow-logs produces grokparsefailure but passes tests](https://discuss.elastic.co/t/grok-filter-for-mysql-slow-logs-produces-grokparsefailure-but-passes-tests/55799)
**Author:** [@Jon\_Slusher](https://discuss.elastic.co/u/Jon_Slusher)\ **Replies:** 4\ **Last updated:** [July 20, 2016, 9:37pm UTC](https://discuss.elastic.co/t/grok-filter-for-mysql-slow-logs-produces-grokparsefailure-but-passes-tests/55799 "2016-07-20T21:37:27Z")
I created a grok filter using the excellent grok constructor online. I pulled an example message string from a parsed mysql-slow-log and tested it using the grok pattern tester and it passed that test. When I apply it wi… --- ## [Aggregation Query possible input ES plugin](https://discuss.elastic.co/t/aggregation-query-possible-input-es-plugin/28159)
**Author:** [@Jungmi\_Yoo](https://discuss.elastic.co/u/Jungmi_Yoo)\ **Replies:** 3\ **Last updated:** [July 20, 2016, 9:07pm UTC](https://discuss.elastic.co/t/aggregation-query-possible-input-es-plugin/28159 "2016-07-20T21:07:22Z")
I really want to know that aggregation is possible in input / output plugin ? input { # Read all documents from Elasticsearch matching the given query elasticsearch { hosts =\> \["192.168.1.5"\] index =\> "m… --- ## [Grok filter for mysql queries?](https://discuss.elastic.co/t/grok-filter-for-mysql-queries/56027)
**Author:** [@Jon\_Slusher](https://discuss.elastic.co/u/Jon_Slusher)\ **Replies:** 0\ **Last updated:** [July 20, 2016, 8:40pm UTC](https://discuss.elastic.co/t/grok-filter-for-mysql-queries/56027 "2016-07-20T20:40:03Z")
I have mysql slowlogs in logstash and I'm wondering if anyone has had any luck parsing the query section to try and categorize statements. I'm stuck right now trying to split up the query by words in capital letters. I'm… --- ## [Custom plugin update problem](https://discuss.elastic.co/t/custom-plugin-update-problem/56013)
**Author:** [@Felipe\_Santos](https://discuss.elastic.co/u/Felipe_Santos)\ **Replies:** 0\ **Last updated:** [July 20, 2016, 6:45pm UTC](https://discuss.elastic.co/t/custom-plugin-update-problem/56013 "2016-07-20T18:45:51Z")
I am trying to upgrade my custom plugin to logstash 5. I changed the gemspec of my plugin from "logstash-core", "\>= 2.0.0", "\< 3.0.0" to "logstash-core-plugin-api", "~\> 2.0" And when I try to run bundle exec rspec I've g… --- ## [Delay in getting the data from postgresql](https://discuss.elastic.co/t/delay-in-getting-the-data-from-postgresql/56000)
**Author:** [@ggp](https://discuss.elastic.co/u/ggp)\ **Replies:** 2\ **Last updated:** [July 20, 2016, 5:03pm UTC](https://discuss.elastic.co/t/delay-in-getting-the-data-from-postgresql/56000 "2016-07-20T17:03:29Z")
I am seeing delay in getting the data from postgresql, here is the configuration i have input { jdbc { # Postgres jdbc connection string to our database, mydb jdbc\_connection\_string =\> "jdbc:postgresql://\<\>:5432/svc… --- ## [Grokparsefailure after some filter in my logstash configuration](https://discuss.elastic.co/t/grokparsefailure-after-some-filter-in-my-logstash-configuration/55942)
**Author:** [@orsa](https://discuss.elastic.co/u/orsa)\ **Replies:** 1\ **Last updated:** [July 20, 2016, 4:55pm UTC](https://discuss.elastic.co/t/grokparsefailure-after-some-filter-in-my-logstash-configuration/55942 "2016-07-20T16:55:00Z")
i have some log from my network devices. log send to Logstash after some filter with log, i have "grokparsefailure". logstash conf grok { match =\> { "message" =\> "^\[a-z0-9,\]\* %{GREEDYDATA:message}" } … --- ## [Slow throughput with beaver](https://discuss.elastic.co/t/slow-throughput-with-beaver/55895)
**Author:** [@smilin\_stan](https://discuss.elastic.co/u/smilin_stan)\ **Replies:** 2\ **Last updated:** [July 20, 2016, 4:29pm UTC](https://discuss.elastic.co/t/slow-throughput-with-beaver/55895 "2016-07-20T16:29:58Z")
I am using beaver to monitor a single log file and dump the contents as JSON to redis, from where logstash picks it up via the redis input plugin. This is working but the throughput is poor. After performing some analy… --- ## [Logstash configuration api](https://discuss.elastic.co/t/logstash-configuration-api/53946)
**Author:** [@prabhakar349](https://discuss.elastic.co/u/prabhakar349)\ **Replies:** 2\ **Last updated:** [July 20, 2016, 3:46pm UTC](https://discuss.elastic.co/t/logstash-configuration-api/53946 "2016-07-20T15:46:28Z")
Hi , Is there any plan to provide rest API for configuring logstash ? --- ## [How do I create a filter for a custom app log with different lines?](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961)
**Author:** [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Replies:** 9\ **Last updated:** [July 20, 2016, 2:53pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961 "2016-07-20T14:53:54Z")
Hello I am knew to Logstash/Elasticsearch.. I have a log file with multiple lines within it. What's the best way to create a filter for a log similar to the example below? 2016-07-12T02:04:32.998+0100: Total time f… --- ## [Logstash not pushing data which are adding in a log file](https://discuss.elastic.co/t/logstash-not-pushing-data-which-are-adding-in-a-log-file/55982)
**Author:** [@Gaurav\_Harsola](https://discuss.elastic.co/u/Gaurav_Harsola)\ **Replies:** 12\ **Last updated:** [July 20, 2016, 2:34pm UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-which-are-adding-in-a-log-file/55982 "2016-07-20T14:34:51Z")
Hi, My logstash conf file is input { file { type =\> "name1" path =\> "/var/log/alerts.json" codec =\> "json" } } filter { } output { #stdout { codec =\> rubydebug } elasticsearch { host… --- ## [How to parse different format logs from same directory](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772)
**Author:** [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\ **Replies:** 3\ **Last updated:** [July 20, 2016, 2:33pm UTC](https://discuss.elastic.co/t/how-to-parse-different-format-logs-from-same-directory/54772 "2016-07-20T14:33:35Z")
I have multiple log files in the same directory, that I want to run through Logstash, every file is having different format than other. My log files can be named randomly by different network providers (I dont have cont… --- ## [Bluecoat proxySG log parsing](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634)
**Author:** [@jpry](https://discuss.elastic.co/u/jpry)\ **Replies:** 8\ **Last updated:** [July 20, 2016, 1:59pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634 "2016-07-20T13:59:28Z")
Hi, I am facing a problem related to bluecoat proxySG logs; I have my bluecoat log files stored in a server running an instance of logstash, let's call it server 1. The input section of the configuration file is also … --- ## [Is it possible to update an old event already stored?](https://discuss.elastic.co/t/is-it-possible-to-update-an-old-event-already-stored/55972)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 3\ **Last updated:** [July 20, 2016, 12:36pm UTC](https://discuss.elastic.co/t/is-it-possible-to-update-an-old-event-already-stored/55972 "2016-07-20T12:36:55Z")
Hi, I would like to visualize if a server is in status ok, warning, critical. So I have events which have a field which contains this information. But how can I show only the latest probe / event? Is it possible to… --- ## [How to create sub indexes for an index in config file of logstash](https://discuss.elastic.co/t/how-to-create-sub-indexes-for-an-index-in-config-file-of-logstash/55881)
**Author:** [@pavana](https://discuss.elastic.co/u/pavana)\ **Replies:** 7\ **Last updated:** [July 20, 2016, 11:48am UTC](https://discuss.elastic.co/t/how-to-create-sub-indexes-for-an-index-in-config-file-of-logstash/55881 "2016-07-20T11:48:36Z")
I have QueueMessage and MessageID as two fields which i defined using csv as shown below. input { log4j { mode =\> "server" host =\> "0.0.0.0" port =\> 3456 type =\> "log4j" } } filter{ csv{ columns=\> … --- ## [Logstash script creation of one field with different alias names](https://discuss.elastic.co/t/logstash-script-creation-of-one-field-with-different-alias-names/55968)
**Author:** [@prasad.ram1431](https://discuss.elastic.co/u/prasad.ram1431)\ **Replies:** 1\ **Last updated:** [July 20, 2016, 11:38am UTC](https://discuss.elastic.co/t/logstash-script-creation-of-one-field-with-different-alias-names/55968 "2016-07-20T11:38:50Z")
Hi, Could you please tell me, if there is any way to declare one field with different alias names as below, field\_name: address alias\_name1: address1 alias\_name:2 address2 Kindly let me know if there is a way, then… --- ## [Parsing tags](https://discuss.elastic.co/t/parsing-tags/55660)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 6\ **Last updated:** [July 20, 2016, 11:32am UTC](https://discuss.elastic.co/t/parsing-tags/55660 "2016-07-20T11:32:08Z")
How can I parse this using grok filter "tags" =\> \[ \[0\] "test-category", \[1\] "beats\_input\_raw\_event" Tried this won't work match =\> { "\[tags\]\[0\]" =\> "%{WORD:category}" } Thanks --- ## [Logstash webhdfs output error](https://discuss.elastic.co/t/logstash-webhdfs-output-error/55832)
**Author:** [@lizhenmxcz](https://discuss.elastic.co/u/lizhenmxcz)\ **Replies:** 2\ **Last updated:** [July 20, 2016, 9:24am UTC](https://discuss.elastic.co/t/logstash-webhdfs-output-error/55832 "2016-07-20T09:24:07Z")
here is the config: input { redis { host =\> "127.0.0.1" port =\> "6379" key =\> "abc" data\_type =\> "list" codec =\> "plain" } } output { webhdfs { host =\> "192.168.xx.xxx" path =\> "/abc.log" … --- ## [CSV Filter help](https://discuss.elastic.co/t/csv-filter-help/55937)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 3\ **Last updated:** [July 20, 2016, 5:30am UTC](https://discuss.elastic.co/t/csv-filter-help/55937 "2016-07-20T05:30:42Z")
Hi There, I am using the following CSV filter after failing to use grok filter for a logfile containing multiple patterns: filter { csv { columns =\> \["@timestamp", "threadType", "requestType", "requestSta… --- ## [How to escape the double quotes and \\r of csv field from longstash](https://discuss.elastic.co/t/how-to-escape-the-double-quotes-and-r-of-csv-field-from-longstash/55884)
**Author:** [@AnantPatil](https://discuss.elastic.co/u/AnantPatil)\ **Replies:** 2\ **Last updated:** [July 19, 2016, 4:03pm UTC](https://discuss.elastic.co/t/how-to-escape-the-double-quotes-and-r-of-csv-field-from-longstash/55884 "2016-07-19T16:03:28Z")
My csv data is like this: S2\_CR\_Browse\_iPhone\_T22\_Click “Cruise&Stay" link from cruise menu,1393,101,46 Error in Logstash command: My Logstash Filter section: filter { if(\[message\]=~"sampler\_label" or \[message\]=~"… --- ## [Reached open files limit](https://discuss.elastic.co/t/reached-open-files-limit/43113)
**Author:** [@abraham](https://discuss.elastic.co/u/abraham)\ **Replies:** 4\ **Last updated:** [July 19, 2016, 3:55pm UTC](https://discuss.elastic.co/t/reached-open-files-limit/43113 "2016-07-19T15:55:58Z")
Hi, I am new to ELK stack. I have been working on it for a while and it was working fine and suddenly the logs stopped loading into kibana dashboard and when i checked the log files in the logstash log file i am getting … --- ## [Does the data input filter work on nested fields?](https://discuss.elastic.co/t/does-the-data-input-filter-work-on-nested-fields/55765)
**Author:** [@mormor1971](https://discuss.elastic.co/u/mormor1971)\ **Replies:** 2\ **Last updated:** [July 19, 2016, 3:25pm UTC](https://discuss.elastic.co/t/does-the-data-input-filter-work-on-nested-fields/55765 "2016-07-19T15:25:45Z")
Hello All, I'm trying to get logstash to parse the date from a nested field in a JSON blob and are having some issues doing so. Here is what the JSON blob looks like: { "count"27, "data":\[{ "hotspotIdentifier":… --- ## [Logstash pipeline getting blocked temporary](https://discuss.elastic.co/t/logstash-pipeline-getting-blocked-temporary/55829)
**Author:** [@Deb](https://discuss.elastic.co/u/Deb)\ **Replies:** 2\ **Last updated:** [July 19, 2016, 3:23pm UTC](https://discuss.elastic.co/t/logstash-pipeline-getting-blocked-temporary/55829 "2016-07-19T15:23:12Z")
I have a logstash deployment which looks like below:- Multiple Logstash-forwarder ----\> Single Logstash Collector ----\> Single Kafka Instance ----\> Single Logstash Indexer ----\> Elastic Search Cluster The logstash Co… --- ## [Logstash Log File is Filling Up Disk](https://discuss.elastic.co/t/logstash-log-file-is-filling-up-disk/55760)
**Author:** [@Rohit\_Singh](https://discuss.elastic.co/u/Rohit_Singh)\ **Replies:** 1\ **Last updated:** [July 19, 2016, 3:20pm UTC](https://discuss.elastic.co/t/logstash-log-file-is-filling-up-disk/55760 "2016-07-19T15:20:30Z")
Hi, we are using logstash, to forward the logs of docker container with the help of filebeat. However, after 4 days of running the node's disk becomes full which is running logstash container. On debugging found out that… --- ## [Kibana .raw fields and "message" field](https://discuss.elastic.co/t/kibana-raw-fields-and-message-field/757)
**Author:** [@alexolivan](https://discuss.elastic.co/u/alexolivan)\ **Replies:** 8\ **Last updated:** [July 19, 2016, 10:19am UTC](https://discuss.elastic.co/t/kibana-raw-fields-and-message-field/757 "2016-07-19T10:19:49Z")
I again fourum... I have optimized my filters... overall very good, but now I have Kibana problem with space separated fields. Reading around, I have stated I have "lost" those .raw fields... Now I know they were there… --- ## [Removing string from this form of time stamp](https://discuss.elastic.co/t/removing-string-from-this-form-of-time-stamp/55838)
**Author:** [@jihoon](https://discuss.elastic.co/u/jihoon)\ **Replies:** 1\ **Last updated:** [July 19, 2016, 10:02am UTC](https://discuss.elastic.co/t/removing-string-from-this-form-of-time-stamp/55838 "2016-07-19T10:02:20Z")
This process had the following timestamp. "2016-07-19 16:02:17" filter { date { match =\> \[ "message", "YYYY-MM-dd HH:mm:ss", "ISO8601" \] target =\> "abctime" } } But these have to do with the time … --- ## [Can I use grok filter on a nested filed](https://discuss.elastic.co/t/can-i-use-grok-filter-on-a-nested-filed/55839)
**Author:** [@gfafei](https://discuss.elastic.co/u/gfafei)\ **Replies:** 1\ **Last updated:** [July 19, 2016, 9:59am UTC](https://discuss.elastic.co/t/can-i-use-grok-filter-on-a-nested-filed/55839 "2016-07-19T09:59:27Z")
The log like this: { meta:{ path: '/app/app1'} } and my filter like this: grok { 'meta.path' =\> '/app/%{WORD:appid}' } it didn't work. How can I get the result I want? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1124) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1126)