# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1162 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1163 --- ## [Reindex data in order to pass the migration Checker](https://discuss.elastic.co/t/reindex-data-in-order-to-pass-the-migration-checker/48335)
**Author:** [@genesis](https://discuss.elastic.co/u/genesis)\ **Replies:** 1\ **Last updated:** [April 26, 2016, 2:51am UTC](https://discuss.elastic.co/t/reindex-data-in-order-to-pass-the-migration-checker/48335 "2016-04-26T02:51:23Z")
Some of our data has dots (.) in them which is preventing us from upgrading from 1.7.1 to 2.3 Our plan is to reindex 433 indices about 15.74 TB of data. This is a sample of our data containing dots logs:gxc.xxxx, logs:gx… --- ## [Passing Multiple Arguments while Logstash startup](https://discuss.elastic.co/t/passing-multiple-arguments-while-logstash-startup/47805)
**Author:** [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Replies:** 14\ **Last updated:** [April 26, 2016, 1:12am UTC](https://discuss.elastic.co/t/passing-multiple-arguments-while-logstash-startup/47805 "2016-04-26T01:12:42Z")
I have more than 5 logstash conf files. I need to start all the conf files all together. How can I pass all the files names together ? logstash -f Logstash-ES1.conf logstash -f Logstash-ES2.conf logstash -f Logstash-E… --- ## [Logstash to haddop](https://discuss.elastic.co/t/logstash-to-haddop/48332)
**Author:** [@Lavanya\_Anne](https://discuss.elastic.co/u/Lavanya_Anne)\ **Replies:** 0\ **Last updated:** [April 25, 2016, 2:27pm UTC](https://discuss.elastic.co/t/logstash-to-haddop/48332 "2016-04-25T14:27:14Z")
1.can beats send data to hadoop? 2. can logstash send data to hadoop?(for this i tried webhdfs) but i am getting the following error. Failed to flush outgoing items {:outgoing\_count=\>1, :exception=\>"WebHDFS::ServerEr… --- ## [Elastic Search Fails To Index](https://discuss.elastic.co/t/elastic-search-fails-to-index/48354)
**Author:** [@jhvh111](https://discuss.elastic.co/u/jhvh111)\ **Replies:** 0\ **Last updated:** [April 25, 2016, 9:56pm UTC](https://discuss.elastic.co/t/elastic-search-fails-to-index/48354 "2016-04-25T21:56:15Z")
Continuing the discussion from Logstash won't create index in ES: Just in case it helps others in my scenario: It looks like there are a lot of reasons why Elasticsearch might not index a Logstash. In my case I sim… --- ## [Best way to add a field if the field contatins a specific string](https://discuss.elastic.co/t/best-way-to-add-a-field-if-the-field-contatins-a-specific-string/48342)
**Author:** [@DavidL](https://discuss.elastic.co/u/DavidL)\ **Replies:** 1\ **Last updated:** [April 25, 2016, 6:01pm UTC](https://discuss.elastic.co/t/best-way-to-add-a-field-if-the-field-contatins-a-specific-string/48342 "2016-04-25T18:01:13Z")
Hello, I am processing logs for a third party application. I already parsed out a field called "error\_message". I know that for some records, there is a error code in the error\_message. For example, "ERROR-11-8888", I w… --- ## [Why not always use %{DATA} as the semantic in the initial grok match?](https://discuss.elastic.co/t/why-not-always-use-data-as-the-semantic-in-the-initial-grok-match/48180)
**Author:** [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Replies:** 3\ **Last updated:** [April 25, 2016, 3:20pm UTC](https://discuss.elastic.co/t/why-not-always-use-data-as-the-semantic-in-the-initial-grok-match/48180 "2016-04-25T15:20:46Z")
From the grok page, the suggested matching for log entries might look like this. %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} In other words, the suggestion is to use syntax … --- ## [Analyze in LogStash or Index in ElasticSearch](https://discuss.elastic.co/t/analyze-in-logstash-or-index-in-elasticsearch/48179)
**Author:** [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Replies:** 2\ **Last updated:** [April 25, 2016, 3:19pm UTC](https://discuss.elastic.co/t/analyze-in-logstash-or-index-in-elasticsearch/48179 "2016-04-25T15:19:04Z")
I have a "best practices" question. Should I try to analyze/index in logstash and then send successful analysis as extra data to ES OR should I read data in logstash as DATA and then analyze using a mapping template and … --- ## [Unable to change index off default logstash-\*](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989)
**Author:** [@kprime](https://discuss.elastic.co/u/kprime)\ **Replies:** 5\ **Last updated:** [April 25, 2016, 1:31pm UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989 "2016-04-25T13:31:51Z")
Hello, I am trying to get logstash to not use the default index but it isn't working for me. I have a JSON template that is successful updating in ES but logstash keeps trying to use the default index and parsing error… --- ## [Logstash abysmal performance from 2.2.0 to 2.3.1](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638)
**Author:** [@KlavsKlavsen](https://discuss.elastic.co/u/KlavsKlavsen)\ **Replies:** 5\ **Last updated:** [April 25, 2016, 12:21pm UTC](https://discuss.elastic.co/t/logstash-abysmal-performance-from-2-2-0-to-2-3-1/47638 "2016-04-25T12:21:23Z")
Hi guys, We are running logstash 2.2.0 (against elasticsearch 2.3.1) and it can index \>1.2mill docs/minute. If we then upgrade to logstash-2.3.1 - it peaks at 200k docs/min.. 1/6 of the performance. There's nothing i… --- ## [\[filter elasticsearch\] sort query](https://discuss.elastic.co/t/filter-elasticsearch-sort-query/48314)
**Author:** [@sen](https://discuss.elastic.co/u/sen)\ **Replies:** 0\ **Last updated:** [April 25, 2016, 11:46am UTC](https://discuss.elastic.co/t/filter-elasticsearch-sort-query/48314 "2016-04-25T11:46:10Z")
Hello everyone, I am pretty new regarding ELK. I want to use the elasticsearch filter plugin in logstash to query event in elasticsearch in order to do some operations between previous log events and the current log ev… --- ## [Logstash - obscure sensitive fields before putting them to the ES](https://discuss.elastic.co/t/logstash-obscure-sensitive-fields-before-putting-them-to-the-es/48253)
**Author:** [@PavelPolyakov](https://discuss.elastic.co/u/PavelPolyakov)\ **Replies:** 8\ **Last updated:** [April 25, 2016, 10:31am UTC](https://discuss.elastic.co/t/logstash-obscure-sensitive-fields-before-putting-them-to-the-es/48253 "2016-04-25T10:31:36Z")
Hi, Recently I came up with the next question. Imagine the next, I have an application which writes the logs. Each log is simple JSON object with the depth of N. Developers don't think about the values they put it th… --- ## [Elasticsearch output problem](https://discuss.elastic.co/t/elasticsearch-output-problem/47954)
**Author:** [@Infra-dev](https://discuss.elastic.co/u/Infra-dev)\ **Replies:** 3\ **Last updated:** [April 25, 2016, 10:07am UTC](https://discuss.elastic.co/t/elasticsearch-output-problem/47954 "2016-04-25T10:07:35Z")
Hi guys I've got a problem after did upgrade to the latest logstash, elasticsearch and kibana. Elasticsearch with kibana works fine. I see data, searches, visualisation and dashboards work perfect. But logstash even does… --- ## [Multiline codec discrepancies between file and stdin inputs](https://discuss.elastic.co/t/multiline-codec-discrepancies-between-file-and-stdin-inputs/47686)
**Author:** [@AlexClineBB](https://discuss.elastic.co/u/AlexClineBB)\ **Replies:** 6\ **Last updated:** [April 25, 2016, 9:10am UTC](https://discuss.elastic.co/t/multiline-codec-discrepancies-between-file-and-stdin-inputs/47686 "2016-04-25T09:10:47Z")
I'm trying to track down a bug in the file input that is combining multiline events separated by a blank line. Here's the alerts.log file which has events: \*\* Alert 1460993714.111282277: mail - syslog,linuxkernel,serv… --- ## [Delete a field in filter, but use it in output](https://discuss.elastic.co/t/delete-a-field-in-filter-but-use-it-in-output/48008)
**Author:** [@JvonRudno](https://discuss.elastic.co/u/JvonRudno)\ **Replies:** 7\ **Last updated:** [April 25, 2016, 8:07am UTC](https://discuss.elastic.co/t/delete-a-field-in-filter-but-use-it-in-output/48008 "2016-04-25T08:07:43Z")
Hi everybody, I am using: input { jdbc { } } filter { elasticsearch { remove\_field =\> \["index\_type"\] } } output { elasticsearch{ index =\> "myindex" … --- ## [Gathering logs through Rsyslog/Redis \> Logstash](https://discuss.elastic.co/t/gathering-logs-through-rsyslog-redis-logstash/48277)
**Author:** [@brayndasilva](https://discuss.elastic.co/u/brayndasilva)\ **Replies:** 0\ **Last updated:** [April 25, 2016, 7:15am UTC](https://discuss.elastic.co/t/gathering-logs-through-rsyslog-redis-logstash/48277 "2016-04-25T07:15:28Z")
Hi, I'm going to be deploying an ELK stack on a small to medium network very soon and have got my documentation and such all ready to go, but I was wondering about one thing. I will be capturing Syslogs from routers/… --- ## [Variable fieldname to be used as GREEDYDATA field](https://discuss.elastic.co/t/variable-fieldname-to-be-used-as-greedydata-field/48170)
**Author:** [@Zfs](https://discuss.elastic.co/u/Zfs)\ **Replies:** 5\ **Last updated:** [April 25, 2016, 5:33am UTC](https://discuss.elastic.co/t/variable-fieldname-to-be-used-as-greedydata-field/48170 "2016-04-25T05:33:01Z")
Hi, I have the following logs: 2016-04-19 12:48:41,628 \[DefaultPool\] INFO DataXXXLog - some log data 2016-04-19 12:48:41,628 \[DefaultPool\] INFO DataYYYLog - some log data 2016-04-19 12:48:41,628 \[DefaultPool\] INFO Work… --- ## [Logstash Reindex Strategy](https://discuss.elastic.co/t/logstash-reindex-strategy/42812)
**Author:** [@James\_Tang](https://discuss.elastic.co/u/James_Tang)\ **Replies:** 3\ **Last updated:** [April 25, 2016, 4:20am UTC](https://discuss.elastic.co/t/logstash-reindex-strategy/42812 "2016-04-25T04:20:27Z")
Created a Gist documenting my approach to reindexing using logstash. Hopes it helps Reindexing your Elasticsearch indice with limited resource can be a painw when you have limited resources and need it running at the s… --- ## [How tcp input add fields?](https://discuss.elastic.co/t/how-tcp-input-add-fields/48164)
**Author:** [@iyaozhen](https://discuss.elastic.co/u/iyaozhen)\ **Replies:** 3\ **Last updated:** [April 25, 2016, 4:00am UTC](https://discuss.elastic.co/t/how-tcp-input-add-fields/48164 "2016-04-25T04:00:34Z")
I use python tail log file, and publish to logstash(tcp input). Because my system problem(#46708), I can't use filebeat, and filebeat have a very useful setting fields. How can I achieve this features by python. --- ## [JDBC duplicate messages - Logstash still created it's own document\_id after replaced with other value](https://discuss.elastic.co/t/jdbc-duplicate-messages-logstash-still-created-its-own-document-id-after-replaced-with-other-value/48139)
**Author:** [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Replies:** 0\ **Last updated:** [April 22, 2016, 8:30am UTC](https://discuss.elastic.co/t/jdbc-duplicate-messages-logstash-still-created-its-own-document-id-after-replaced-with-other-value/48139 "2016-04-22T08:30:33Z")
Dear supports, I'm trying to use jdbc plugin to upload SQL server data to Elasticsearch, I'm using logstash cluster with 2 nodes, this is my config on 2 nodes: +input { jdbc { ...... statement =\> "SELECT \* FROM log… --- ## [Logstash-plugin missing from yum repo?](https://discuss.elastic.co/t/logstash-plugin-missing-from-yum-repo/48242)
**Author:** [@vchav73](https://discuss.elastic.co/u/vchav73)\ **Replies:** 3\ **Last updated:** [April 25, 2016, 1:06am UTC](https://discuss.elastic.co/t/logstash-plugin-missing-from-yum-repo/48242 "2016-04-25T01:06:49Z")
I am updating from a 1.x installation to 2.3 on a CentOS 6 server. I removed the 1.x install using yum, saving the config files. I then installed 2.3 using this procedure: https://www.elastic.co/guide/en/logstash/current… --- ## [How to repeat grok-Part n-times](https://discuss.elastic.co/t/how-to-repeat-grok-part-n-times/44497)
**Author:** [@jupp](https://discuss.elastic.co/u/jupp)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 9:45pm UTC](https://discuss.elastic.co/t/how-to-repeat-grok-part-n-times/44497 "2016-04-24T21:45:37Z")
Hello Community ! I try to parse following: -- New Customer --------- time customerData1 time customerData2 time date id: 1 time Service: a time OnlineRequest time OnlineResponse time text time \[Status\] Code: Text ti… --- ## [Aggregate?](https://discuss.elastic.co/t/aggregate/47486)
**Author:** [@Rubytor](https://discuss.elastic.co/u/Rubytor)\ **Replies:** 10\ **Last updated:** [April 24, 2016, 9:12pm UTC](https://discuss.elastic.co/t/aggregate/47486 "2016-04-24T21:12:48Z")
What is the best solution for one of the basic requirements in log's analisis. "CALCULATE DURATION" The problem is to calculate the duration between 2 events. For example: Logs lines: 15-04-2016T10:00:00:000+UTC 1… --- ## [Logstash crashing:NoMethodError: undefined method \`map\_cleanup' for nil:NilClass](https://discuss.elastic.co/t/logstash-crashing-nomethoderror-undefined-method-map-cleanup-for-nil-nilclass/46630)
**Author:** [@baneshbanna](https://discuss.elastic.co/u/baneshbanna)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 7:43pm UTC](https://discuss.elastic.co/t/logstash-crashing-nomethoderror-undefined-method-map-cleanup-for-nil-nilclass/46630 "2016-04-24T19:43:13Z")
Logstash crashed with the below exception. No information available about this exception. Thanks in advance. NoMethodError: undefined method \`map\_cleanup' for nil:NilClass start at /apps/logstash-2.2.2/vendor/bundle/jr… --- ## [Logstash output cannot match a field set in filebeat](https://discuss.elastic.co/t/logstash-output-cannot-match-a-field-set-in-filebeat/46599)
**Author:** [@jbrelsford](https://discuss.elastic.co/u/jbrelsford)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 7:39pm UTC](https://discuss.elastic.co/t/logstash-output-cannot-match-a-field-set-in-filebeat/46599 "2016-04-24T19:39:12Z")
I've set up a prospector in filebeat that looks as follows: paths: - /mycompany/somedata/\* fields: service: dataevent env: int appName: Myapp So I'd like to route any item that is written to /… --- ## [Grok patterns for BIND DNS server](https://discuss.elastic.co/t/grok-patterns-for-bind-dns-server/48203)
**Author:** [@icb](https://discuss.elastic.co/u/icb)\ **Replies:** 4\ **Last updated:** [April 24, 2016, 6:44pm UTC](https://discuss.elastic.co/t/grok-patterns-for-bind-dns-server/48203 "2016-04-24T18:44:54Z")
Does anyone have a set of good grok patterns for the BIND DNS server query logs? The messages I'm interested in look like this: Apr 22 15:40:26 ns.example.com named\[1351\]: queries: info: client 192.0.2.215#53591: query… --- ## [Change default Logstash Document Type](https://discuss.elastic.co/t/change-default-logstash-document-type/48187)
**Author:** [@Kryten](https://discuss.elastic.co/u/Kryten)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 5:26pm UTC](https://discuss.elastic.co/t/change-default-logstash-document-type/48187 "2016-04-24T17:26:34Z")
Hi, Is there any way to change the default Logstash document type from "logs" to something else? For example: { "myindex-2016.04.21": { "aliases": {}, "mappings": { "logs": { … --- ## [How do are check existing fields, if doesnt exists add field](https://discuss.elastic.co/t/how-do-are-check-existing-fields-if-doesnt-exists-add-field/48148)
**Author:** [@KabeloShongwe](https://discuss.elastic.co/u/KabeloShongwe)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 4:10pm UTC](https://discuss.elastic.co/t/how-do-are-check-existing-fields-if-doesnt-exists-add-field/48148 "2016-04-24T16:10:32Z")
I'm indexing cisco logs, were some fileds dont exists in other events. please help. --- ## [How to get RSS feed plugin from indexing duplicate posts](https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776)
**Author:** [@MaxOrelus](https://discuss.elastic.co/u/MaxOrelus)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 12:16pm UTC](https://discuss.elastic.co/t/how-to-get-rss-feed-plugin-from-indexing-duplicate-posts/40776 "2016-04-24T12:16:13Z")
I'm currently playing with the RSS plugin for Logstash to index articles from my website. The plugin works good, but I notice if your interval is every 30 seconds or so, Logstash seems to re-index the same posts over and… --- ## [Logstash 5.0 stops after sometime](https://discuss.elastic.co/t/logstash-5-0-stops-after-sometime/47351)
**Author:** [@sukhada369](https://discuss.elastic.co/u/sukhada369)\ **Replies:** 2\ **Last updated:** [April 24, 2016, 1:42am UTC](https://discuss.elastic.co/t/logstash-5-0-stops-after-sometime/47351 "2016-04-24T01:42:02Z")
I am exploring on Elastic 5.0 alpha. I installed all the components(ELKB) on same machine. I have used the same configuration as I did on working stack of ELKB in its previous version. The problem is that, logstash stops… --- ## [Run logstash config file](https://discuss.elastic.co/t/run-logstash-config-file/48141)
**Author:** [@sarbjeet](https://discuss.elastic.co/u/sarbjeet)\ **Replies:** 1\ **Last updated:** [April 24, 2016, 12:21am UTC](https://discuss.elastic.co/t/run-logstash-config-file/48141 "2016-04-24T00:21:39Z")
Default pipeline workers: 1 Error: The setting host in plugin elasticsearch is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:po… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1161) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1163)