# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=907 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 908 --- ## [Logstash adding additional one row](https://discuss.elastic.co/t/logstash-adding-additional-one-row/100747)
**Author:** [@saafianihan](https://discuss.elastic.co/u/saafianihan)\ **Replies:** 2\ **Last updated:** [September 19, 2017, 5:57am UTC](https://discuss.elastic.co/t/logstash-adding-additional-one-row/100747 "2017-09-19T05:57:01Z")
While executing the configuration file, logstash adding one additional row with the Query result. The additional row contains the "document\_id" details. I am using "document\_id" to prevent duplication of data for every r… --- ## [Advice for new deployment?](https://discuss.elastic.co/t/advice-for-new-deployment/100690)
**Author:** [@Speedman](https://discuss.elastic.co/u/Speedman)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 5:29am UTC](https://discuss.elastic.co/t/advice-for-new-deployment/100690 "2017-09-19T05:29:20Z")
Hi all. We're looking to start using the Elastic Stack in our systems, and while I've messed around with it personally in the past and know all of the basic concepts and ideas, the one sticking point right now is Logstas… --- ## [Bad output logstash](https://discuss.elastic.co/t/bad-output-logstash/100941)
**Author:** [@Erick\_Garcia\_Perez](https://discuss.elastic.co/u/Erick_Garcia_Perez)\ **Replies:** 3\ **Last updated:** [September 19, 2017, 4:10am UTC](https://discuss.elastic.co/t/bad-output-logstash/100941 "2017-09-19T04:10:34Z")
I'm trying to parse a log of apache\_access, when it have diferente entries, concatenates the whole file in the same message, having an entry similar to this: can not separate them by lines \\n66.250.73.158 - - \[06/Sep/2… --- ## [To run Logstash 5.5.2 need any server like windows server 2008 r2?](https://discuss.elastic.co/t/to-run-logstash-5-5-2-need-any-server-like-windows-server-2008-r2/100059)
**Author:** [@ramesh205](https://discuss.elastic.co/u/ramesh205)\ **Replies:** 3\ **Last updated:** [September 18, 2017, 10:20pm UTC](https://discuss.elastic.co/t/to-run-logstash-5-5-2-need-any-server-like-windows-server-2008-r2/100059 "2017-09-18T22:20:54Z")
Steps: 1.Downloaded latest logstash .zip file and unziped. 2.Created logstash.conf file: input{stdin{}} output{stdout{}} 3. In cmd prompt ran cd C:/Logstash5.5.2/bin/ 4.C:\\logstash-5.5.2\\bin\>logstash -f logstash.c… --- ## [Running Logstash as a service on windows using NSSM is not working](https://discuss.elastic.co/t/running-logstash-as-a-service-on-windows-using-nssm-is-not-working/100370)
**Author:** [@rajesh](https://discuss.elastic.co/u/rajesh)\ **Replies:** 9\ **Last updated:** [September 18, 2017, 2:37pm UTC](https://discuss.elastic.co/t/running-logstash-as-a-service-on-windows-using-nssm-is-not-working/100370 "2017-09-18T14:37:45Z")
Hi, I am trying to run Logstash as a service on windows using NSSM. It says the Logstash service is started successfully but I don't see any output logs(set the level as debug). Could it be related to permission issue? … --- ## [Logstash is not reading files in conf.d when running as a service(using NSSM) on windows](https://discuss.elastic.co/t/logstash-is-not-reading-files-in-conf-d-when-running-as-a-service-using-nssm-on-windows/100792)
**Author:** [@rajesh](https://discuss.elastic.co/u/rajesh)\ **Replies:** 2\ **Last updated:** [September 18, 2017, 2:34pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-files-in-conf-d-when-running-as-a-service-using-nssm-on-windows/100792 "2017-09-18T14:34:59Z")
Hi, I am running Logstash as a service on windows using NSSM, but it's not picking conf files from conf.d. I gave the path to config files in logstash.yml. When I pass the arguments(in NSSM) as '-f C:logstash\\install\\b… --- ## [Input tcp does not close the connection](https://discuss.elastic.co/t/input-tcp-does-not-close-the-connection/100869)
**Author:** [@Tonni\_Hult](https://discuss.elastic.co/u/Tonni_Hult)\ **Replies:** 3\ **Last updated:** [September 18, 2017, 1:51pm UTC](https://discuss.elastic.co/t/input-tcp-does-not-close-the-connection/100869 "2017-09-18T13:51:59Z")
Hi My problem is that the tcp plugin seems to keep a connection open instead of closing it after each call. Is this normal behaviour? We have a loadbalancer that constantly make GET requests to see that each node in the… --- ## [Parsing IIS logs having two different format](https://discuss.elastic.co/t/parsing-iis-logs-having-two-different-format/100779)
**Author:** [@kitex](https://discuss.elastic.co/u/kitex)\ **Replies:** 1\ **Last updated:** [September 18, 2017, 1:28pm UTC](https://discuss.elastic.co/t/parsing-iis-logs-having-two-different-format/100779 "2017-09-18T13:28:44Z")
I have iss giving logs in below format: September 16th 2017, 20:24:50.440 6.22.40.176 - - \[14/Jul/2017:08:34:57 +0545\] "GET /erport.aspx HTTP/1.1" 200 875 September 16th 2017, 20:24:50.440 2017-09-15 23:40:41 W3SVC2 WIN… --- ## [Logstash adding duplicate rows in every run](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453)
**Author:** [@saafianihan](https://discuss.elastic.co/u/saafianihan)\ **Replies:** 12\ **Last updated:** [September 18, 2017, 11:36am UTC](https://discuss.elastic.co/t/logstash-adding-duplicate-rows-in-every-run/100453 "2017-09-18T11:36:34Z")
i am using Logstash to pull the date from MySQL database. I am using two different configuration files to create two indexes with different MySQL queries. I run the conf files as logstash -f path\\to\\bin\*.conf . While r… --- ## [Cannot sending data from mongodb into elasticsearch using logstash](https://discuss.elastic.co/t/cannot-sending-data-from-mongodb-into-elasticsearch-using-logstash/100835)
**Author:** [@Muhammad\_Rizky](https://discuss.elastic.co/u/Muhammad_Rizky)\ **Replies:** 1\ **Last updated:** [September 18, 2017, 9:16am UTC](https://discuss.elastic.co/t/cannot-sending-data-from-mongodb-into-elasticsearch-using-logstash/100835 "2017-09-18T09:16:25Z")
Hi all, i have an issue when i try to get data from mongodb to elasticsearch using logstash-input-mongodb. There is no document that get into elastic but when i stdout using rubydebug, logstash show me the documents in … --- ## [How to use Translate plugin in grok filter](https://discuss.elastic.co/t/how-to-use-translate-plugin-in-grok-filter/100830)
**Author:** [@sreejiths](https://discuss.elastic.co/u/sreejiths)\ **Replies:** 5\ **Last updated:** [September 18, 2017, 8:23am UTC](https://discuss.elastic.co/t/how-to-use-translate-plugin-in-grok-filter/100830 "2017-09-18T08:23:34Z")
https://www.elastic.co/guide/en/logstash/5.2/plugins-filters-translate.html#plugins-filters-translate-dictionary\_path Hi , I have a specific requirement , I will have a csv file with data as below , what i want is if th… --- ## [Elasticsearch producing Duplicate entries in Kibana](https://discuss.elastic.co/t/elasticsearch-producing-duplicate-entries-in-kibana/100632)
**Author:** [@scayzer65](https://discuss.elastic.co/u/scayzer65)\ **Replies:** 9\ **Last updated:** [September 18, 2017, 7:45am UTC](https://discuss.elastic.co/t/elasticsearch-producing-duplicate-entries-in-kibana/100632 "2017-09-18T07:45:16Z")
Hi I have a Windows FTP server that produces 3 different logs and they are all output in to the same folder on C: Drive. I am trying to get a separation between the logs so that I can create custom filters. I have set u… --- ## [Help needed in configuring Logstash input plugin from Azure Storage Table](https://discuss.elastic.co/t/help-needed-in-configuring-logstash-input-plugin-from-azure-storage-table/100829)
**Author:** [@Prasad\_Kris](https://discuss.elastic.co/u/Prasad_Kris)\ **Replies:** 0\ **Last updated:** [September 18, 2017, 7:21am UTC](https://discuss.elastic.co/t/help-needed-in-configuring-logstash-input-plugin-from-azure-storage-table/100829 "2017-09-18T07:21:00Z")
Greetings, Could some one please help me to correctly configure the logstash-input-azurewadtable ELK plugin, I tried the setup guidelines in the below url, The plugin is getting installed properly, but unable to proce… --- ## [Grok Pattern for Apache Access logs (facing \_grokparsefailure error)](https://discuss.elastic.co/t/grok-pattern-for-apache-access-logs-facing--grokparsefailure-error/100796)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 4\ **Last updated:** [September 18, 2017, 4:35am UTC](https://discuss.elastic.co/t/grok-pattern-for-apache-access-logs-facing--grokparsefailure-error/100796 "2017-09-18T04:35:16Z")
Code input { file { path =\> "/data/Ops\_analytics\_EBS/oracle\_ebs\_eu/j201s648/access\_log.1476316800" start\_position =\> "beginning" type =\> "apache\_access" } } filter { grok { match =\> {'message'=\>'%{IP:clientip… --- ## [Query regarding GeoIP](https://discuss.elastic.co/t/query-regarding-geoip/99210)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 22\ **Last updated:** [September 18, 2017, 4:03am UTC](https://discuss.elastic.co/t/query-regarding-geoip/99210 "2017-09-18T04:03:17Z")
Hi Guys, I am taking nginx logs and with custom Index name and logs are being injested properly even I see Log/lt properly country name absilutely fine. However when I do visualization I see that regular error “No Comp… --- ## [Logstash not picking up all log data](https://discuss.elastic.co/t/logstash-not-picking-up-all-log-data/100518)
**Author:** [@Shrawan\_Bhagwat](https://discuss.elastic.co/u/Shrawan_Bhagwat)\ **Replies:** 1\ **Last updated:** [September 17, 2017, 5:08pm UTC](https://discuss.elastic.co/t/logstash-not-picking-up-all-log-data/100518 "2017-09-17T17:08:10Z")
Hi all, I am using logstash-1.5.3 since more than a year now. today i have noticed one of the issue, that logstash is not picking up all log data from log file. Logs are generating at 1000/min rate. Any idea. I know … --- ## [Logstash6 beta1- template output to Elasticsearch - 64bit int issue](https://discuss.elastic.co/t/logstash6-beta1-template-output-to-elasticsearch-64bit-int-issue/100793)
**Author:** [@randude](https://discuss.elastic.co/u/randude)\ **Replies:** 0\ **Last updated:** [September 17, 2017, 2:28pm UTC](https://discuss.elastic.co/t/logstash6-beta1-template-output-to-elasticsearch-64bit-int-issue/100793 "2017-09-17T14:28:50Z")
Hi, I'm using logstash6 beta 1 to send json logs to my elasticsearch server. I'm using logstash's elasticsearch output plugin with the template option. How can i set the default mappings of all integers to be longs? … --- ## [CheckPoint Netflow and Elasticsearch](https://discuss.elastic.co/t/checkpoint-netflow-and-elasticsearch/99993)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 8\ **Last updated:** [September 17, 2017, 12:38pm UTC](https://discuss.elastic.co/t/checkpoint-netflow-and-elasticsearch/99993 "2017-09-17T12:38:34Z")
Hi Guys, Is anyone here able to successfully configured netflow on checkpoint with elasticssearch? Is there any such use case? If any one can you help me with logstash configuration? --- ## [Couchdb\_changes plugin resetting .couchdb\_seq](https://discuss.elastic.co/t/couchdb-changes-plugin-resetting-couchdb-seq/100038)
**Author:** [@uschtwill](https://discuss.elastic.co/u/uschtwill)\ **Replies:** 3\ **Last updated:** [September 17, 2017, 10:13am UTC](https://discuss.elastic.co/t/couchdb-changes-plugin-resetting-couchdb-seq/100038 "2017-09-17T10:13:23Z")
I am using the Couchdb\_changes plugin. Recently, irregularly, during a 3 day index, the .couchdb\_seq file is deleted/resetted, so Logstash starts indexing from the start. I am fairly certain the file is not deleted from… --- ## [Subtract term X from Y on the same log?](https://discuss.elastic.co/t/subtract-term-x-from-y-on-the-same-log/100782)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 1\ **Last updated:** [September 17, 2017, 8:44am UTC](https://discuss.elastic.co/t/subtract-term-x-from-y-on-the-same-log/100782 "2017-09-17T08:44:34Z")
How can I Subtract term X from Y on the same log? I saw this, but it doesn't do any arithmetical use. (This uses mutate). I also tried through a scripted field in Kibana but the fields are not saved in ES and I need to… --- ## [Index and id tag to my log stash config](https://discuss.elastic.co/t/index-and-id-tag-to-my-log-stash-config/100719)
**Author:** [@Milton\_Sam\_Moses](https://discuss.elastic.co/u/Milton_Sam_Moses)\ **Replies:** 2\ **Last updated:** [September 17, 2017, 5:07am UTC](https://discuss.elastic.co/t/index-and-id-tag-to-my-log-stash-config/100719 "2017-09-17T05:07:13Z")
Hello, Exerts I have a logstash config with works well. Just need your advice to auto creat index and id to the output file. Config: input { file { path =\> "/tmp/sam" } } output { file { path =\> "/tmp/sam\_outpu… --- ## [The parsing process is abnormal when json contains 'message'](https://discuss.elastic.co/t/the-parsing-process-is-abnormal-when-json-contains-message/100772)
**Author:** [@Terence\_Chuen](https://discuss.elastic.co/u/Terence_Chuen)\ **Replies:** 0\ **Last updated:** [September 16, 2017, 6:19pm UTC](https://discuss.elastic.co/t/the-parsing-process-is-abnormal-when-json-contains-message/100772 "2017-09-16T18:19:51Z")
I try to parsing modsecurity json log like this: { "transaction": { "client\_ip": "116.30.195.51", "time\_stamp": "Sun Sep 17 01:16:56 2017", "server\_id": "d4425e970f21aa46cba26e2f672308813dcda22d", "client\_port": 63083,… --- ## [Strange Logstash index fields](https://discuss.elastic.co/t/strange-logstash-index-fields/100689)
**Author:** [@Diggy](https://discuss.elastic.co/u/Diggy)\ **Replies:** 3\ **Last updated:** [September 16, 2017, 2:20pm UTC](https://discuss.elastic.co/t/strange-logstash-index-fields/100689 "2017-09-16T14:20:27Z")
Good morning or afternoon, folks. I hope I present my issue understandably, and in the right forum. I'm running Elastic Stack 5.6, and most things are working well. However, when I try to create a Logstash-based visua… --- ## [JSON logs not being converted in filebeat/logstash](https://discuss.elastic.co/t/json-logs-not-being-converted-in-filebeat-logstash/100759)
**Author:** [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\ **Replies:** 0\ **Last updated:** [September 16, 2017, 10:35am UTC](https://discuss.elastic.co/t/json-logs-not-being-converted-in-filebeat-logstash/100759 "2017-09-16T10:35:15Z")
{"@timestamp":"2017-08-15T21:30:03.776+00:00","@version":1,"message":"Registry invocation httpStatus: 401 ","logger\_name":"invoker.CloudRegistryInvoker","thread\_name":"http-nio-7010-exec-10","level":"INFO","level\_value":… --- ## [Get field with nested array using ruby filter](https://discuss.elastic.co/t/get-field-with-nested-array-using-ruby-filter/100661)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [September 16, 2017, 5:19am UTC](https://discuss.elastic.co/t/get-field-with-nested-array-using-ruby-filter/100661 "2017-09-16T05:19:34Z")
Hi All I have a json data in this format \`{"foo":"bar","test":{"steps":\[{"response\_time":"100"},{"response\_time":"101","more\_nested":\[{"hello":"world"},{"hello2":"world2"}\]}\]}}\` i am getting result in kibana as below … --- ## [\_\_dateparsefailure trying match date](https://discuss.elastic.co/t/--dateparsefailure-trying-match-date/100734)
**Author:** [@SamuelF](https://discuss.elastic.co/u/SamuelF)\ **Replies:** 1\ **Last updated:** [September 16, 2017, 5:16am UTC](https://discuss.elastic.co/t/--dateparsefailure-trying-match-date/100734 "2017-09-16T05:16:58Z")
Hi, I'm trying to match timestamp from log file and i have an \_dateparsefailure :frowning: This is my grok pattern (it match) : grok { match =\> { "message" =\> \["\\\[(?\%{MONTHDAY}-%{MONTH}-%{YEAR}\\s+%{TIME… --- ## [Math function in Logstash](https://discuss.elastic.co/t/math-function-in-logstash/100721)
**Author:** [@ashvita](https://discuss.elastic.co/u/ashvita)\ **Replies:** 2\ **Last updated:** [September 16, 2017, 5:10am UTC](https://discuss.elastic.co/t/math-function-in-logstash/100721 "2017-09-16T05:10:56Z")
Hi, I need to calculate 100-total. Which filter should I use? This is the message that I receive - NE\_217\_225- Cpu(s): 16.6%us, 7.4%sy, 0.0%ni, 74.9%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st My grok filter is as follo… --- ## [Filter specific words from a field in logstash](https://discuss.elastic.co/t/filter-specific-words-from-a-field-in-logstash/100614)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 3\ **Last updated:** [September 16, 2017, 2:32am UTC](https://discuss.elastic.co/t/filter-specific-words-from-a-field-in-logstash/100614 "2017-09-16T02:32:23Z")
I'm pulling data from SQL and one of the table has field called 'description' which contains values like alarm created, alarm discontinued, sound generated, api called, api call failed. All I wanted to do is capture onl… --- ## [Grok Pattern for Apache Error Logs](https://discuss.elastic.co/t/grok-pattern-for-apache-error-logs/100582)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 16\ **Last updated:** [September 15, 2017, 1:31pm UTC](https://discuss.elastic.co/t/grok-pattern-for-apache-error-logs/100582 "2017-09-15T13:31:00Z")
Below is the error log: \[Mon Nov 28 04:38:24 2016\] \[error\] \[client 10.114.34.43\] File does not exist: /fep10/oraapps/appl/fep10comn/portal/FEP10\_j201s648/favicon.ico I had written grok pattern as "message"=\> "\[(?%{DAY… --- ## [Load Indexed Data Into Elastic Search Engine through Logstash by executing a Stored Procedure Returning multiple resultsets](https://discuss.elastic.co/t/load-indexed-data-into-elastic-search-engine-through-logstash-by-executing-a-stored-procedure-returning-multiple-resultsets/100686)
**Author:** [@Akanksha](https://discuss.elastic.co/u/Akanksha)\ **Replies:** 0\ **Last updated:** [September 15, 2017, 12:50pm UTC](https://discuss.elastic.co/t/load-indexed-data-into-elastic-search-engine-through-logstash-by-executing-a-stored-procedure-returning-multiple-resultsets/100686 "2017-09-15T12:50:34Z")
Pls help regarding Loading of Indexed Data Into Elastic Search Engine through Logstash by executing a Stored Procedure Returning multiple resultsets. [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=906) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=908)