# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=976 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 977 --- ## [Sql server sourse with nvarchar(max) field as json document , how can i parse it into elastic as nested document](https://discuss.elastic.co/t/sql-server-sourse-with-nvarchar-max-field-as-json-document-how-can-i-parse-it-into-elastic-as-nested-document/86900)
**Author:** [@neo](https://discuss.elastic.co/u/neo)\ **Replies:** 1\ **Last updated:** [May 28, 2017, 6:39pm UTC](https://discuss.elastic.co/t/sql-server-sourse-with-nvarchar-max-field-as-json-document-how-can-i-parse-it-into-elastic-as-nested-document/86900 "2017-05-28T18:39:47Z")
sql server sourse with nvarchar(max) field as json document , how can i parse it into elastic as nested document --- ## [Logstash as a service windows](https://discuss.elastic.co/t/logstash-as-a-service-windows/87217)
**Author:** [@kalman](https://discuss.elastic.co/u/kalman)\ **Replies:** 1\ **Last updated:** [May 28, 2017, 4:24pm UTC](https://discuss.elastic.co/t/logstash-as-a-service-windows/87217 "2017-05-28T16:24:35Z")
Is there a official way you can run logstash as a service on windows? because logstash is started with a bat, how can I run it as a manged service to monitor if it crashes or if something happens in production --- ## [How to write values from dynamic kv filter into csv file](https://discuss.elastic.co/t/how-to-write-values-from-dynamic-kv-filter-into-csv-file/87368)
**Author:** [@gvhome](https://discuss.elastic.co/u/gvhome)\ **Replies:** 0\ **Last updated:** [May 28, 2017, 4:16pm UTC](https://discuss.elastic.co/t/how-to-write-values-from-dynamic-kv-filter-into-csv-file/87368 "2017-05-28T16:16:40Z")
Hi I am new to Logstash/ELK and venturing on parsing a complex epm log format. The record has fixed part with fixed number of fields and dynamic part with variable number of fields with (field=value) pairs. The value can… --- ## [Logstash JSON Parse error.... With valid JSON data](https://discuss.elastic.co/t/logstash-json-parse-error-with-valid-json-data/50455)
**Author:** [@jstmatt](https://discuss.elastic.co/u/jstmatt)\ **Replies:** 4\ **Last updated:** [May 28, 2017, 10:29am UTC](https://discuss.elastic.co/t/logstash-json-parse-error-with-valid-json-data/50455 "2017-05-28T10:29:23Z")
Hi All, We have an issue where data from Filebeat (1.2.2) is being read from files, sent to Logstash (2.3.1). We are receiving, regularly, errors stating that the data can not be parsed. I verified that the JSON is, … --- ## [\-e doesn't work using logstash image from docker.elastic.co](https://discuss.elastic.co/t/e-doesnt-work-using-logstash-image-from-docker-elastic-co/87351)
**Author:** [@kinghuang](https://discuss.elastic.co/u/kinghuang)\ **Replies:** 2\ **Last updated:** [May 28, 2017, 2:37am UTC](https://discuss.elastic.co/t/e-doesnt-work-using-logstash-image-from-docker-elastic-co/87351 "2017-05-28T02:37:45Z")
I'm trying to switch from the logstash official image on Docker Hub to Elastic's registry, given the Docker Hub images won't be updated past June 20. But, the image provided by Elastic's registry doesn't seem to work pro… --- ## [How to mutate convert field in Json file input](https://discuss.elastic.co/t/how-to-mutate-convert-field-in-json-file-input/87327)
**Author:** [@Tal\_Bar\_Or](https://discuss.elastic.co/u/Tal_Bar_Or)\ **Replies:** 0\ **Last updated:** [May 27, 2017, 8:01am UTC](https://discuss.elastic.co/t/how-to-mutate-convert-field-in-json-file-input/87327 "2017-05-27T08:01:24Z")
Hi, I am reading Json file and trying to mutate/convert some fields to float for example , but its not working for me. Its seems that i am missing the right way doing so ,below i have my config file and json example s… --- ## [Log files authority should be controlled](https://discuss.elastic.co/t/log-files-authority-should-be-controlled/87324)
**Author:** [@caixiac1](https://discuss.elastic.co/u/caixiac1)\ **Replies:** 0\ **Last updated:** [May 27, 2017, 6:28am UTC](https://discuss.elastic.co/t/log-files-authority-should-be-controlled/87324 "2017-05-27T06:28:01Z")
Kibana and Logstash has their log files stored under /var/log by default and they have the following authority settings. But based on CIS security scan report, no application should have Others Read to be set false to en… --- ## [Logstash with systemctl not work](https://discuss.elastic.co/t/logstash-with-systemctl-not-work/85538)
**Author:** [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\ **Replies:** 4\ **Last updated:** [May 27, 2017, 1:08am UTC](https://discuss.elastic.co/t/logstash-with-systemctl-not-work/85538 "2017-05-27T01:08:37Z")
after I installed x-pack, I could only make logstash work by /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/index\_from\_redis.conf --config.reload.automatic --log.level=debug sudo systemctl start logstash.ser… --- ## [Logstash Adds unwanted records](https://discuss.elastic.co/t/logstash-adds-unwanted-records/87298)
**Author:** [@Lorenc\_Hysi](https://discuss.elastic.co/u/Lorenc_Hysi)\ **Replies:** 1\ **Last updated:** [May 26, 2017, 10:02pm UTC](https://discuss.elastic.co/t/logstash-adds-unwanted-records/87298 "2017-05-26T22:02:29Z")
Hello, I have logstash with the following config: Kafka Input Filter metrics Syslog Output File Output Logstash is reading from a kafka topic as type plain However at different points in time after it has consumed … --- ## [Extract date from message \[SOLVED\]](https://discuss.elastic.co/t/extract-date-from-message-solved/87296)
**Author:** [@javier-lopez](https://discuss.elastic.co/u/javier-lopez)\ **Replies:** 1\ **Last updated:** [May 26, 2017, 9:56pm UTC](https://discuss.elastic.co/t/extract-date-from-message-solved/87296 "2017-05-26T21:56:22Z")
Hello forum, I'm new to logstash and having issues trying to extract a log substring as timestamp. The following set May 22 as the @timestamp value: echo "May 22 12:42:16" | ./logstash-5.4.0/bin/logstash -e "input … --- ## [Logstash BigQuery plugin health](https://discuss.elastic.co/t/logstash-bigquery-plugin-health/87288)
**Author:** [@criccomini](https://discuss.elastic.co/u/criccomini)\ **Replies:** 0\ **Last updated:** [May 26, 2017, 6:30pm UTC](https://discuss.elastic.co/t/logstash-bigquery-plugin-health/87288 "2017-05-26T18:30:04Z")
Hey all, I want to know the status of the Logstash bigquery communicty contributed plugin: https://github.com/logstash-plugins/logstash-output-google\_bigquery We have started running it, and found several pretty … --- ## [ODSEE 11g multiline log processing in Logstash 5.4](https://discuss.elastic.co/t/odsee-11g-multiline-log-processing-in-logstash-5-4/86338)
**Author:** [@abhijitnt](https://discuss.elastic.co/u/abhijitnt)\ **Replies:** 1\ **Last updated:** [May 26, 2017, 3:03pm UTC](https://discuss.elastic.co/t/odsee-11g-multiline-log-processing-in-logstash-5-4/86338 "2017-05-26T15:03:29Z")
Hello All. I am new to ELK and started learning with installing latest ELK stack 5.4 on linux. Setup is working fine and able to stash logs and search in Kibana. But would like to structure logs so its more relevant and … --- ## [Duplicate Events captured in Indexes](https://discuss.elastic.co/t/duplicate-events-captured-in-indexes/86899)
**Author:** [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Replies:** 2\ **Last updated:** [May 26, 2017, 2:49pm UTC](https://discuss.elastic.co/t/duplicate-events-captured-in-indexes/86899 "2017-05-26T14:49:59Z")
I have 2 indexes named MyIndex1 and MyIndex2. I have 2 different conf files targeting multiple paths. Both conf has different paths completely. These conf at the end has individual index names. The only common thing is… --- ## [KV Filter to include spaces in selected fields/values](https://discuss.elastic.co/t/kv-filter-to-include-spaces-in-selected-fields-values/86733)
**Author:** [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Replies:** 5\ **Last updated:** [May 26, 2017, 2:49pm UTC](https://discuss.elastic.co/t/kv-filter-to-include-spaces-in-selected-fields-values/86733 "2017-05-26T14:49:24Z")
My log line has a field with "\[reasonPhrase : Internal Server Error\]" as value. I need to extract the reasonPhrase field and value as Internal Server Error. But with the current filter or the default kv filter, only Int… --- ## [How to read log.gz?](https://discuss.elastic.co/t/how-to-read-log-gz/87267)
**Author:** [@BERUSO](https://discuss.elastic.co/u/BERUSO)\ **Replies:** 0\ **Last updated:** [May 26, 2017, 2:48pm UTC](https://discuss.elastic.co/t/how-to-read-log-gz/87267 "2017-05-26T14:48:11Z")
Hi! any idea of how to read log.gz in logstash-Linux, it's UTF-8 format. I've tried "gzip\_lines", but it doesn't load any data. This is the message... Sending Logstash's logs to /usr1/kibana/logstash-5.4.0/logs which i… --- ## [Http/gelf/beats input host mask not working (docker network)](https://discuss.elastic.co/t/http-gelf-beats-input-host-mask-not-working-docker-network/86128)
**Author:** [@pir1981](https://discuss.elastic.co/u/pir1981)\ **Replies:** 4\ **Last updated:** [May 26, 2017, 2:36pm UTC](https://discuss.elastic.co/t/http-gelf-beats-input-host-mask-not-working-docker-network/86128 "2017-05-26T14:36:26Z")
Hi, I am trying to allow input only from my own network and thus have the following .conf input{ gelf { type =\> "log" # host =\> "172.16.0.0" # port =\> 12201 } beats { # … --- ## [Problem with jdbc\_streaming](https://discuss.elastic.co/t/problem-with-jdbc-streaming/86622)
**Author:** [@nano](https://discuss.elastic.co/u/nano)\ **Replies:** 3\ **Last updated:** [May 22, 2017, 2:59pm UTC](https://discuss.elastic.co/t/problem-with-jdbc-streaming/86622 "2017-05-22T14:59:27Z")
Good morning, please i have some quetions about jdbc\_streaming , i made such a configuration to extract some data using jdbc\_streaming but i didn't acheive to my goal . there is my configuration file jdbc\_streami… --- ## [How to Seperate docs in Logstash to different ES index](https://discuss.elastic.co/t/how-to-seperate-docs-in-logstash-to-different-es-index/87249)
**Author:** [@yasvanth](https://discuss.elastic.co/u/yasvanth)\ **Replies:** 4\ **Last updated:** [May 26, 2017, 1:19pm UTC](https://discuss.elastic.co/t/how-to-seperate-docs-in-logstash-to-different-es-index/87249 "2017-05-26T13:19:20Z")
Hi All, I am inserting data to logstash(collector) via nxlog(sender) to single port. The logs are already parsed on nxlog side from .xml to json , these parsed doc contains logs from different application such as apach… --- ## [Synchronize 2 index continuously](https://discuss.elastic.co/t/synchronize-2-index-continuously/86423)
**Author:** [@raphaelkh2ad](https://discuss.elastic.co/u/raphaelkh2ad)\ **Replies:** 2\ **Last updated:** [May 26, 2017, 1:16pm UTC](https://discuss.elastic.co/t/synchronize-2-index-continuously/86423 "2017-05-26T13:16:31Z")
Hello guys, I would like to use logstash to read from an index (from cluster 1) and write to another index (in cluster 2), and keep this second index always synced with the input index. Is there a way to achieved this … --- ## [Hash generation](https://discuss.elastic.co/t/hash-generation/87146)
**Author:** [@setshocker](https://discuss.elastic.co/u/setshocker)\ **Replies:** 1\ **Last updated:** [May 26, 2017, 10:24am UTC](https://discuss.elastic.co/t/hash-generation/87146 "2017-05-26T10:24:18Z")
Hi everyone, I would like to know if Logstash has any function that generate a hash of every log that it receives. The main purporse of this is to ensure (in the future) that logs has not been tampered(calculating agai… --- ## [Logstash Does Not Read In Last Line of Log](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706)
**Author:** [@michaellizhou](https://discuss.elastic.co/u/michaellizhou)\ **Replies:** 15\ **Last updated:** [May 26, 2017, 10:21am UTC](https://discuss.elastic.co/t/logstash-does-not-read-in-last-line-of-log/2706 "2017-05-26T10:21:24Z")
So recently noticed that my logstash config is not reading in the last line from my log file. I think that it may be my input config fault though I do not see any problems with it right now. What else could it be? Here i… --- ## [Combine message parts from Logfiles that share the same id and write them into new document](https://discuss.elastic.co/t/combine-message-parts-from-logfiles-that-share-the-same-id-and-write-them-into-new-document/87226)
**Author:** [@jasdf](https://discuss.elastic.co/u/jasdf)\ **Replies:** 0\ **Last updated:** [May 26, 2017, 10:01am UTC](https://discuss.elastic.co/t/combine-message-parts-from-logfiles-that-share-the-same-id-and-write-them-into-new-document/87226 "2017-05-26T10:01:21Z")
Hello, I have several log files that share the same id. I want to extract the message part out of them and write these in a new document so you can see the "progress" in one document. I tried the aggregate filter and … --- ## [Logstash split log file](https://discuss.elastic.co/t/logstash-split-log-file/87214)
**Author:** [@JIELOGAN](https://discuss.elastic.co/u/JIELOGAN)\ **Replies:** 0\ **Last updated:** [May 26, 2017, 8:08am UTC](https://discuss.elastic.co/t/logstash-split-log-file/87214 "2017-05-26T08:08:19Z")
hello,i could not parse my data use logstash ,please help Here is my data You can see lots of "#",I want split data by different display,how can i achieve this ? please give me some advice ,thankyou --- ## [\[WARN \]\[logstash.outputs.elasticsearch\] Failed action. {:status=\>400, :action=\>\["index",](https://discuss.elastic.co/t/warn-logstash-outputs-elasticsearch-failed-action-status-400-action-index/85175)
**Author:** [@shaonbean](https://discuss.elastic.co/u/shaonbean)\ **Replies:** 3\ **Last updated:** [May 26, 2017, 7:31am UTC](https://discuss.elastic.co/t/warn-logstash-outputs-elasticsearch-failed-action-status-400-action-index/85175 "2017-05-26T07:31:43Z")
logstash error \[2017-05-10T08:50:56,328\]\[WARN \]\[logstash.outputs.elasticsearch\] Failed action. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"filebeat-aniutv-access-2017.05.10", :\_type=\>"aniutv-access", :\_routin… --- ## [Add field not fetching the value from target, instead printing as a string](https://discuss.elastic.co/t/add-field-not-fetching-the-value-from-target-instead-printing-as-a-string/87209)
**Author:** [@Gopi\_K1](https://discuss.elastic.co/u/Gopi_K1)\ **Replies:** 0\ **Last updated:** [May 26, 2017, 7:00am UTC](https://discuss.elastic.co/t/add-field-not-fetching-the-value-from-target-instead-printing-as-a-string/87209 "2017-05-26T07:00:34Z")
my conf: \[root@ci-000c294400e8 conf.d\]# cat sample\_test.conf input { file { codec =\> multiline { pattern =\> '^\\s\\s\\s\\s}' negate =\> true what =\>… --- ## [How to set timezone？logstash version 5.x](https://discuss.elastic.co/t/how-to-set-timezone-logstash-version-5-x/87193)
**Author:** [@Swime](https://discuss.elastic.co/u/Swime)\ **Replies:** 1\ **Last updated:** [May 26, 2017, 1:36am UTC](https://discuss.elastic.co/t/how-to-set-timezone-logstash-version-5-x/87193 "2017-05-26T01:36:07Z")
Hello, resently I use logstash 5.4 and try to change the timezone like the way bellow.But it does not work. Is that any fault ? And how can I set the timezone? Thx,please. filter{ mutate{ add\_field =\> \["time",… --- ## [Logstash Lumberjack uses insecure cipher](https://discuss.elastic.co/t/logstash-lumberjack-uses-insecure-cipher/86987)
**Author:** [@fmhwong](https://discuss.elastic.co/u/fmhwong)\ **Replies:** 0\ **Last updated:** [May 24, 2017, 1:49pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-uses-insecure-cipher/86987 "2017-05-24T13:49:45Z")
We use Lumberjack as input in Logstash 2.4. It looks like it requires 3DES cipher. If 3DES\_EDE\_CBC and DESede are disabled on the JRE running Logstash, Logstash could not be started. If 3DES\_EDE\_CBC and DESede are dis… --- ## [Geoip index mapping defaults to using logstash-\*](https://discuss.elastic.co/t/geoip-index-mapping-defaults-to-using-logstash/58981)
**Author:** [@christopher\_pax](https://discuss.elastic.co/u/christopher_pax)\ **Replies:** 7\ **Last updated:** [May 25, 2017, 9:39pm UTC](https://discuss.elastic.co/t/geoip-index-mapping-defaults-to-using-logstash/58981 "2017-05-25T21:39:48Z")
It seemes that when useing the geoip filter. It creates a mapping to convert the location to a geo\_point. But only for index's called logstash-\*. But I named my index's apache-\*. I have been trying to utilize the "temp… --- ## [Logstash exception in pipeline worker](https://discuss.elastic.co/t/logstash-exception-in-pipeline-worker/87004)
**Author:** [@Singard123](https://discuss.elastic.co/u/Singard123)\ **Replies:** 3\ **Last updated:** [May 25, 2017, 7:19pm UTC](https://discuss.elastic.co/t/logstash-exception-in-pipeline-worker/87004 "2017-05-25T19:19:33Z")
I'm getting this error when I publish the events in filebeat: 2017/05/24 15:20:00.468812 sync.go:85: ERR Failed to publish events caused by: read tcp 10.0.3.150:56617-\>167.114.251.27:19130: i/o timeout 2017/05/24 15:2… --- ## [Issue with Index filtering within Logstash](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150)
**Author:** [@kmroz](https://discuss.elastic.co/u/kmroz)\ **Replies:** 3\ **Last updated:** [May 25, 2017, 7:14pm UTC](https://discuss.elastic.co/t/issue-with-index-filtering-within-logstash/87150 "2017-05-25T19:14:19Z")
Hello, we are currently using 5.3 of Elastic stack. We have two different teams using logstash so we have two different logstash conf files. One file is using UDP input with port 4558 and the other is using beats input w… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=975) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=977)