# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=393 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 394 --- ## [Removing agent.\* metadata doesn't work in Filebet nor Logstash](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314)
**Author:** [@ehsank777](https://discuss.elastic.co/u/ehsank777)\ **Replies:** 4\ **Last updated:** [March 9, 2020, 2:27pm UTC](https://discuss.elastic.co/t/removing-agent-metadata-doesnt-work-in-filebet-nor-logstash/222314 "2020-03-09T14:27:31Z")
Hi, I have Elasticsearch and Logstash version 7.5.0 installed in an ELK stack. So far I was using filebeat version 6.8.0 on my machines. Today I upgraded some of them to version 7.6.0 and after that I see the following … --- ## [LogStash-Outputs -ElasticSearch - URI is not valid - host is not specified](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-uri-is-not-valid-host-is-not-specified/222664)
**Author:** [@Austin\_Lai](https://discuss.elastic.co/u/Austin_Lai)\ **Replies:** 1\ **Last updated:** [March 9, 2020, 2:16pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-uri-is-not-valid-host-is-not-specified/222664 "2020-03-09T14:16:33Z")
Anyone solved the issue of below error message?? logstash7.6.1 | \[2020-03-08T06:49:05,409\]\[DEBUG\]\[logstash.plugins.registry\] On demand adding plugin to the registry {:name=\>"elasticsearch", :type=\>"output", :class=\>LogSt… --- ## [Logstash not appearing in kibana index](https://discuss.elastic.co/t/logstash-not-appearing-in-kibana-index/222721)
**Author:** [@oumy](https://discuss.elastic.co/u/oumy)\ **Replies:** 0\ **Last updated:** [March 9, 2020, 2:02pm UTC](https://discuss.elastic.co/t/logstash-not-appearing-in-kibana-index/222721 "2020-03-09T14:02:56Z")
Hello there, i can't get logstash to appear on the kibana index, even though it exist in localhost indices. --- ## [Empty fields appear in InfluwDB filled with the variable name, i.e. %{\[kubernetes\_node\_name\]}](https://discuss.elastic.co/t/empty-fields-appear-in-influwdb-filled-with-the-variable-name-i-e-kubernetes-node-name/222718)
**Author:** [@jeanflorentine](https://discuss.elastic.co/u/jeanflorentine)\ **Replies:** 0\ **Last updated:** [March 9, 2020, 1:55pm UTC](https://discuss.elastic.co/t/empty-fields-appear-in-influwdb-filled-with-the-variable-name-i-e-kubernetes-node-name/222718 "2020-03-09T13:55:44Z")
In InfluxDB output plugin configured with: data\_points =\> { "\_kubernetes\_namespace" =\> "%{\[kubernetes\_namespace\]}" "\_field\_env" =\> "%{\[field\_env\]}" "\_event\_dataset" =\> "%{\[event\_datas… --- ## [Logstash failing at output stdout](https://discuss.elastic.co/t/logstash-failing-at-output-stdout/222483)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 2\ **Last updated:** [March 9, 2020, 1:04pm UTC](https://discuss.elastic.co/t/logstash-failing-at-output-stdout/222483 "2020-03-09T13:04:24Z")
I'm having trouble testing a grok filter. I've been able to run a very basic test using the file input and then stdout output. All I'm trying to do is run a test with a grok filter that takes a file (oneline.txt), util… --- ## [How to create multiple Indices of elasticsearch using logstash configuration](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537)
**Author:** [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)\ **Replies:** 14\ **Last updated:** [March 9, 2020, 12:16pm UTC](https://discuss.elastic.co/t/how-to-create-multiple-indices-of-elasticsearch-using-logstash-configuration/222537 "2020-03-09T12:16:57Z")
Hello everyone, I am trying to create different indices of elasticsearch using logstash but i am not able to do this.can anyone help me out with this.How we can create and send multiple different index to elasticsearc… --- ## [Logstash - Multiple Logs](https://discuss.elastic.co/t/logstash-multiple-logs/220600)
**Author:** [@santos1204](https://discuss.elastic.co/u/santos1204)\ **Replies:** 3\ **Last updated:** [March 9, 2020, 11:06am UTC](https://discuss.elastic.co/t/logstash-multiple-logs/220600 "2020-03-09T11:06:21Z")
Hi, I'm just starting to play with ELK, we're looking at shipping different logs with different filters from each server to a centralised place. filebeat config; filebeat.inputs: Each - is an input. Most options can … --- ## [Logstash JDBC to MSSQL - this driver is not configured for integrated authentication](https://discuss.elastic.co/t/logstash-jdbc-to-mssql-this-driver-is-not-configured-for-integrated-authentication/222674)
**Author:** [@Mattness](https://discuss.elastic.co/u/Mattness)\ **Replies:** 2\ **Last updated:** [March 9, 2020, 10:21am UTC](https://discuss.elastic.co/t/logstash-jdbc-to-mssql-this-driver-is-not-configured-for-integrated-authentication/222674 "2020-03-09T10:21:22Z")
Hello, I am trying to connect logstash to my sql server 2019. My .conf file: input { jdbc { jdbc\_driver\_library =\> "C:/sqljdbc\_8.2/enu/mssql-jdbc-8.2.0.jre13.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.… --- ## [Import Multiple Files CSV - Aggregate Data](https://discuss.elastic.co/t/import-multiple-files-csv-aggregate-data/222612)
**Author:** [@brunojpsantos](https://discuss.elastic.co/u/brunojpsantos)\ **Replies:** 2\ **Last updated:** [March 9, 2020, 9:23am UTC](https://discuss.elastic.co/t/import-multiple-files-csv-aggregate-data/222612 "2020-03-09T09:23:51Z")
Hi, I'm working on a project that I need to import 2 csv files, rename/modify that entries and agregate that 2 files... For example, I have a "file1" that contains the columns : id, name, state And I have a "file2" wi… --- ## [Logstash Beats port not opening](https://discuss.elastic.co/t/logstash-beats-port-not-opening/222663)
**Author:** [@tbhaxor](https://discuss.elastic.co/u/tbhaxor)\ **Replies:** 0\ **Last updated:** [March 9, 2020, 8:04am UTC](https://discuss.elastic.co/t/logstash-beats-port-not-opening/222663 "2020-03-09T08:04:34Z")
I have set up logstash to receive the beats data. But unfortunately only 9600 port is opening beats config input { beats { port =\> 5044 } } output { elasticsearch { hosts =\> \["https://localhost:9200"\] … --- ## [\[FATAL\] 2020-03-09 05:19:25.972 \[LogStash::Runner\] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting](https://discuss.elastic.co/t/fatal-2020-03-09-0525-972-logstash-runner-runner-logstash-could-not-be-started-because-there-is-already-another-instance-using-the-configured-data-directory-if-you-wish-to-run-multiple-instances-you-must-change-the-path-data-setting/222645)
**Author:** [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)\ **Replies:** 2\ **Last updated:** [March 9, 2020, 6:44am UTC](https://discuss.elastic.co/t/fatal-2020-03-09-0525-972-logstash-runner-runner-logstash-could-not-be-started-because-there-is-already-another-instance-using-the-configured-data-directory-if-you-wish-to-run-multiple-instances-you-must-change-the-path-data-setting/222645 "2020-03-09T06:44:56Z")
I created two different droplets in Digitalocen and install nginx and filebeat on those two droplets.I configured my filebeat.yml file as below: VM1 filebeat.yml:\_ filebeat.prospectors: type: log paths: /var/log/ng… --- ## [How can i combine log and database data with the common field](https://discuss.elastic.co/t/how-can-i-combine-log-and-database-data-with-the-common-field/222642)
**Author:** [@saravana\_hariharan](https://discuss.elastic.co/u/saravana_hariharan)\ **Replies:** 0\ **Last updated:** [March 9, 2020, 5:28am UTC](https://discuss.elastic.co/t/how-can-i-combine-log-and-database-data-with-the-common-field/222642 "2020-03-09T05:28:32Z")
I would like to join/combine log and database data with the common filed in a same index pattern . is it possible in ELK? Pls Help @magnusbaeck --- ## [Extracting a new field from the message field](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407)
**Author:** [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Replies:** 8\ **Last updated:** [March 9, 2020, 3:51am UTC](https://discuss.elastic.co/t/extracting-a-new-field-from-the-message-field/222407 "2020-03-09T03:51:33Z")
2020-03-06 09:36:52.801907464 re0:ndp:25786 lltp\_debug message = "NDP-DBG:NC\_FSM\_HANDLER:1634:: state 7, event 4, NexthopId 44001" This is my log. I need to extract state and NexthopId from the message field match =\> … --- ## [Using a variable as an index of an array of patterns](https://discuss.elastic.co/t/using-a-variable-as-an-index-of-an-array-of-patterns/222617)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 2\ **Last updated:** [March 8, 2020, 8:39pm UTC](https://discuss.elastic.co/t/using-a-variable-as-an-index-of-an-array-of-patterns/222617 "2020-03-08T20:39:12Z")
I have this filter in a logstash pipeline filter{ ruby { code =\> 'event.set("patternmatch1", event.get("message").scan(/TEXT.\*?(\\w\[\\w,.\]\*)\\r?$/))' } # Retrieve the lenght of the array cointaining all the ma… --- ## [Logstash filter semantic fields not loading in Kibana](https://discuss.elastic.co/t/logstash-filter-semantic-fields-not-loading-in-kibana/222619)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 0\ **Last updated:** [March 8, 2020, 8:01pm UTC](https://discuss.elastic.co/t/logstash-filter-semantic-fields-not-loading-in-kibana/222619 "2020-03-08T20:01:21Z")
I thought my knowledge of the ELK stack was pretty good but this weekend I have tried and tried to figure out what is going on with my deployment. The result that I am not getting is that I create my filters in logstash… --- ## [How to flatten the json, not working with previous post](https://discuss.elastic.co/t/how-to-flatten-the-json-not-working-with-previous-post/222608)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 6\ **Last updated:** [March 8, 2020, 6:38pm UTC](https://discuss.elastic.co/t/how-to-flatten-the-json-not-working-with-previous-post/222608 "2020-03-08T18:38:52Z")
Hi All I have a case, wherein i need to flatten the json data.To get started i am just following the previous post. But i am not successful with the result I am getting below result { "@timestamp" : "2020-03-08T15… --- ## [Feed file input to an ES query](https://discuss.elastic.co/t/feed-file-input-to-an-es-query/222598)
**Author:** [@Sineos](https://discuss.elastic.co/u/Sineos)\ **Replies:** 2\ **Last updated:** [March 8, 2020, 5:53pm UTC](https://discuss.elastic.co/t/feed-file-input-to-an-es-query/222598 "2020-03-08T17:53:47Z")
Dear all, I'm trying to achieve the following: Read a file: One value per line Transform this value into a wildcard value, e.g. "\*value\*" Feed the value into an ES query Return the results of the query These are my s… --- ## [How to get all the source from json that indexed with logstash](https://discuss.elastic.co/t/how-to-get-all-the-source-from-json-that-indexed-with-logstash/222590)
**Author:** [@Bahadir\_Eyuboglu](https://discuss.elastic.co/u/Bahadir_Eyuboglu)\ **Replies:** 0\ **Last updated:** [March 8, 2020, 8:26am UTC](https://discuss.elastic.co/t/how-to-get-all-the-source-from-json-that-indexed-with-logstash/222590 "2020-03-08T08:26:19Z")
I'm newbie and i need index large json data to elasticsearch so i use logstash to index my json files in some directory here is the logstash config: \< input { file { path =\> "/home/abahadir/Desktop/Proje/searchflask/… --- ## [How to make dynamic document id from name of the json file](https://discuss.elastic.co/t/how-to-make-dynamic-document-id-from-name-of-the-json-file/222577)
**Author:** [@Bahadir\_Eyuboglu](https://discuss.elastic.co/u/Bahadir_Eyuboglu)\ **Replies:** 3\ **Last updated:** [March 7, 2020, 10:57pm UTC](https://discuss.elastic.co/t/how-to-make-dynamic-document-id-from-name-of-the-json-file/222577 "2020-03-07T22:57:49Z")
I'm nooby when it's come to elasticsearch what i'm trying to do is, while indexing the large json file want to give it a document id same with file's name dummy.json document\_id =\> "dummy" kinda \< input { file { p… --- ## [How to get systemId field value using logstash-input-azure\_blob\_storage plugin?](https://discuss.elastic.co/t/how-to-get-systemid-field-value-using-logstash-input-azure-blob-storage-plugin/222573)
**Author:** [@ls09](https://discuss.elastic.co/u/ls09)\ **Replies:** 0\ **Last updated:** [March 7, 2020, 7:08pm UTC](https://discuss.elastic.co/t/how-to-get-systemid-field-value-using-logstash-input-azure-blob-storage-plugin/222573 "2020-03-07T19:08:13Z")
How to get systemId field value using logstash-input-azure\_blob\_storage? please HELP! The Azure NSG log format is described here: my logstash.conf file contents: input { azure\_blob\_storage { storageaccount =\> … --- ## [Logstash not parsing logs into Elasticsearch](https://discuss.elastic.co/t/logstash-not-parsing-logs-into-elasticsearch/222555)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 1\ **Last updated:** [March 7, 2020, 3:04pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-logs-into-elasticsearch/222555 "2020-03-07T15:04:31Z")
I have setup my paths to the logs files as stated in the documentation, the problem is when, for example; I turn on the nginx module the logs seems to be parsed and sent to elasticsearch. But when I turn off the nginx m… --- ## [Help me to remove json tag from the given json](https://discuss.elastic.co/t/help-me-to-remove-json-tag-from-the-given-json/222481)
**Author:** [@jahaber\_sadhik](https://discuss.elastic.co/u/jahaber_sadhik)\ **Replies:** 2\ **Last updated:** [March 7, 2020, 9:36am UTC](https://discuss.elastic.co/t/help-me-to-remove-json-tag-from-the-given-json/222481 "2020-03-07T09:36:32Z")
"@timestamp" =\> 2020-03-06T15:24:44.555Z, "@version" =\> "1", "host" =\> "worker02.stg.xxxxx.com", "path" =\> "/var/log/spark2/sparkBatchsla-file.log", "payload" =\> { "slaBatchRes" =\> { // Like to remove json tag "res… --- ## [Logstash tried to bind to port range 9600..9700, but all the ports are already in use. You can specify a new port by launching logstash with the --http.port option](https://discuss.elastic.co/t/logstash-tried-to-bind-to-port-range-9600-9700-but-all-the-ports-are-already-in-use-you-can-specify-a-new-port-by-launching-logstash-with-the-http-port-option/222540)
**Author:** [@nipin1509](https://discuss.elastic.co/u/nipin1509)\ **Replies:** 0\ **Last updated:** [March 7, 2020, 5:59am UTC](https://discuss.elastic.co/t/logstash-tried-to-bind-to-port-range-9600-9700-but-all-the-ports-are-already-in-use-you-can-specify-a-new-port-by-launching-logstash-with-the-http-port-option/222540 "2020-03-07T05:59:11Z")
I am getting " Logstash tried to bind to port range ' when installing logstash in openshift. Below is the Config map and Logstsh Yaml file. Logstash Config apiVersion: v1 kind: ConfigMap metadata: name: logstash-co… --- ## [Logstash with docker](https://discuss.elastic.co/t/logstash-with-docker/222501)
**Author:** [@alxfernando](https://discuss.elastic.co/u/alxfernando)\ **Replies:** 0\ **Last updated:** [March 6, 2020, 6:44pm UTC](https://discuss.elastic.co/t/logstash-with-docker/222501 "2020-03-06T18:44:01Z")
This error appears when I run the kibana docker-compose, elastisearch, logstash logstash | \[2020-03-06T18:41:21,278\]\[WARN \]\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance, … --- ## [Logstash-output-tcp fails to connect to ssl enabled flume endpoint](https://discuss.elastic.co/t/logstash-output-tcp-fails-to-connect-to-ssl-enabled-flume-endpoint/222497)
**Author:** [@vpayala](https://discuss.elastic.co/u/vpayala)\ **Replies:** 0\ **Last updated:** [March 6, 2020, 5:40pm UTC](https://discuss.elastic.co/t/logstash-output-tcp-fails-to-connect-to-ssl-enabled-flume-endpoint/222497 "2020-03-06T17:40:53Z")
Hello, I am new to Logstash and I am trying to setup Logstash output to stream raw application logs over to a Flume endpoint that is SSL enabled. I am hoping to achieve this using the logstash tcp output plugin. Here's … --- ## [Logstash Helm Chart sync elastic with database](https://discuss.elastic.co/t/logstash-helm-chart-sync-elastic-with-database/218812)
**Author:** [@m-garin](https://discuss.elastic.co/u/m-garin)\ **Replies:** 1\ **Last updated:** [March 6, 2020, 4:56pm UTC](https://discuss.elastic.co/t/logstash-helm-chart-sync-elastic-with-database/218812 "2020-03-06T16:56:27Z")
Hi! Anybody have .yalm sample file for sync elasticsearch with postgres using logstash? I cant install logstash-input-jdbc plugin. --- ## [Create Grok filter date](https://discuss.elastic.co/t/create-grok-filter-date/222285)
**Author:** [@dbviz](https://discuss.elastic.co/u/dbviz)\ **Replies:** 4\ **Last updated:** [March 6, 2020, 3:46pm UTC](https://discuss.elastic.co/t/create-grok-filter-date/222285 "2020-03-06T15:46:17Z")
Hi everyone, I tried to parse date from this line Wed Nov 27 07:31:40 CET 2019;User123;NEWPORT.WEBSERVICE.EXAMPLE like that in grok filter grok { match =\> {"message" =\> \["%{GREEDYDATA:date};%{USER:user};%{GRE… --- ## [Understanding differences in similar logstash-pipelines - metrics/logs to look for?](https://discuss.elastic.co/t/understanding-differences-in-similar-logstash-pipelines-metrics-logs-to-look-for/222473)
**Author:** [@Andre\_Zoufahl](https://discuss.elastic.co/u/Andre_Zoufahl)\ **Replies:** 0\ **Last updated:** [March 6, 2020, 2:27pm UTC](https://discuss.elastic.co/t/understanding-differences-in-similar-logstash-pipelines-metrics-logs-to-look-for/222473 "2020-03-06T14:27:44Z")
Hi all, I'm trying to get a better understanding and improve my debugging-skills when looking at logstash behaviour. Currently I'm having 2 almost identical logstash processes (logstash-oss 7.5.1). Both consume from t… --- ## [When i add this logstash filter for apache logs get following error](https://discuss.elastic.co/t/when-i-add-this-logstash-filter-for-apache-logs-get-following-error/221679)
**Author:** [@shahzaib123](https://discuss.elastic.co/u/shahzaib123)\ **Replies:** 3\ **Last updated:** [March 6, 2020, 1:56pm UTC](https://discuss.elastic.co/t/when-i-add-this-logstash-filter-for-apache-logs-get-following-error/221679 "2020-03-06T13:56:03Z")
I am facing a issue in logstash filter with grok filter my log data is: 192.168.1.200 - - \[02/Mar/2020:12:25:45 +0500\] "GET / HTTP/1.1" 200 472 "-" "Mozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko… --- ## [Modifying an arrays value by index](https://discuss.elastic.co/t/modifying-an-arrays-value-by-index/221045)
**Author:** [@Viktor\_Zsolnai](https://discuss.elastic.co/u/Viktor_Zsolnai)\ **Replies:** 4\ **Last updated:** [March 6, 2020, 1:39pm UTC](https://discuss.elastic.co/t/modifying-an-arrays-value-by-index/221045 "2020-03-06T13:39:12Z")
Hello, I'm pretty new to logstash, and I need to replace a value in an array, using a kv filter's value, but when I want to use the index variable nothing happens, if I use a simple integer instead, it works fine, here… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=392) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=394)