# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=253 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 254 --- ## [Azure Activity Logs in Elastic](https://discuss.elastic.co/t/azure-activity-logs-in-elastic/264856)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 4\ **Last updated:** [February 22, 2021, 11:28am UTC](https://discuss.elastic.co/t/azure-activity-logs-in-elastic/264856 "2021-02-22T11:28:59Z")
Hi, currently we are tryting to ship logs from azure eventhub to logstash. We use this: Azure Event Hubs plugin | Logstash Reference \[7.11\] | Elastic to forward the logs to elastic cluster. As far as I have learnt from… --- ## [Can Logstash accepts GraphQL data and convert into json?](https://discuss.elastic.co/t/can-logstash-accepts-graphql-data-and-convert-into-json/265055)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 0\ **Last updated:** [February 22, 2021, 11:00am UTC](https://discuss.elastic.co/t/can-logstash-accepts-graphql-data-and-convert-into-json/265055 "2021-02-22T11:00:48Z")
I have one endpoint giving data in GarphQL, is it possible to add a codec filter to parse data from GraphQL to JSON ? or is there any way I can covert GraphQL data to Elasticsearch readable format? --- ## [Logstash input plugin jdbc schedule issue on k8s](https://discuss.elastic.co/t/logstash-input-plugin-jdbc-schedule-issue-on-k8s/263535)
**Author:** [@Dongsheng\_XI](https://discuss.elastic.co/u/Dongsheng_XI)\ **Replies:** 1\ **Last updated:** [February 22, 2021, 9:01am UTC](https://discuss.elastic.co/t/logstash-input-plugin-jdbc-schedule-issue-on-k8s/263535 "2021-02-22T09:01:14Z")
our team is using logstash to sync data between MySQL and ElasticSearch, I am following How to keep Elasticsearch synchronized with a relational database using Logstash and JDBC to set up my pipeline. when the logstash i… --- ## [Logstash conditional output](https://discuss.elastic.co/t/logstash-conditional-output/264826)
**Author:** [@petegriggs](https://discuss.elastic.co/u/petegriggs)\ **Replies:** 6\ **Last updated:** [February 22, 2021, 12:54am UTC](https://discuss.elastic.co/t/logstash-conditional-output/264826 "2021-02-22T00:54:11Z")
Hello, Looking to use logstash to host multiple syslog listeners to start to aggregate logs from different vendors and then forward on for now to another syslog server. The question is whether we can set the syslog out… --- ## [Why does it have duplicate ID with normal logstash output and fingerprint filter?](https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931)
**Author:** [@Bo\_Pham\_Nguyen](https://discuss.elastic.co/u/Bo_Pham_Nguyen)\ **Replies:** 1\ **Last updated:** [February 21, 2021, 9:47pm UTC](https://discuss.elastic.co/t/why-does-it-have-duplicate-id-with-normal-logstash-output-and-fingerprint-filter/264931 "2021-02-21T21:47:43Z")
Hi team, This is my logstash config to get data, and I use fingerprint as \_id in order to prevent duplication from data. But it still has duplicated event as below. Any help? filter { if \[type\] == "cve" { … --- ## [New to Elastic/LogStash and have a few questions](https://discuss.elastic.co/t/new-to-elastic-logstash-and-have-a-few-questions/264322)
**Author:** [@Skatman](https://discuss.elastic.co/u/Skatman)\ **Replies:** 6\ **Last updated:** [February 21, 2021, 5:37pm UTC](https://discuss.elastic.co/t/new-to-elastic-logstash-and-have-a-few-questions/264322 "2021-02-21T17:37:44Z")
Hi all, Just started using Elastic Stack after hearing rave reviews from friends in the security industry. I used the free trial thing just to have a prod around the interface before setting my own instance up. Got a f… --- ## [HTTP request failure on using the logstash-filter-http](https://discuss.elastic.co/t/http-request-failure-on-using-the-logstash-filter-http/264985)
**Author:** [@pratz](https://discuss.elastic.co/u/pratz)\ **Replies:** 0\ **Last updated:** [February 21, 2021, 2:11pm UTC](https://discuss.elastic.co/t/http-request-failure-on-using-the-logstash-filter-http/264985 "2021-02-21T14:11:45Z")
I'm using http filter to fetch some json data from a server. Below is my conf file input { file { path =\> "/opt/data/dum.txt" start\_position =\> "beginning" } } filter { http { url =\> "ht… --- ## [How to combine the complete data in a csv/txt file in to a single column in message field](https://discuss.elastic.co/t/how-to-combine-the-complete-data-in-a-csv-txt-file-in-to-a-single-column-in-message-field/264550)
**Author:** [@nikhilesh](https://discuss.elastic.co/u/nikhilesh)\ **Replies:** 4\ **Last updated:** [February 21, 2021, 2:16pm UTC](https://discuss.elastic.co/t/how-to-combine-the-complete-data-in-a-csv-txt-file-in-to-a-single-column-in-message-field/264550 "2021-02-21T14:16:07Z")
Hi all, I have a below kind of data in the file of csv format. when I try to pull the logs to kibana. its showing as a multiple lines in the kibana console(PFA) I want the complete data in the file to come as a single … --- ## [Unrecognized VM option 'UseConcMarkSweepGC'](https://discuss.elastic.co/t/unrecognized-vm-option-useconcmarksweepgc/264960)
**Author:** [@realtech2338](https://discuss.elastic.co/u/realtech2338)\ **Replies:** 10\ **Last updated:** [February 20, 2021, 11:26pm UTC](https://discuss.elastic.co/t/unrecognized-vm-option-useconcmarksweepgc/264960 "2021-02-20T23:26:55Z")
C:\\elastic\_stack\\logstash-7.11.1-windows-x86\_64\\logstash-7.11.1\\bin\>logstash -f logstash-simple.conf Using JAVA\_HOME defined java: C:\\Program Files\\Java\\jdk-15.0.2 WARNING, using JAVA\_HOME while Logstash distribution c… --- ## [How to use Ruby to preprocess k,v -pairs value](https://discuss.elastic.co/t/how-to-use-ruby-to-preprocess-k-v-pairs-value/263684)
**Author:** [@GL\_Choong](https://discuss.elastic.co/u/GL_Choong)\ **Replies:** 5\ **Last updated:** [February 20, 2021, 8:46pm UTC](https://discuss.elastic.co/t/how-to-use-ruby-to-preprocess-k-v-pairs-value/263684 "2021-02-20T20:46:07Z")
Hi, I\`m new to Logstash and Ruby. Would like to get help. Currently I have a data as below test:result testcase: #TEMP\_BTWLAN:ok; loop:24;Temperature:28; My initial target is by using ruby to generate as below Outpu… --- ## [How to get rid of nested field and other issues](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945)
**Author:** [@jkost](https://discuss.elastic.co/u/jkost)\ **Replies:** 6\ **Last updated:** [February 20, 2021, 6:17pm UTC](https://discuss.elastic.co/t/how-to-get-rid-of-nested-field-and-other-issues/264945 "2021-02-20T18:17:25Z")
Hallo, I 'm using logstash 7.11 and my data have the following format: { field1: ..., field2: ..., aList: \[ { Id ... }, { fielda1 ... }, { fielda2 ... }, { fielda2 ... }, … --- ## [Unable to replace my log\_timestamp with @timestamp](https://discuss.elastic.co/t/unable-to-replace-my-log-timestamp-with-timestamp/264796)
**Author:** [@sanath\_ind](https://discuss.elastic.co/u/sanath_ind)\ **Replies:** 4\ **Last updated:** [February 20, 2021, 10:08am UTC](https://discuss.elastic.co/t/unable-to-replace-my-log-timestamp-with-timestamp/264796 "2021-02-20T10:08:07Z")
Hi, I am facing an issue while trying to set my log\_timestamp to @timestamp. I have tried many combinations from various date filter topics from here. But it is not giving me any positive result. My original logstash.c… --- ## [Grok Filter on Spring Logs](https://discuss.elastic.co/t/grok-filter-on-spring-logs/264930)
**Author:** [@seshu\_rao](https://discuss.elastic.co/u/seshu_rao)\ **Replies:** 0\ **Last updated:** [February 20, 2021, 2:57am UTC](https://discuss.elastic.co/t/grok-filter-on-spring-logs/264930 "2021-02-20T02:57:03Z")
i was trying to apply grok pattern on below log and couldn't make it successfully. some one please help me on this. this is the log file pattern coming to kibana. {"@timestamp":"2021-02-19T10:27:42.275+00:00","severity… --- ## [Skip to end of filter if the grok pattern does not match](https://discuss.elastic.co/t/skip-to-end-of-filter-if-the-grok-pattern-does-not-match/264925)
**Author:** [@newmember](https://discuss.elastic.co/u/newmember)\ **Replies:** 2\ **Last updated:** [February 20, 2021, 2:50am UTC](https://discuss.elastic.co/t/skip-to-end-of-filter-if-the-grok-pattern-does-not-match/264925 "2021-02-20T02:50:21Z")
Is there a option for a filter so that when the grok pattern does not match skip all the other filter plugins and leave the filter? Ideal situation is if the grok pattern does not match skip the rest of the filter plu… --- ## [Config file is not processed by logstash](https://discuss.elastic.co/t/config-file-is-not-processed-by-logstash/264903)
**Author:** [@Antonis\_Michael](https://discuss.elastic.co/u/Antonis_Michael)\ **Replies:** 15\ **Last updated:** [February 20, 2021, 1:13am UTC](https://discuss.elastic.co/t/config-file-is-not-processed-by-logstash/264903 "2021-02-20T01:13:15Z")
Hello, I am having my pipeline set to look for .conf files under /etc/logstash/conf.d/, there i have multiple config files and I make sure to tag each one for filtering indices. Everything works fine for all the other c… --- ## [Logstash problems with xml filter](https://discuss.elastic.co/t/logstash-problems-with-xml-filter/264667)
**Author:** [@Er1csson](https://discuss.elastic.co/u/Er1csson)\ **Replies:** 8\ **Last updated:** [February 19, 2021, 11:15pm UTC](https://discuss.elastic.co/t/logstash-problems-with-xml-filter/264667 "2021-02-19T23:15:26Z")
Hi all, I want to collect only certain tags from my xml file (data with Parameter tag). Here is the example of xml file I want to parse: // \ // \ **Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [February 19, 2021, 6:55pm UTC](https://discuss.elastic.co/t/date-parse-fail/264483 "2021-02-19T18:55:00Z") Trying to ingest the below event: {"modified": "2019-12-27 16:08:00", "published": "2015-03-08 02:59:00", "access": {"authentication": "none", "complexity": "low", "vector": "network"}, "assigner": "cve@mitre.org", "cap… --- ## [Updating the geoLite2 Database](https://discuss.elastic.co/t/updating-the-geolite2-database/264722)
**Author:** [@Falikou1](https://discuss.elastic.co/u/Falikou1)\ **Replies:** 6\ **Last updated:** [February 19, 2021, 6:42pm UTC](https://discuss.elastic.co/t/updating-the-geolite2-database/264722 "2021-02-19T18:42:30Z")
I have configured the geoLite2 Database in logstash. geoip { default\_database\_type =\> 'City' database =\> '/usr/share/logstash/GeoLite2-City/GeoLite2-City.mmdb' source =\> "src" target =\> "src\_geoip" } Is there … --- ## [Logstash events count and get alert](https://discuss.elastic.co/t/logstash-events-count-and-get-alert/264850)
**Author:** [@bdn](https://discuss.elastic.co/u/bdn)\ **Replies:** 2\ **Last updated:** [February 19, 2021, 6:16pm UTC](https://discuss.elastic.co/t/logstash-events-count-and-get-alert/264850 "2021-02-19T18:16:51Z")
I want to count events in Logstash and get me alert if events hit more than 10 times within 2mins. E.g. I have a log like below. Feb 19 15:22:01 DEVICE-01 Device is online. Feb 19 15:22:01 DEVICE-02 Device is online. F… --- ## [Logstash 7.9.1 file input plugin is not able to read json file](https://discuss.elastic.co/t/logstash-7-9-1-file-input-plugin-is-not-able-to-read-json-file/264548)
**Author:** [@sagarpatel](https://discuss.elastic.co/u/sagarpatel)\ **Replies:** 6\ **Last updated:** [February 19, 2021, 3:25pm UTC](https://discuss.elastic.co/t/logstash-7-9-1-file-input-plugin-is-not-able-to-read-json-file/264548 "2021-02-19T15:25:40Z")
I have created sample.conf file for reading json file but it is not showing any output. below is my sample.cong file input { file { path =\> "C:\\Users\\Sagar\\sample.json" start\_position =\> "beginning" codec =\> "js… --- ## [Logstash grok multiple pattern , multi-line](https://discuss.elastic.co/t/logstash-grok-multiple-pattern-multi-line/264860)
**Author:** [@Daniel\_Jankech](https://discuss.elastic.co/u/Daniel_Jankech)\ **Replies:** 1\ **Last updated:** [February 19, 2021, 1:30pm UTC](https://discuss.elastic.co/t/logstash-grok-multiple-pattern-multi-line/264860 "2021-02-19T13:30:01Z")
Hello everyone , Im using grok to parse log file consisting of multiple pattern lines , these multiple lines represent one task being done in the system. My question is how should I do this if I need to add fields to the… --- ## [Jdbc connection to MS-SQL server with windows credentials](https://discuss.elastic.co/t/jdbc-connection-to-ms-sql-server-with-windows-credentials/264542)
**Author:** [@ravnen.flyver](https://discuss.elastic.co/u/ravnen.flyver)\ **Replies:** 1\ **Last updated:** [February 19, 2021, 1:10pm UTC](https://discuss.elastic.co/t/jdbc-connection-to-ms-sql-server-with-windows-credentials/264542 "2021-02-19T13:10:29Z")
I am sitting on a WindownServer(2019) where I connect to a Windown sql 2016 server. On my WindownServer I can connect to the SQL server through MS SQL Management Studio, so firewall and all that is ok. And my Windown AD… --- ## [How to create multiple indexs with multiple input in logstash](https://discuss.elastic.co/t/how-to-create-multiple-indexs-with-multiple-input-in-logstash/264416)
**Author:** [@Falikou1](https://discuss.elastic.co/u/Falikou1)\ **Replies:** 7\ **Last updated:** [February 19, 2021, 9:06am UTC](https://discuss.elastic.co/t/how-to-create-multiple-indexs-with-multiple-input-in-logstash/264416 "2021-02-19T09:06:02Z")
I would like to have some help to configure multiple indexes from multiple entries with logstash. Is my configuration below correct? input { tcp { port =\> "5140" codec =\> json type =\> "syslog" } tcp { port =\> "5… --- ## [Logstash adding unwanted "" & \\\\n to json input](https://discuss.elastic.co/t/logstash-adding-unwanted-n-to-json-input/264786)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [February 19, 2021, 4:48am UTC](https://discuss.elastic.co/t/logstash-adding-unwanted-n-to-json-input/264786 "2021-02-19T04:48:39Z")
Hi, I have a log that I converted from plain text to json with JQ. I want to ingest this file into Elasticsearch as is. If I copy the text below add send it to Elasticsearch manually it shows up properly formatted etc. … --- ## [Logstash error on Raspberry Pi 4 running Ubuntu Server 20.04.2](https://discuss.elastic.co/t/logstash-error-on-raspberry-pi-4-running-ubuntu-server-20-04-2/264202)
**Author:** [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Replies:** 2\ **Last updated:** [February 19, 2021, 2:46am UTC](https://discuss.elastic.co/t/logstash-error-on-raspberry-pi-4-running-ubuntu-server-20-04-2/264202 "2021-02-19T02:46:40Z")
Hello, I'm running 64-bit version of Ubuntu Server 20.04.2 on a Raspberry Pi 4 (4 GB RAM, hardware revision 1.1) I've installed logstash using apt. Configuration being used on this Raspberry Pi work on a different boar… --- ## [\[.monitoring-logstash\] Pipeline terminated {"pipeline.id"=\>".monitoring-logstash"}](https://discuss.elastic.co/t/monitoring-logstash-pipeline-terminated-pipeline-id-monitoring-logstash/264619)
**Author:** [@grillo](https://discuss.elastic.co/u/grillo)\ **Replies:** 4\ **Last updated:** [February 18, 2021, 7:47pm UTC](https://discuss.elastic.co/t/monitoring-logstash-pipeline-terminated-pipeline-id-monitoring-logstash/264619 "2021-02-18T19:47:30Z")
Hello everyone, I have a cluster of four nodes running - secured with certificates. I have followed the tutorials to include a logstash to the cluster. At the moment I don't even use /conf.d/\* configuration files to all… --- ## [Split string into array and then into key value pairs](https://discuss.elastic.co/t/split-string-into-array-and-then-into-key-value-pairs/264741)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 2\ **Last updated:** [February 18, 2021, 7:06pm UTC](https://discuss.elastic.co/t/split-string-into-array-and-then-into-key-value-pairs/264741 "2021-02-18T19:06:18Z")
Is there a way to split a string into an array and then turn each of those values into key value pairs within the array using logstash. For example: Turning this: SomeField -\> "key1:value1;key2:value2;key3:value3" In… --- ## [Elasticsearch output - ILM availablity not properly detected with expired ES license](https://discuss.elastic.co/t/elasticsearch-output-ilm-availablity-not-properly-detected-with-expired-es-license/264709)
**Author:** [@SH\_HSOC](https://discuss.elastic.co/u/SH_HSOC)\ **Replies:** 3\ **Last updated:** [February 18, 2021, 6:11pm UTC](https://discuss.elastic.co/t/elasticsearch-output-ilm-availablity-not-properly-detected-with-expired-es-license/264709 "2021-02-18T18:11:50Z")
Hi community, I think I discovered a corner case bug in the elasticsearch output plugin. Currently the license of my elasticsearch cluster has expired and it feels like the ILM availability is no longer properly detecte… --- ## [Multiple vs one filter for different inputs](https://discuss.elastic.co/t/multiple-vs-one-filter-for-different-inputs/264696)
**Author:** [@sastorsl](https://discuss.elastic.co/u/sastorsl)\ **Replies:** 1\ **Last updated:** [February 18, 2021, 5:27pm UTC](https://discuss.elastic.co/t/multiple-vs-one-filter-for-different-inputs/264696 "2021-02-18T17:27:17Z")
In logstash we have multiple sources of input data - so we use conditionals on a document field (doc\_type) to apply the various filters. Is there any real difference or benefit between having one filter with conditional… --- ## [Fargate with logstash container consume iam role](https://discuss.elastic.co/t/fargate-with-logstash-container-consume-iam-role/264671)
**Author:** [@J\_Kit](https://discuss.elastic.co/u/J_Kit)\ **Replies:** 0\ **Last updated:** [February 18, 2021, 9:29am UTC](https://discuss.elastic.co/t/fargate-with-logstash-container-consume-iam-role/264671 "2021-02-18T09:29:56Z")
Hi I tried to set up logstash in AWS ECS fargate and the logstash will read from the s3 bucket. I configuring logstash input s3 with access\_key\_id =\> ''" secret\_access\_key =\> ''" and able to read s3 files successfully. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=252) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=254)