# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=228 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 229 --- ## [Mapping two string values or create kv pair](https://discuss.elastic.co/t/mapping-two-string-values-or-create-kv-pair/273518)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 11:38am UTC](https://discuss.elastic.co/t/mapping-two-string-values-or-create-kv-pair/273518 "2021-05-20T11:38:28Z")
my current pipeline in logstash after running on input log data is producing below output , i need to map it one to one..or create kv such as .. pl-3: 361 pl-4: 366 etc.... "header" =\> "Parameter … --- ## [Grok and curly brackets](https://discuss.elastic.co/t/grok-and-curly-brackets/273491)
**Author:** [@vaclav1](https://discuss.elastic.co/u/vaclav1)\ **Replies:** 1\ **Last updated:** [May 20, 2021, 10:36am UTC](https://discuss.elastic.co/t/grok-and-curly-brackets/273491 "2021-05-20T10:36:25Z")
Dear team, I have a grok filter which needs to parse following part of line from the Haproxy logs: {\*/\*||en-US|10.12.132.200:30012||||1997-04-09-17.02.03.246664|pclient\_us} Now how do I remove the brackets from beginn… --- ## [GROK: help with pattern](https://discuss.elastic.co/t/grok-help-with-pattern/273396)
**Author:** [@Guillermo\_Valles\_Cas](https://discuss.elastic.co/u/Guillermo_Valles_Cas)\ **Replies:** 3\ **Last updated:** [May 20, 2021, 9:18am UTC](https://discuss.elastic.co/t/grok-help-with-pattern/273396 "2021-05-20T09:18:38Z")
Hello!! I'm trying to get the pattern of a file in Logstash with GROK, but I do not know how to make it. The line I want to GROK is: {"type":"log","@timestamp":"2021-05-19T08:45:42+02:00","tags":\["info","plugins","actio… --- ## [Logstash unparsed events exceptions](https://discuss.elastic.co/t/logstash-unparsed-events-exceptions/273495)
**Author:** [@ima](https://discuss.elastic.co/u/ima)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 8:55am UTC](https://discuss.elastic.co/t/logstash-unparsed-events-exceptions/273495 "2021-05-20T08:55:10Z")
Hello, is there anybody who knows where I can find logstash events exceptions and what other healthchecks should I do for logstash --- ## [Event based data pushing](https://discuss.elastic.co/t/event-based-data-pushing/273478)
**Author:** [@Abhishek\_Chauhan](https://discuss.elastic.co/u/Abhishek_Chauhan)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 7:22am UTC](https://discuss.elastic.co/t/event-based-data-pushing/273478 "2021-05-20T07:22:14Z")
Hello guys, I have deployed logstash to push data from postgres to elasticsearch but as the quantity of data has increased, pushing thousands of rows(all data in that db) every 60 seconds just for a small change or addi… --- ## [Load more than 1 million row from SQL Server to ES with Logstash JDBC](https://discuss.elastic.co/t/load-more-than-1-million-row-from-sql-server-to-es-with-logstash-jdbc/273476)
**Author:** [@Dai\_Thai\_Hoa\_Vo](https://discuss.elastic.co/u/Dai_Thai_Hoa_Vo)\ **Replies:** 0\ **Last updated:** [May 20, 2021, 7:06am UTC](https://discuss.elastic.co/t/load-more-than-1-million-row-from-sql-server-to-es-with-logstash-jdbc/273476 "2021-05-20T07:06:36Z")
Hi every onne. I have a trouble when load the tables SalesTrans has more than 1 million rows. I used logstash to load them into ES. It works fine but very slow. Please any one support to me. Thank you. --- ## [Grok make log missing](https://discuss.elastic.co/t/grok-make-log-missing/273255)
**Author:** [@DawitCh](https://discuss.elastic.co/u/DawitCh)\ **Replies:** 4\ **Last updated:** [May 20, 2021, 6:53am UTC](https://discuss.elastic.co/t/grok-make-log-missing/273255 "2021-05-20T06:53:07Z")
Here is my example log \<134\>May 24 17:15:52 asdsgag.com 1,yyyy/mm/dd 17:15:52,001801056715,TRAFFIC,end,1,yyyy/mm/dd 17:15:52,xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx,0.0.0.0,0.0.0.0,TEST,,,incomplete,local1,wifi,office,cd5.32,rt… --- ## [Why does Logstash's https communication work without OpenSSL?](https://discuss.elastic.co/t/why-does-logstashs-https-communication-work-without-openssl/273460)
**Author:** [@TakaSeki](https://discuss.elastic.co/u/TakaSeki)\ **Replies:** 2\ **Last updated:** [May 20, 2021, 5:25am UTC](https://discuss.elastic.co/t/why-does-logstashs-https-communication-work-without-openssl/273460 "2021-05-20T05:25:19Z")
I'm running logstash using docker on CentOS7. Openssl is not installed on CentOS 7 and docker containers. However, logstash input https communication is working. Why is this? Does logstash include openssl functionali… --- ## [Azure App Services to ELK](https://discuss.elastic.co/t/azure-app-services-to-elk/273327)
**Author:** [@madmunki](https://discuss.elastic.co/u/madmunki)\ **Replies:** 15\ **Last updated:** [May 20, 2021, 4:31am UTC](https://discuss.elastic.co/t/azure-app-services-to-elk/273327 "2021-05-20T04:31:49Z")
I currently manage more than 50 Azure App Services and looking for a solution to collect the logs and put them where devs can look at the logs and troubleshoot issues. If I go into the App Log Stream, it displays what I … --- ## [Help with http http\_poller input plugin](https://discuss.elastic.co/t/help-with-http-http-poller-input-plugin/273226)
**Author:** [@jeromeat](https://discuss.elastic.co/u/jeromeat)\ **Replies:** 2\ **Last updated:** [May 20, 2021, 3:28am UTC](https://discuss.elastic.co/t/help-with-http-http-poller-input-plugin/273226 "2021-05-20T03:28:11Z")
I can pull json data using the following curl command successfully: curl -H "Api-Token: xxxxxxxxxx" -H "Api-Secret: xxxxxxxxxxxxxxx" https://site.cyz10.local:443/api/v1/tags?page=1&page\_size=10&type=ip Below is the con… --- ## [SSL certificate in email output plugin logstash](https://discuss.elastic.co/t/ssl-certificate-in-email-output-plugin-logstash/273450)
**Author:** [@jrojasp95](https://discuss.elastic.co/u/jrojasp95)\ **Replies:** 4\ **Last updated:** [May 19, 2021, 10:20pm UTC](https://discuss.elastic.co/t/ssl-certificate-in-email-output-plugin-logstash/273450 "2021-05-19T22:20:38Z")
Hi I am trying to use the email output plugin in logstash but I can't get it to work on the server with docker, on localhost it works with port 587 but on the server I get an OpenSSL hostname not match with ssl certifica… --- ## [\_grokparsefailure](https://discuss.elastic.co/t/grokparsefailure/273447)
**Author:** [@legolas\_bilbao](https://discuss.elastic.co/u/legolas_bilbao)\ **Replies:** 0\ **Last updated:** [May 19, 2021, 9:11pm UTC](https://discuss.elastic.co/t/grokparsefailure/273447 "2021-05-19T21:11:47Z")
Good night, I've use the dev tools for deploy the grook rule for squid logs The config file is the following input { file { path =\> "/var/elastik/access.log" start\_position =\> "beginning" } } filter { grok { … --- ## [This interval creates too many buckets to show in the selected time rage](https://discuss.elastic.co/t/this-interval-creates-too-many-buckets-to-show-in-the-selected-time-rage/273001)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 27\ **Last updated:** [May 19, 2021, 8:29pm UTC](https://discuss.elastic.co/t/this-interval-creates-too-many-buckets-to-show-in-the-selected-time-rage/273001 "2021-05-19T20:29:17Z")
Hello guys. I have set a elastic search and kibana to receive some syslogs from our OnPrem server. When I go to the discover section and set the time range to refresh every second. I got the warning This interval crea… --- ## [Logstash Http Input Plugin - Encrypted Private Key File is not Valid](https://discuss.elastic.co/t/logstash-http-input-plugin-encrypted-private-key-file-is-not-valid/273438)
**Author:** [@dickysum](https://discuss.elastic.co/u/dickysum)\ **Replies:** 0\ **Last updated:** [May 19, 2021, 6:21pm UTC](https://discuss.elastic.co/t/logstash-http-input-plugin-encrypted-private-key-file-is-not-valid/273438 "2021-05-19T18:21:24Z")
Hi, I am trying to use an encrypted private key for a Logstash HTTP Input where both the private key and certificate are stored locally on the server. However, if I encrypt the private key and configure the ssl\_key\_passp… --- ## [Prettified json is not parsed by logstash s3 input plugin](https://discuss.elastic.co/t/prettified-json-is-not-parsed-by-logstash-s3-input-plugin/273398)
**Author:** [@Anubha](https://discuss.elastic.co/u/Anubha)\ **Replies:** 4\ **Last updated:** [May 19, 2021, 6:01pm UTC](https://discuss.elastic.co/t/prettified-json-is-not-parsed-by-logstash-s3-input-plugin/273398 "2021-05-19T18:01:03Z")
We're trying to parse multiline json file from an s3 bucket which results in "\_jsonparsefailure". It reads the file line by line. The codec we're using is json\_lines. Our logstash config looks like this: input { … --- ## [Remove first few lines of csv in logstash](https://discuss.elastic.co/t/remove-first-few-lines-of-csv-in-logstash/273359)
**Author:** [@Osiris](https://discuss.elastic.co/u/Osiris)\ **Replies:** 1\ **Last updated:** [May 19, 2021, 4:42pm UTC](https://discuss.elastic.co/t/remove-first-few-lines-of-csv-in-logstash/273359 "2021-05-19T16:42:16Z")
This is the input I am using for logstash. ItemId AssetId ItemName Comment 11111 07 ABCDa XYZa 11112 07 ABCDb XYZb 11113 07 ABCDc XYZc 11114 07 ABCDd XYZd 11115 07 ABCD… --- ## [How far to push the CPU on logstash](https://discuss.elastic.co/t/how-far-to-push-the-cpu-on-logstash/273363)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 1\ **Last updated:** [May 19, 2021, 4:26pm UTC](https://discuss.elastic.co/t/how-far-to-push-the-cpu-on-logstash/273363 "2021-05-19T16:26:47Z")
Hi, I have a 8 Core CPU with 16GB of RAM. 8 GB has been allocated to Logstash. Nothing else is running on the machine. The logstash is running a single pipeline only. I have seen that even when I configure the number… --- ## [Load data from SQL server to Elasticsearch on local](https://discuss.elastic.co/t/load-data-from-sql-server-to-elasticsearch-on-local/273196)
**Author:** [@Dai\_Thai\_Hoa\_Vo](https://discuss.elastic.co/u/Dai_Thai_Hoa_Vo)\ **Replies:** 9\ **Last updated:** [May 19, 2021, 4:15pm UTC](https://discuss.elastic.co/t/load-data-from-sql-server-to-elasticsearch-on-local/273196 "2021-05-19T16:15:24Z")
I have local sql server and local ELK . I configed logstash.conf connect to SQL Server. The logstash have read data from SQL but when write data to Elasticsearch only 1 row. Please any one explain to me about this. inp… --- ## [Ingest CSV file with Logstash fails](https://discuss.elastic.co/t/ingest-csv-file-with-logstash-fails/273431)
**Author:** [@Oscar\_Lopez](https://discuss.elastic.co/u/Oscar_Lopez)\ **Replies:** 1\ **Last updated:** [May 19, 2021, 3:36pm UTC](https://discuss.elastic.co/t/ingest-csv-file-with-logstash-fails/273431 "2021-05-19T15:36:28Z")
I am trying to ingest a CSV file but I get an error when parsing the fields with the following configuration: input { file { path =\> "/etc/logstash/conf.d/taconería.conf" start\_position =\> "beginning" #s… --- ## [Parsing json array](https://discuss.elastic.co/t/parsing-json-array/273218)
**Author:** [@Amy007](https://discuss.elastic.co/u/Amy007)\ **Replies:** 5\ **Last updated:** [May 19, 2021, 2:07pm UTC](https://discuss.elastic.co/t/parsing-json-array/273218 "2021-05-19T14:07:33Z")
Hi , I have json having same structure as the below . How should I parse it?I tried split filter but its not working . \[ { "accounting": \[ { "firstName": "a35aa", "lastName": "Doe", "age": 23 }, { "firstName": … --- ## [Extract specific line from java stack trace](https://discuss.elastic.co/t/extract-specific-line-from-java-stack-trace/273180)
**Author:** [@gerry1](https://discuss.elastic.co/u/gerry1)\ **Replies:** 2\ **Last updated:** [May 19, 2021, 1:45pm UTC](https://discuss.elastic.co/t/extract-specific-line-from-java-stack-trace/273180 "2021-05-19T13:45:31Z")
hi everyone, what i need to do is to extract from a java stack trace a specific line using the elk stack. What i have, as example, is the exception below and in kibana I want to have only the line starting with com.hos… --- ## [KV filter and fields separator](https://discuss.elastic.co/t/kv-filter-and-fields-separator/273286)
**Author:** [@qttv](https://discuss.elastic.co/u/qttv)\ **Replies:** 23\ **Last updated:** [May 19, 2021, 12:13pm UTC](https://discuss.elastic.co/t/kv-filter-and-fields-separator/273286 "2021-05-19T12:13:05Z")
Good morning. I'm using a KV filter in logstash to parse the content of a JSON log. I have to use a comma "," as separator between different fileds, but there is some values wich contains string with commas in the text, … --- ## [Logstash filtering drives me nuts](https://discuss.elastic.co/t/logstash-filtering-drives-me-nuts/273258)
**Author:** [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Replies:** 7\ **Last updated:** [May 19, 2021, 10:44am UTC](https://discuss.elastic.co/t/logstash-filtering-drives-me-nuts/273258 "2021-05-19T10:44:25Z")
Hi, For some time now I have this stack, filbeats shipping logs of several application servers to elasticsearch via logstash. Logstash filtering is done in 4 pipelines. Now recently our supplier decided to change a fiel… --- ## [Bulk Requests to HTTP Input](https://discuss.elastic.co/t/bulk-requests-to-http-input/273381)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [May 19, 2021, 10:01am UTC](https://discuss.elastic.co/t/bulk-requests-to-http-input/273381 "2021-05-19T10:01:32Z")
Hi, I have an http input configured to receive http bulk requests in application/x-ndjson format. However, logs are not flowing. The input config looks like this: input { http { codec =\> "json" … --- ## [Load data from SQL server to Elasticsearch with document\_id on local](https://discuss.elastic.co/t/load-data-from-sql-server-to-elasticsearch-with-document-id-on-local/273373)
**Author:** [@Dai\_Thai\_Hoa\_Vo](https://discuss.elastic.co/u/Dai_Thai_Hoa_Vo)\ **Replies:** 6\ **Last updated:** [May 19, 2021, 10:08am UTC](https://discuss.elastic.co/t/load-data-from-sql-server-to-elasticsearch-with-document-id-on-local/273373 "2021-05-19T10:08:04Z")
Hi every one. My local table has 3000 rows. When I config logstash output with document\_id =\> "%{countyId}". Just only lastest row is inserted to ElasticSearch. But when I not use document\_id =\> "%{countyId}" so it wo… --- ## [Monitor logstash missing data](https://discuss.elastic.co/t/monitor-logstash-missing-data/273277)
**Author:** [@vikramdayma](https://discuss.elastic.co/u/vikramdayma)\ **Replies:** 6\ **Last updated:** [May 19, 2021, 9:32am UTC](https://discuss.elastic.co/t/monitor-logstash-missing-data/273277 "2021-05-19T09:32:23Z")
Hi, I setup a pipeline in logstash for getting data from sql db by jdbc connector and insert into elastic search index on daily basis. I setup cron time and sync pipeline with sql db on a date field. My problem is: H… --- ## [What is the propper way to create in logstash checkpoint filter](https://discuss.elastic.co/t/what-is-the-propper-way-to-create-in-logstash-checkpoint-filter/273309)
**Author:** [@Enrique\_Pedroza](https://discuss.elastic.co/u/Enrique_Pedroza)\ **Replies:** 4\ **Last updated:** [May 19, 2021, 8:55am UTC](https://discuss.elastic.co/t/what-is-the-propper-way-to-create-in-logstash-checkpoint-filter/273309 "2021-05-19T08:55:21Z")
I have a ELK stack in my server, in other pc i have filebeat listening for Checkpoint syslogs. I know i have to write the filters in a .conf file. This is mine : filter{ if "checkpoint-firewall" in \[tags\] { gro… --- ## [Using different Index names in Output logstash](https://discuss.elastic.co/t/using-different-index-names-in-output-logstash/273364)
**Author:** [@Saravana37](https://discuss.elastic.co/u/Saravana37)\ **Replies:** 4\ **Last updated:** [May 19, 2021, 8:21am UTC](https://discuss.elastic.co/t/using-different-index-names-in-output-logstash/273364 "2021-05-19T08:21:08Z")
Hello All , I want to re-use the config lines in output logstash as shown below. I am using if condition but is throwing some error. Please help me out. output { elasticsearch { hosts =\> \["https://… --- ## [Wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - no response loading for long time](https://discuss.elastic.co/t/wget-qo-https-artifacts-elastic-co-gpg-key-elasticsearch-sudo-apt-key-add-no-response-loading-for-long-time/273230)
**Author:** [@Rahul\_Ravichandran](https://discuss.elastic.co/u/Rahul_Ravichandran)\ **Replies:** 2\ **Last updated:** [May 19, 2021, 1:52am UTC](https://discuss.elastic.co/t/wget-qo-https-artifacts-elastic-co-gpg-key-elasticsearch-sudo-apt-key-add-no-response-loading-for-long-time/273230 "2021-05-19T01:52:44Z")
Hey guys Im trying to install logstash and the very first command is wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - as given in the steps to install in this link Installing Logstash… --- ## [Grok patterns with quotation marks](https://discuss.elastic.co/t/grok-patterns-with-quotation-marks/273003)
**Author:** [@qttv](https://discuss.elastic.co/u/qttv)\ **Replies:** 17\ **Last updated:** [May 18, 2021, 10:01pm UTC](https://discuss.elastic.co/t/grok-patterns-with-quotation-marks/273003 "2021-05-18T22:01:48Z")
Good Evening, I'm trying to extrapolate some fields from the following log message: { "message": "Throw: Il prodotto estratto non è presente tra quelli del menu a tendina in fase di emissione della proposta", "level":… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=227) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=229)