# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=281 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 282 --- ## [Logstash pipeline terminates immediately after logstash container starts up](https://discuss.elastic.co/t/logstash-pipeline-terminates-immediately-after-logstash-container-starts-up/255208)
**Author:** [@Sam\_Mellors](https://discuss.elastic.co/u/Sam_Mellors)\ **Replies:** 1\ **Last updated:** [November 17, 2020, 12:18pm UTC](https://discuss.elastic.co/t/logstash-pipeline-terminates-immediately-after-logstash-container-starts-up/255208 "2020-11-17T12:18:32Z")
Hi everyone I'm trying to set up logstash in a kubernetes cluster, but whenever the container starts the logs say it terminates the pipeline I have: \[\[.monitoring-logstash\]-pipeline-manager\] javapipeline - Pipeline term… --- ## [Logstash docker error -\> exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"input\\", \\"filter\\", \\"output\\" at line 1, column 1 (byte 1)"](https://discuss.elastic.co/t/logstash-docker-error-exception-logstash-configurationerror-message-expected-one-of-t-r-n-input-filter-output-at-line-1-column-1-byte-1/255210)
**Author:** [@s0umen](https://discuss.elastic.co/u/s0umen)\ **Replies:** 2\ **Last updated:** [November 17, 2020, 11:55am UTC](https://discuss.elastic.co/t/logstash-docker-error-exception-logstash-configurationerror-message-expected-one-of-t-r-n-input-filter-output-at-line-1-column-1-byte-1/255210 "2020-11-17T11:55:59Z")
Hi, I am facing a peculiar issue. I have seen the error faced by many before me, but the solution (only a few of them said resolved by fresh creation by typing the logstash.conf file) is not working for me. Please help… --- ## [Converting value in field](https://discuss.elastic.co/t/converting-value-in-field/255527)
**Author:** [@Lokesh\_Mayan](https://discuss.elastic.co/u/Lokesh_Mayan)\ **Replies:** 2\ **Last updated:** [November 17, 2020, 9:57am UTC](https://discuss.elastic.co/t/converting-value-in-field/255527 "2020-11-17T09:57:17Z")
Hi i am having a log list like this \<#\> 20200924 15:14:44.552 000000997 FD.WRN \[ MVINB-LSL2.T1G1\_ATMTH\_SL main.main ConfigService.ConfigService \] Unable to get CommonParamConfig of \[DAC\_TEST\_WHILE\_IDLE… --- ## [Does logstash continue getting log from Kafka if it cannot send log to ELS?](https://discuss.elastic.co/t/does-logstash-continue-getting-log-from-kafka-if-it-cannot-send-log-to-els/255648)
**Author:** [@AkatsukiPain](https://discuss.elastic.co/u/AkatsukiPain)\ **Replies:** 1\ **Last updated:** [November 17, 2020, 9:40am UTC](https://discuss.elastic.co/t/does-logstash-continue-getting-log-from-kafka-if-it-cannot-send-log-to-els/255648 "2020-11-17T09:40:43Z")
I want to build a topology like this, but I don't know the behavior of logstash when elasticsearch dies ( Logstash cannot send the log to Elasticsearch). There are 2 situations that I'm thinking Logstash will stop get… --- ## [Parse the json \[{},{},{}\]\[{},{},{}\] with logstash](https://discuss.elastic.co/t/parse-the-json-with-logstash/255642)
**Author:** [@thibd](https://discuss.elastic.co/u/thibd)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 8:48am UTC](https://discuss.elastic.co/t/parse-the-json-with-logstash/255642 "2020-11-17T08:48:48Z")
Hello, I have a json from scrapy with this format: \[{"website1":"myurl1","importdate":"X"},{"website2":"myurl2","importdate":"X"}\] \[{"website1":"myurl1","importdate":"Y"},{"website3":"myurl3","importdate":"Y"}\] So a n… --- ## [Logstash - Json appearing in Message field](https://discuss.elastic.co/t/logstash-json-appearing-in-message-field/255641)
**Author:** [@weiyentan](https://discuss.elastic.co/u/weiyentan)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 8:46am UTC](https://discuss.elastic.co/t/logstash-json-appearing-in-message-field/255641 "2020-11-17T08:46:21Z")
Hi, I am kind of new to the elasticsearch family so thought I would try to create some use cases. One of those is to upload system information into logstash from json documents gathered by ansible . Ansible then reads … --- ## [How to sort the data in Logstash?](https://discuss.elastic.co/t/how-to-sort-the-data-in-logstash/255628)
**Author:** [@yummycake](https://discuss.elastic.co/u/yummycake)\ **Replies:** 4\ **Last updated:** [November 17, 2020, 7:37am UTC](https://discuss.elastic.co/t/how-to-sort-the-data-in-logstash/255628 "2020-11-17T07:37:39Z")
I have ingested the data from CSV into Elasticsearch, however, when I view them in 'Discover, Kibana' the ID did not ingest in order. The field 'ID' in the CSV doc starts from 1 For example, instead of: 1 2 3 4 It… --- ## [Logstash, if statement](https://discuss.elastic.co/t/logstash-if-statement/255629)
**Author:** [@Guhan\_S](https://discuss.elastic.co/u/Guhan_S)\ **Replies:** 2\ **Last updated:** [November 17, 2020, 7:34am UTC](https://discuss.elastic.co/t/logstash-if-statement/255629 "2020-11-17T07:34:39Z")
Hi guys, we are using PINO logger.... as per their docs... trace: 10, debug: 20, info: 30, warn: 40, error: 50, fatal: 60 I am getting logs like ("level":30 ). {"level":30,"time":....,"pid":1,"hostname":"ip" My … --- ## [To Lookup the data which is coming from two different Kafka topics using Logstash](https://discuss.elastic.co/t/to-lookup-the-data-which-is-coming-from-two-different-kafka-topics-using-logstash/255618)
**Author:** [@Sangeetha\_Sivaji](https://discuss.elastic.co/u/Sangeetha_Sivaji)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 5:07am UTC](https://discuss.elastic.co/t/to-lookup-the-data-which-is-coming-from-two-different-kafka-topics-using-logstash/255618 "2020-11-17T05:07:25Z")
Hello, I have a requirement that "Logstash need to read the data from two different Kafka topics, lookup the read data using filter plugin and finally send it to Elasticsearch". Two Kafka topics are presented, one cons… --- ## [Parsing syslog param1=value1 param2="value2"](https://discuss.elastic.co/t/parsing-syslog-param1-value1-param2-value2/255565)
**Author:** [@Elitlogik](https://discuss.elastic.co/u/Elitlogik)\ **Replies:** 3\ **Last updated:** [November 16, 2020, 9:52pm UTC](https://discuss.elastic.co/t/parsing-syslog-param1-value1-param2-value2/255565 "2020-11-16T21:52:00Z")
Trying to learn GROK by using the GROK debugger tool in Kibana. Data is stuctured like this: param1=value1 param2="value 2" param4=param4 param1=value1 param2="value 2" param3=param3 param4=param4 Questions are: W… --- ## [Logstash error after updating to 7.9.1](https://discuss.elastic.co/t/logstash-error-after-updating-to-7-9-1/255366)
**Author:** [@noobman2logstash](https://discuss.elastic.co/u/noobman2logstash)\ **Replies:** 8\ **Last updated:** [November 16, 2020, 6:46pm UTC](https://discuss.elastic.co/t/logstash-error-after-updating-to-7-9-1/255366 "2020-11-16T18:46:07Z")
\[2020-11-13T16:36:21,302\]\[ERROR\]\[logstash.agent \] Failed to execute action {:id=\>:default, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, acti… --- ## [Solution for Logstash json file input - solution pour injecter un fichier json formaté comme ci dessous](https://discuss.elastic.co/t/solution-for-logstash-json-file-input-solution-pour-injecter-un-fichier-json-formate-comme-ci-dessous/255564)
**Author:** [@pmo](https://discuss.elastic.co/u/pmo)\ **Replies:** 0\ **Last updated:** [November 16, 2020, 3:37pm UTC](https://discuss.elastic.co/t/solution-for-logstash-json-file-input-solution-pour-injecter-un-fichier-json-formate-comme-ci-dessous/255564 "2020-11-16T15:37:53Z")
I've json file that are formated like this (j'ai des fichiers json formatés comme ci dessous - les données ont été changées) \[ { "crossConnectVlanRanges": "", "productAndRelease": "FX-I.2.0", "role": "", … --- ## [Droping log when missing time entry](https://discuss.elastic.co/t/droping-log-when-missing-time-entry/255521)
**Author:** [@iccMe](https://discuss.elastic.co/u/iccMe)\ **Replies:** 2\ **Last updated:** [November 16, 2020, 3:10pm UTC](https://discuss.elastic.co/t/droping-log-when-missing-time-entry/255521 "2020-11-16T15:10:42Z")
Hi, I was wondering if anyone could advise the best way to drop a log when the actual time is missing from the logged event. Normally the majority of the logs look like: \[19/10/2020 00:00:01\] servername - Processor… --- ## [Logstash Validation Failed: 1: no requests](https://discuss.elastic.co/t/logstash-validation-failed-1-no-requests/255379)
**Author:** [@Tosh](https://discuss.elastic.co/u/Tosh)\ **Replies:** 3\ **Last updated:** [November 16, 2020, 3:04pm UTC](https://discuss.elastic.co/t/logstash-validation-failed-1-no-requests/255379 "2020-11-16T15:04:11Z")
Hi , Logstash gives below error intermittently and then stops processing data after a while . Logstash version : 7.0.1 Elastic version : 7.0.1 Java : JDK8 Could it be due to malformed requests ? We do see some malfo… --- ## [Logstash monitoring not showing one pipeline](https://discuss.elastic.co/t/logstash-monitoring-not-showing-one-pipeline/255559)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 0\ **Last updated:** [November 16, 2020, 2:56pm UTC](https://discuss.elastic.co/t/logstash-monitoring-not-showing-one-pipeline/255559 "2020-11-16T14:56:12Z")
Hi, I have two machines with logstash Machine 1: only logstash 7.7 with 4 pipelines, one of them called "main" Machine 2: logstash 7.3, kibana 7.3 and elasticsearch 7.3. Logstash only one pipeline called "main" On bo… --- ## [Regex Pattern to read log from application server](https://discuss.elastic.co/t/regex-pattern-to-read-log-from-application-server/255492)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [November 16, 2020, 10:40am UTC](https://discuss.elastic.co/t/regex-pattern-to-read-log-from-application-server/255492 "2020-11-16T10:40:22Z")
HI Team, WE are using ELK 6.4.0 in production and by using Rsyslog service we are sending log from application server to logstash. In application log we have multline data and we are not sure which we can use to send t… --- ## [Using s3 input plugin logstash file\_descriptors aren't closed leading to crash of system](https://discuss.elastic.co/t/using-s3-input-plugin-logstash-file-descriptors-arent-closed-leading-to-crash-of-system/254270)
**Author:** [@petersedivec](https://discuss.elastic.co/u/petersedivec)\ **Replies:** 4\ **Last updated:** [November 16, 2020, 8:03am UTC](https://discuss.elastic.co/t/using-s3-input-plugin-logstash-file-descriptors-arent-closed-leading-to-crash-of-system/254270 "2020-11-16T08:03:22Z")
Been trying to figure out our problem for the past week and haven't figured out a solution other than restart logstash multiple times when it runs out of resources. Looking for any help or suggestions on what we can try… --- ## [Logstash to log file processing start and end time](https://discuss.elastic.co/t/logstash-to-log-file-processing-start-and-end-time/255462)
**Author:** [@sam281](https://discuss.elastic.co/u/sam281)\ **Replies:** 2\ **Last updated:** [November 16, 2020, 3:59am UTC](https://discuss.elastic.co/t/logstash-to-log-file-processing-start-and-end-time/255462 "2020-11-16T03:59:41Z")
Hello, I am trying to forward like 365 gzip historical data files to another log management tool through syslog output plugin and using file input with Read mode. I need to track and log timestamp for every file when i… --- ## [Logstash saving data into two indices (Duplicated Data)](https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457)
**Author:** [@gonzalo2kx](https://discuss.elastic.co/u/gonzalo2kx)\ **Replies:** 1\ **Last updated:** [November 16, 2020, 12:16am UTC](https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457 "2020-11-16T00:16:00Z")
Good day, I am currently processing the data from filebeat's system module using Logstash. The data provided by filebeat's system module (syslog and auth) is properly parsed but unfortunately it was saving the structure… --- ## [System has not been booted with systemd as init system (PID 1)](https://discuss.elastic.co/t/system-has-not-been-booted-with-systemd-as-init-system-pid-1/255422)
**Author:** [@Naba01](https://discuss.elastic.co/u/Naba01)\ **Replies:** 0\ **Last updated:** [November 15, 2020, 5:04am UTC](https://discuss.elastic.co/t/system-has-not-been-booted-with-systemd-as-init-system-pid-1/255422 "2020-11-15T05:04:03Z")
I have installed elasticsearch and kibana on my WSL.Able to start services using "sudo /etc/init.d/" command But unable to start services for logstash Below command is not working: root@Flamingo:~# sudo systemctl star… --- ## [Logstash error after installing ssl](https://discuss.elastic.co/t/logstash-error-after-installing-ssl/254807)
**Author:** [@Cosmin\_Ciobanu](https://discuss.elastic.co/u/Cosmin_Ciobanu)\ **Replies:** 4\ **Last updated:** [November 11, 2020, 12:52am UTC](https://discuss.elastic.co/t/logstash-error-after-installing-ssl/254807 "2020-11-11T00:52:09Z")
\[ERROR\] 2020-11-09 10:08:12.337 \[\[main\]-pipeline-manager\] javapipeline - Pipeline error {:pipeline\_id=\>"main", :exception=\>#\ **Author:** [@Lokesh\_Mayan](https://discuss.elastic.co/u/Lokesh_Mayan)\ **Replies:** 11\ **Last updated:** [November 14, 2020, 5:19pm UTC](https://discuss.elastic.co/t/i-want-to-convert-my-date-format-yyyymmdd-into-month-name-date-and-year-like-mmm-dd-yyy-apr-30-2012/255394 "2020-11-14T17:19:26Z") Hi I am having a log like the below \<#\> 20200924 15:14:40.918 280018000 EV.INF \[ MVINB-LSL2.T1G1\_ATMTH\_SL main.main TSP1.T1G1\_ATMTH\_SL \] Process T1G1\_ATMTH\_SL starting in progress... in the … --- ## [How to seperate message field filebeat](https://discuss.elastic.co/t/how-to-seperate-message-field-filebeat/255167)
**Author:** [@Hi\_u\_Nguy\_n\_Dang](https://discuss.elastic.co/u/Hi_u_Nguy_n_Dang)\ **Replies:** 5\ **Last updated:** [November 14, 2020, 1:00pm UTC](https://discuss.elastic.co/t/how-to-seperate-message-field-filebeat/255167 "2020-11-14T13:00:39Z")
Guys, how can I seperate message field with filebeat to more field ? It keep just giving me message field, I am trying to use grok filter but it does not work. Filebeat agent work in 1 machine meanwhile elastic server wo… --- ## [Elastic and Logstash pipeline for Customer history](https://discuss.elastic.co/t/elastic-and-logstash-pipeline-for-customer-history/255181)
**Author:** [@ratheeshkrishnan92](https://discuss.elastic.co/u/ratheeshkrishnan92)\ **Replies:** 2\ **Last updated:** [November 14, 2020, 9:00am UTC](https://discuss.elastic.co/t/elastic-and-logstash-pipeline-for-customer-history/255181 "2020-11-14T09:00:14Z")
Hello Team, I am using Elasticsearch version 6.2.4. I have below scenario and need a work around for this. Planning to have two indices with below properties. EmployeeId, EmployeeName, Designation. index 1 : Employee… --- ## [Does logstash-input-s3-sns-sqs support backup\_to\_bucket?](https://discuss.elastic.co/t/does-logstash-input-s3-sns-sqs-support-backup-to-bucket/255389)
**Author:** [@Ying-Yi\_Huang](https://discuss.elastic.co/u/Ying-Yi_Huang)\ **Replies:** 0\ **Last updated:** [November 14, 2020, 12:48am UTC](https://discuss.elastic.co/t/does-logstash-input-s3-sns-sqs-support-backup-to-bucket/255389 "2020-11-14T00:48:05Z")
Hello, I want to use the plugin to read from S3 and then backup the files completed to a backup S3 back. It doesn't seem to work. Here is my config: input { s3snssqs { region =\> "us-e… --- ## [Logstash import CSV data classification](https://discuss.elastic.co/t/logstash-import-csv-data-classification/255362)
**Author:** [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\ **Replies:** 4\ **Last updated:** [November 13, 2020, 7:59pm UTC](https://discuss.elastic.co/t/logstash-import-csv-data-classification/255362 "2020-11-13T19:59:07Z")
First off, I am relatively inexperienced with ELK and am trying to perform a demo of ELK because I'm convinced it is a powerful and useful application for much of our work. I have been presented with a CSV export of dat… --- ## [Logstash not reading CSV file when running as CSV](https://discuss.elastic.co/t/logstash-not-reading-csv-file-when-running-as-csv/255374)
**Author:** [@errupeshmca](https://discuss.elastic.co/u/errupeshmca)\ **Replies:** 1\ **Last updated:** [November 13, 2020, 7:46pm UTC](https://discuss.elastic.co/t/logstash-not-reading-csv-file-when-running-as-csv/255374 "2020-11-13T19:46:16Z")
Hi, I am trying to run ELK through service on windows, Kibana and Elasticsearch is working fine as service. I have configured service through nssm and used below reference to configure logstash Logstash starting p… --- ## [Parser error when logstash reading updated CSV file](https://discuss.elastic.co/t/parser-error-when-logstash-reading-updated-csv-file/254813)
**Author:** [@errupeshmca](https://discuss.elastic.co/u/errupeshmca)\ **Replies:** 2\ **Last updated:** [November 13, 2020, 7:19pm UTC](https://discuss.elastic.co/t/parser-error-when-logstash-reading-updated-csv-file/254813 "2020-11-13T19:19:58Z")
Hi, Logstash giving parser error whenever CSV file gets updated with new records. However logstash works fine when you delete the old csv file and replace with new updated CSV file, but in that scenerio logstash all rec… --- ## [Unable to post the client's cert to Webapi](https://discuss.elastic.co/t/unable-to-post-the-clients-cert-to-webapi/255076)
**Author:** [@sdr](https://discuss.elastic.co/u/sdr)\ **Replies:** 2\ **Last updated:** [November 13, 2020, 2:31pm UTC](https://discuss.elastic.co/t/unable-to-post-the-clients-cert-to-webapi/255076 "2020-11-13T14:31:40Z")
I'm using elk version 6.7.0. Issue with sending the client certificate using logstash output plugin to webapi. --- ## [How to get entire enriched "event" size (not message size)](https://discuss.elastic.co/t/how-to-get-entire-enriched-event-size-not-message-size/255184)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 2\ **Last updated:** [November 13, 2020, 10:05am UTC](https://discuss.elastic.co/t/how-to-get-entire-enriched-event-size-not-message-size/255184 "2020-11-13T10:05:53Z")
hi As per the link below, we got the size of the "message" field But after doing transformation, we need to have the size of the final event also. How to get it? I tried event.set('event\_size', event.get('\_source').… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=280) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=282)