# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=473 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 474 --- ## [Pipeline aborted due to error.PatternError: pattern %{REMOTEIP:remoteip} not defined\>](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-patternerror-pattern-remoteip-remoteip-not-defined/200677)
**Author:** [@Deepanshu\_Malik](https://discuss.elastic.co/u/Deepanshu_Malik)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 11:08am UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-patternerror-pattern-remoteip-remoteip-not-defined/200677 "2019-09-23T11:08:47Z")
here is my config file input { file { path =\> "/log/apache-log/access\_log" type =\> "accesslogs" } } filter { if \[type\] == "accesslogs" { grok { patterns\_dir =\> "/opt/elastic/logstash/patterns/" match =\> { "mes… --- ## [AVRO Single-object encoding](https://discuss.elastic.co/t/avro-single-object-encoding/200657)
**Author:** [@pinoniq](https://discuss.elastic.co/u/pinoniq)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 9:42am UTC](https://discuss.elastic.co/t/avro-single-object-encoding/200657 "2019-09-23T09:42:29Z")
Azure allows one to store events in a Blob storage coming from an Azure Event hub. These are stores as Single-Objevt encoded avro files. The specs: https://avro.apache.org/docs/1.8.2/spec.html#N104B8 This is currently … --- ## [Codec multiline in Input section doesn't load last line](https://discuss.elastic.co/t/codec-multiline-in-input-section-doesnt-load-last-line/200637)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 1\ **Last updated:** [September 23, 2019, 7:47am UTC](https://discuss.elastic.co/t/codec-multiline-in-input-section-doesnt-load-last-line/200637 "2019-09-23T07:47:53Z")
I have below configuration , Last line of log is not getting loaded. codec =\> multiline { pattern =\> "(^\[0-9\]{4})|(^\[A-Z,a-z\]{3})" negate =\> "true" what =\> "previous" auto\_flush\_interval =\> 60 } } --- ## [Logstash unable to start](https://discuss.elastic.co/t/logstash-unable-to-start/200474)
**Author:** [@sk545](https://discuss.elastic.co/u/sk545)\ **Replies:** 5\ **Last updated:** [September 23, 2019, 7:05am UTC](https://discuss.elastic.co/t/logstash-unable-to-start/200474 "2019-09-23T07:05:33Z")
I have installed logstash 7.3.2 on EC2 RHEL linux , when i try to start the service i am getting below error systemctl start logstash "Failed to start logstash.service: Unit not found." i have downloaded logstash RPM … --- ## [\[ERROR\]\[logstash.agent Failed to execute action {:id=\>:main...}](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-id-main/200248)
**Author:** [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Replies:** 1\ **Last updated:** [September 23, 2019, 6:17am UTC](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-id-main/200248 "2019-09-23T06:17:46Z")
Hello Everyone, i have been experiencing a problem with starting the pipeline ever since turning on SSL connection for Elasticsearch, Kibana and Logstash. My configuration is below: input { beats { port =\> 6969 } } … --- ## [Clarification on logstash configuration](https://discuss.elastic.co/t/clarification-on-logstash-configuration/200619)
**Author:** [@durgaram](https://discuss.elastic.co/u/durgaram)\ **Replies:** 2\ **Last updated:** [September 23, 2019, 5:09am UTC](https://discuss.elastic.co/t/clarification-on-logstash-configuration/200619 "2019-09-23T05:09:11Z")
I am trying to install ELK stack 7.3.2, I (filebeat included) have used the the following logstash conf., the fieldname logappname is not found in the kibana. Am I missing something? input { beats { port =\> "5044" } … --- ## [Clarification on if/else behavior in output](https://discuss.elastic.co/t/clarification-on-if-else-behavior-in-output/200617)
**Author:** [@TechGeekNZ](https://discuss.elastic.co/u/TechGeekNZ)\ **Replies:** 2\ **Last updated:** [September 23, 2019, 4:23am UTC](https://discuss.elastic.co/t/clarification-on-if-else-behavior-in-output/200617 "2019-09-23T04:23:01Z")
Heya all, I want to confirm whether the particular behavior i'm seeing is normal and if there is another approach I should take towards my config. Situation is that I have multiple inputs on a single pipeline, and am a… --- ## [LOgstash "Citcuit brealing exception"](https://discuss.elastic.co/t/logstash-citcuit-brealing-exception/200613)
**Author:** [@Prabia\_Sharma](https://discuss.elastic.co/u/Prabia_Sharma)\ **Replies:** 0\ **Last updated:** [September 23, 2019, 1:53am UTC](https://discuss.elastic.co/t/logstash-citcuit-brealing-exception/200613 "2019-09-23T01:53:18Z")
l l Hello Experts I am experiencing issues with logstash which is remotely conneced to elasticsearch having bth the images on docker logstash\_1 | \[2019-09-20T03:06:21,384\]\[INFO \]\[logstash.outputs.elasticsearch\] retr… --- ## [Could not index event to Elasticsearch after upgrade](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-after-upgrade/200593)
**Author:** [@hackdefendr](https://discuss.elastic.co/u/hackdefendr)\ **Replies:** 4\ **Last updated:** [September 22, 2019, 3:15pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-after-upgrade/200593 "2019-09-22T15:15:46Z")
\[2019-09-22T09:37:34,941\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"logstash-2019.09.22", :\_type=\>"doc", :routing=\>nil}, #\ **Author:** [@madurad](https://discuss.elastic.co/u/madurad)\ **Replies:** 0\ **Last updated:** [September 22, 2019, 1:05pm UTC](https://discuss.elastic.co/t/logstash-is-not-pushing-logs-to-es-intermittently/200582 "2019-09-22T13:05:56Z") I've configured the ES in AWS in elasticsearch service, and logstash is configured separately in another server. All my production servers are pushing logs to logstash via filebeat services. This system was working end t… --- ## [Decode URL Query After use KV Filter](https://discuss.elastic.co/t/decode-url-query-after-use-kv-filter/200579)
**Author:** [@gholeh89](https://discuss.elastic.co/u/gholeh89)\ **Replies:** 0\ **Last updated:** [September 22, 2019, 11:53am UTC](https://discuss.elastic.co/t/decode-url-query-after-use-kv-filter/200579 "2019-09-22T11:53:05Z")
Hi, I am new to Logstash and my case need get all logs from s3 cloudfront logs format the problem when using decode filter plugin this my code Blockquote input { s3 { bucket =\> "test" access\_key\_id =\> "test" … --- ## [Ingesting only new documents from Elasticsearch](https://discuss.elastic.co/t/ingesting-only-new-documents-from-elasticsearch/200520)
**Author:** [@eebee](https://discuss.elastic.co/u/eebee)\ **Replies:** 0\ **Last updated:** [September 21, 2019, 8:50am UTC](https://discuss.elastic.co/t/ingesting-only-new-documents-from-elasticsearch/200520 "2019-09-21T08:50:30Z")
I'm using the elasticsearch input plugin to get documents from another elasticsearch instance. Then the elasticsearch output plugin to ingest it in another instance. But how do I schedule/filter the plugins/pipeline, so… --- ## [Netflow](https://discuss.elastic.co/t/netflow/200511)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 0\ **Last updated:** [September 21, 2019, 6:05am UTC](https://discuss.elastic.co/t/netflow/200511 "2019-09-21T06:05:53Z")
I have configured and deployed netflow plugin successfully using logstash....But in the logs I am not getting any data related to local traffic...i.e. if two pc are transferring any data in the same network then should i… --- ## [Unable to fetch metrics from AWS/ECS - Cloudwatch input plugin](https://discuss.elastic.co/t/unable-to-fetch-metrics-from-aws-ecs-cloudwatch-input-plugin/200477)
**Author:** [@Arpan\_Patnaik](https://discuss.elastic.co/u/Arpan_Patnaik)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 7:10pm UTC](https://discuss.elastic.co/t/unable-to-fetch-metrics-from-aws-ecs-cloudwatch-input-plugin/200477 "2019-09-20T19:10:53Z")
Hi All, I have been trying to get the cloudwatch input plugin working with the AWS/ECS metrics, but I have been unable to get any data out. There are strangely no errors, but no data is picked up my the plugin either. P… --- ## [Logstash not reading files changes](https://discuss.elastic.co/t/logstash-not-reading-files-changes/199305)
**Author:** [@Sergioc](https://discuss.elastic.co/u/Sergioc)\ **Replies:** 2\ **Last updated:** [September 20, 2019, 6:34pm UTC](https://discuss.elastic.co/t/logstash-not-reading-files-changes/199305 "2019-09-20T18:34:16Z")
I have Logstash installed and im reading a file that is remote and also i have created a pipeline for a local file. I was thinking at the beginning that the issue was the remote file but the local one is not read at all … --- ## [How to verify what logs have been read from dead\_letter\_queue](https://discuss.elastic.co/t/how-to-verify-what-logs-have-been-read-from-dead-letter-queue/199371)
**Author:** [@RITZ\_VERMA](https://discuss.elastic.co/u/RITZ_VERMA)\ **Replies:** 1\ **Last updated:** [September 20, 2019, 6:19pm UTC](https://discuss.elastic.co/t/how-to-verify-what-logs-have-been-read-from-dead-letter-queue/199371 "2019-09-20T18:19:55Z")
I am reading events from dead queue using dead\_letter\_queue plugin but i am not able to verify if these logs are sent to elasticsearch.is there a way to see what logs have been sent to elasticsearch so that i can delete … --- ## [Http filter leaves fields, that are inserted to elasticsearch](https://discuss.elastic.co/t/http-filter-leaves-fields-that-are-inserted-to-elasticsearch/200470)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 2\ **Last updated:** [September 20, 2019, 6:11pm UTC](https://discuss.elastic.co/t/http-filter-leaves-fields-that-are-inserted-to-elasticsearch/200470 "2019-09-20T18:11:52Z")
Basically my pipeline looks like this: input { (...) #some input } filter { http { (...) #http request } } output{ elasticsearch{ (...) action =\> "update" doc\_as\_upsert =\> … --- ## [Few words in data stream rsults in data loss using logstash elastic search](https://discuss.elastic.co/t/few-words-in-data-stream-rsults-in-data-loss-using-logstash-elastic-search/200453)
**Author:** [@chintanvasani](https://discuss.elastic.co/u/chintanvasani)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 3:35pm UTC](https://discuss.elastic.co/t/few-words-in-data-stream-rsults-in-data-loss-using-logstash-elastic-search/200453 "2019-09-20T15:35:34Z")
Hi I am using logstash 5.5.0 and elastic search 5.5.0 for inserting the data from txt file. I have observed if there are few words like CONSOLIDATED,MANDATE,DATED,DATE,UPDATED,OUTDATED,UOI AS part of the field value th… --- ## [Netflow --setup to ES SSL not working](https://discuss.elastic.co/t/netflow-setup-to-es-ssl-not-working/200440)
**Author:** [@frankfoti](https://discuss.elastic.co/u/frankfoti)\ **Replies:** 1\ **Last updated:** [September 20, 2019, 3:32pm UTC](https://discuss.elastic.co/t/netflow-setup-to-es-ssl-not-working/200440 "2019-09-20T15:32:20Z")
Here is my command line attempt to setup netflow in kibana: /usr/share/logstash/bin/logstash --modules netflow --setup -M "netflow.var.kibana.host=:5601" -M "netflow.var.kibana.username=elastic" -M "netflow.var.kibana.p… --- ## [Logstash 6.6.1: log showing “error:1e00007d:Cipher functions:OPENSSL\_internal:INVALID\_NONCE” on s390x](https://discuss.elastic.co/t/logstash-6-6-1-log-showing-errorcipher-functionsinvalid-nonce-on-s390x/200448)
**Author:** [@AnitaNayak](https://discuss.elastic.co/u/AnitaNayak)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 3:02pm UTC](https://discuss.elastic.co/t/logstash-6-6-1-log-showing-errorcipher-functionsinvalid-nonce-on-s390x/200448 "2019-09-20T15:02:57Z")
Hi, we are testing ELK 6.6.1 on s390x platform (Linux on Z). After starting the ELK 6.6.1 stack, everything seems working fine and we can see logs on Kibana UI, but logstash log has following entries from time to time: … --- ## [Can't yet decode flowset id error when collecting netflow](https://discuss.elastic.co/t/cant-yet-decode-flowset-id-error-when-collecting-netflow/200443)
**Author:** [@shazbot](https://discuss.elastic.co/u/shazbot)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 2:26pm UTC](https://discuss.elastic.co/t/cant-yet-decode-flowset-id-error-when-collecting-netflow/200443 "2019-09-20T14:26:08Z")
I am running netflow inputs with s3 and file outputs. my output is JSON. It's all working but I get these errors. I'm not sure how to handle this. There is a Catalyst 6807 with Sup2T and an ASA sending the flows. P… --- ## [\[WARN \]\[logstash.inputs.jdbc \] Exception when executing JDBC query {:exception=\>#\}](https://discuss.elastic.co/t/warn-logstash-inputs-jdbc-exception-when-executing-jdbc-query-exception-sequel-java-missing-converter-handling-for-full-class-name-org-postgresql-util-pgobject-simple-name-pgobject/200407)
**Author:** [@abu](https://discuss.elastic.co/u/abu)\ **Replies:** 1\ **Last updated:** [September 20, 2019, 1:50pm UTC](https://discuss.elastic.co/t/warn-logstash-inputs-jdbc-exception-when-executing-jdbc-query-exception-sequel-java-missing-converter-handling-for-full-class-name-org-postgresql-util-pgobject-simple-name-pgobject/200407 "2019-09-20T13:50:23Z")
I want fetch data from postgresql DB that contain jsonb format input to logstash and output to elasticsearch but I get the warning message and didn't proceed next step --- ## [Persistent queue in Logstash 6.1 causes problems with cef parsing](https://discuss.elastic.co/t/persistent-queue-in-logstash-6-1-causes-problems-with-cef-parsing/200435)
**Author:** [@Mugen](https://discuss.elastic.co/u/Mugen)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 1:49pm UTC](https://discuss.elastic.co/t/persistent-queue-in-logstash-6-1-causes-problems-with-cef-parsing/200435 "2019-09-20T13:49:48Z")
After enabling the persistent queue some of our cef logs could no longer be parsed. The Logs semed to be split wrong. The config looks as follows: input { tcp { port =\> 5514 type =\> "forti\_log" codec… --- ## [Error parsing simple log](https://discuss.elastic.co/t/error-parsing-simple-log/200418)
**Author:** [@javisabalete](https://discuss.elastic.co/u/javisabalete)\ **Replies:** 1\ **Last updated:** [September 20, 2019, 1:48pm UTC](https://discuss.elastic.co/t/error-parsing-simple-log/200418 "2019-09-20T13:48:06Z")
Hi! I'm working with logstash in order to parse auditd logs. I tested my patters in http://grokconstructor.appspot.com/do/match and https://grokdebug.herokuapp.com/ and seems working as expected. But.. it's not working… --- ## [How to create a consistent id, base on caseID string](https://discuss.elastic.co/t/how-to-create-a-consistent-id-base-on-caseid-string/200411)
**Author:** [@ErikWarming](https://discuss.elastic.co/u/ErikWarming)\ **Replies:** 2\ **Last updated:** [September 20, 2019, 1:40pm UTC](https://discuss.elastic.co/t/how-to-create-a-consistent-id-base-on-caseid-string/200411 "2019-09-20T13:40:25Z")
I need advise: Everyday I create a log using logstash input-jdbc to run a SQL query on a readonly\_database. In the log I have a long alpha-numeric caseID and I need a human-friendly id number instead. And this id number… --- ## [Index Joined sql data in Elasticsearch via logstash](https://discuss.elastic.co/t/index-joined-sql-data-in-elasticsearch-via-logstash/200421)
**Author:** [@sharko](https://discuss.elastic.co/u/sharko)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 12:49pm UTC](https://discuss.elastic.co/t/index-joined-sql-data-in-elasticsearch-via-logstash/200421 "2019-09-20T12:49:26Z")
Hello , I am trying to index already joined sql data into elastic search via logstash the data I read look like this {"family": "ABC", "person":{"name":"aaa","surname":"bbb"}} {"family": "ABC", "person":{"name":"ccc"… --- ## [Could not index event to Elasticsearch. Illegal argument exception](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-illegal-argument-exception/199076)
**Author:** [@Erich\_King](https://discuss.elastic.co/u/Erich_King)\ **Replies:** 4\ **Last updated:** [September 20, 2019, 9:45am UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch-illegal-argument-exception/199076 "2019-09-20T09:45:04Z")
Hi there. I am getting the following warning which is clogging up my logs. Would anyone know what the issue might be please? \[2019-09-11T11:43:42,204\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Ela… --- ## [How to ship data from mongodb to elasticsearch](https://discuss.elastic.co/t/how-to-ship-data-from-mongodb-to-elasticsearch/200367)
**Author:** [@chandu5565](https://discuss.elastic.co/u/chandu5565)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 7:46am UTC](https://discuss.elastic.co/t/how-to-ship-data-from-mongodb-to-elasticsearch/200367 "2019-09-20T07:46:15Z")
Hi, I wanted to ship the data from mongo db to elastic search. In my mongodb collection for some records I replaced the object id with custom id. So mongodb plugin is not working for me. In mongodb i have documents with… --- ## [Logstash output plugin](https://discuss.elastic.co/t/logstash-output-plugin/200130)
**Author:** [@111212](https://discuss.elastic.co/u/111212)\ **Replies:** 2\ **Last updated:** [September 20, 2019, 6:15am UTC](https://discuss.elastic.co/t/logstash-output-plugin/200130 "2019-09-20T06:15:47Z")
My version is 7.2.1 ，Write plugins through ‘logstash-output-java\_output\_example’ , use netty(nio) , Two parameters are passed in the construction parameter。 But there was a problem during the installation What sh… --- ## [CSV Output replace file](https://discuss.elastic.co/t/csv-output-replace-file/200332)
**Author:** [@Sketchy](https://discuss.elastic.co/u/Sketchy)\ **Replies:** 0\ **Last updated:** [September 20, 2019, 3:04am UTC](https://discuss.elastic.co/t/csv-output-replace-file/200332 "2019-09-20T03:04:03Z")
I am using logstash to create CSV based dictionary's for use with other pipelines, all works well except I want it to replace the CSV each time it runs. With the file output you have the option of write\_behavior = "over… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=472) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=474)