# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=495 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 496 --- ## [Invalid FieldReference issue in logstash](https://discuss.elastic.co/t/invalid-fieldreference-issue-in-logstash/194742)
**Author:** [@sukku77](https://discuss.elastic.co/u/sukku77)\ **Replies:** 1\ **Last updated:** [August 11, 2019, 12:34pm UTC](https://discuss.elastic.co/t/invalid-fieldreference-issue-in-logstash/194742 "2019-08-11T12:34:36Z")
Hi All, I am getting the below logstash exception stack trace after migrated from 6.5.4 to 7.3.0. Does this mean Logstash is dropping out some events? Do I need to update any template to suppress these warnings. There … --- ## [How To parse Json in json](https://discuss.elastic.co/t/how-to-parse-json-in-json/194720)
**Author:** [@zbawio](https://discuss.elastic.co/u/zbawio)\ **Replies:** 1\ **Last updated:** [August 11, 2019, 12:16pm UTC](https://discuss.elastic.co/t/how-to-parse-json-in-json/194720 "2019-08-11T12:16:07Z")
Hi , i am new to ELK and i wonder how can i parse json have json For ex: my "Content" field have String but this string is also a json { "Method":"some string", "Request":"some string", "Time":"Datetime", "Content"… --- ## [Csv s3 format](https://discuss.elastic.co/t/csv-s3-format/194744)
**Author:** [@Vinay\_Kumar1](https://discuss.elastic.co/u/Vinay_Kumar1)\ **Replies:** 0\ **Last updated:** [August 11, 2019, 11:44am UTC](https://discuss.elastic.co/t/csv-s3-format/194744 "2019-08-11T11:44:02Z")
How to convert logstash output of JDBC input plugin (fetching data from psql DB) to csv format and store the same in s3 bucket? I want to store the files in s3 bucket only in csv gzip format. S3 plugin does not provide … --- ## [Kafka input works but index not created in es](https://discuss.elastic.co/t/kafka-input-works-but-index-not-created-in-es/194734)
**Author:** [@real\_legend](https://discuss.elastic.co/u/real_legend)\ **Replies:** 0\ **Last updated:** [August 11, 2019, 6:48am UTC](https://discuss.elastic.co/t/kafka-input-works-but-index-not-created-in-es/194734 "2019-08-11T06:48:39Z")
i use : kafka input plugin es output plugin logstash 7.3 es 7.3 my config file for logstash: input{ kafka { bootstrap\_servers =\> "192.168.50.97:9092" topics =\> \["log-aggregator"… --- ## [Repeated record every time new events happens](https://discuss.elastic.co/t/repeated-record-every-time-new-events-happens/194714)
**Author:** [@Mohammad.ali](https://discuss.elastic.co/u/Mohammad.ali)\ **Replies:** 2\ **Last updated:** [August 10, 2019, 4:17pm UTC](https://discuss.elastic.co/t/repeated-record-every-time-new-events-happens/194714 "2019-08-10T16:17:44Z")
I have a very basic pipeline.config: input { file { path =\> \["/var/log/mylog.log"\] start\_position =\> "beginning" } } filter { } output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "test" } } let… --- ## [Logstash failed to index](https://discuss.elastic.co/t/logstash-failed-to-index/194712)
**Author:** [@Guyk](https://discuss.elastic.co/u/Guyk)\ **Replies:** 1\ **Last updated:** [August 10, 2019, 1:47pm UTC](https://discuss.elastic.co/t/logstash-failed-to-index/194712 "2019-08-10T13:47:28Z")
Hey guys, last week i decide to add Logsatsh service to my Elasticsearch cluster(3 nodes in AWS vpc) until now we are use to index new docs using the Elastic Api without using Logstash. The data type is JSON, and some… --- ## [Multiple line and fully unstructured log file](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616)
**Author:** [@Mohammad.ali](https://discuss.elastic.co/u/Mohammad.ali)\ **Replies:** 6\ **Last updated:** [August 10, 2019, 12:40pm UTC](https://discuss.elastic.co/t/multiple-line-and-fully-unstructured-log-file/194616 "2019-08-10T12:40:40Z")
Hello my friends I got some logfiles that are fully unstructured what I am so far done is just for the first line of each log which are DATE HOUR TIMEZONE and some information %{DATE\_YMD:login\_date} %{TIME:login\_time}… --- ## [Upsert / Update if exist, create if not](https://discuss.elastic.co/t/upsert-update-if-exist-create-if-not/194681)
**Author:** [@ELK-Developer](https://discuss.elastic.co/u/ELK-Developer)\ **Replies:** 2\ **Last updated:** [August 10, 2019, 6:59am UTC](https://discuss.elastic.co/t/upsert-update-if-exist-create-if-not/194681 "2019-08-10T06:59:33Z")
Hello, how do I use upsert, to create or update entries = documents via the output-plugin to elasticsearch, dependent on the document with the given id is already in elasticsearch or even not. My data will come from tw… --- ## [Invalid checksum in DLQ](https://discuss.elastic.co/t/invalid-checksum-in-dlq/194480)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 1\ **Last updated:** [August 8, 2019, 6:08pm UTC](https://discuss.elastic.co/t/invalid-checksum-in-dlq/194480 "2019-08-08T18:08:22Z")
I am getting an exception as follows: \[2019-08-08T09:52:26,478\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:dlqInstance Plugin: \ **Author:** [@Jim\_Thunder](https://discuss.elastic.co/u/Jim_Thunder)\ **Replies:** 1\ **Last updated:** [August 9, 2019, 7:58pm UTC](https://discuss.elastic.co/t/skip-first-two-grok-pattern-matches/194666 "2019-08-09T19:58:13Z") I am trying to parse out the last number of a field (that is a string). My grok pattern, currently, pulls the first number and puts it in the new field. How can make that grok pattern skip the first 2 numbers it comes ac… --- ## [Parse any field that has a specific name](https://discuss.elastic.co/t/parse-any-field-that-has-a-specific-name/194664)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 8\ **Last updated:** [August 9, 2019, 7:56pm UTC](https://discuss.elastic.co/t/parse-any-field-that-has-a-specific-name/194664 "2019-08-09T19:56:20Z")
Is it possible to parse any field in Json, under any root, that has a unix time format? I might have in my json file this field "timestamp": 1554940800000 it might be at any root in the json, with this naming conventi… --- ## [Group field from multiple rows based on common key](https://discuss.elastic.co/t/group-field-from-multiple-rows-based-on-common-key/194501)
**Author:** [@shri\_palchuri](https://discuss.elastic.co/u/shri_palchuri)\ **Replies:** 7\ **Last updated:** [August 9, 2019, 5:27pm UTC](https://discuss.elastic.co/t/group-field-from-multiple-rows-based-on-common-key/194501 "2019-08-09T17:27:53Z")
spotlight.csv: ENTITY\_ID,ENTITY\_TYPE,SPN\_FIRM\_ID,IA\_ID,BRANCH\_ID 1323575,AC,101,10503,11002 1323575,AC,101,10503,359006 Expecting Output JSON: { "keyInformation": { "entityId": 1323575, "entityType": "AC", "firm… --- ## [What is the number prefix in less-than/greater-than signs in the Message Field, before Date](https://discuss.elastic.co/t/what-is-the-number-prefix-in-less-than-greater-than-signs-in-the-message-field-before-date/194630)
**Author:** [@samk137](https://discuss.elastic.co/u/samk137)\ **Replies:** 8\ **Last updated:** [August 9, 2019, 4:23pm UTC](https://discuss.elastic.co/t/what-is-the-number-prefix-in-less-than-greater-than-signs-in-the-message-field-before-date/194630 "2019-08-09T16:23:30Z")
Hello, I'm new to the ELK stack and am having trouble figuring out something that's coming through in all my logs. The 'message' field stars with a \<##\>MMM DD HH:MM:SS I cannot for the life of me figure out what the \<#… --- ## [Where are the internal logstash logs when run in a Docker container?](https://discuss.elastic.co/t/where-are-the-internal-logstash-logs-when-run-in-a-docker-container/194623)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 2\ **Last updated:** [August 9, 2019, 2:37pm UTC](https://discuss.elastic.co/t/where-are-the-internal-logstash-logs-when-run-in-a-docker-container/194623 "2019-08-09T14:37:36Z")
Where are the internal Logstash logs? Docs state, " Logstash emits internal logs during its operation, which are placed in LS\_HOME/logs (or /var/log/logstash for DEB/RPM)." \[Ref 1\] When I open a shell into the cont… --- ## [Logstash join two csv files, one to many data](https://discuss.elastic.co/t/logstash-join-two-csv-files-one-to-many-data/193019)
**Author:** [@genice\_ladroma](https://discuss.elastic.co/u/genice_ladroma)\ **Replies:** 5\ **Last updated:** [August 9, 2019, 2:27pm UTC](https://discuss.elastic.co/t/logstash-join-two-csv-files-one-to-many-data/193019 "2019-08-09T14:27:29Z")
Hi, I was able to follow instructions on joining 2 csv files from this link However, this is just for joining one to one data, e.g. file1.csv id,name 1,john 2,jonathan file2.csv id,lastname 1,wick 2,smith thi… --- ## [IBM\\WebSphere was unexpected at this time](https://discuss.elastic.co/t/ibm-websphere-was-unexpected-at-this-time/193870)
**Author:** [@nim\_alan1](https://discuss.elastic.co/u/nim_alan1)\ **Replies:** 1\ **Last updated:** [August 9, 2019, 1:11pm UTC](https://discuss.elastic.co/t/ibm-websphere-was-unexpected-at-this-time/193870 "2019-08-09T13:11:10Z")
elk newbie here, trying to follow the logstash guide for version 7.3.0. when running the command as in guide "bin\\logstash -e 'input { stdin { } } output { stdout {} }'" I get the below error. "IBM\\WebSphere was unexpe… --- ## [Logstash JSON Parsing Error](https://discuss.elastic.co/t/logstash-json-parsing-error/194609)
**Author:** [@Tejas\_Shah](https://discuss.elastic.co/u/Tejas_Shah)\ **Replies:** 1\ **Last updated:** [August 9, 2019, 1:01pm UTC](https://discuss.elastic.co/t/logstash-json-parsing-error/194609 "2019-08-09T13:01:47Z")
Hi, I have sample json that I am pushing first into SQLite db and later to Logstash over TCP input. When I am using Logstash version 7.1.1 , everything is working fine. However, when I use Logstash 7.2.0 / 7.3.0 it thr… --- ## [Regexp in conditional](https://discuss.elastic.co/t/regexp-in-conditional/194223)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 9\ **Last updated:** [August 9, 2019, 12:59pm UTC](https://discuss.elastic.co/t/regexp-in-conditional/194223 "2019-08-09T12:59:22Z")
Hi, I'm trying to apply a tag based on the contents of a field. I've been trying it like this: if \[field.keyword\] =~ /^(TEST|test)-.\*$/ { mutate { add\_tag =\> \[ "TEST" \] } } } The above does not work, … --- ## [Active Directory - Parse Message Log and Username](https://discuss.elastic.co/t/active-directory-parse-message-log-and-username/194537)
**Author:** [@awataszko](https://discuss.elastic.co/u/awataszko)\ **Replies:** 1\ **Last updated:** [August 9, 2019, 12:54pm UTC](https://discuss.elastic.co/t/active-directory-parse-message-log-and-username/194537 "2019-08-09T12:54:08Z")
Hi, I am trying to build a simple report for something like AD group changes. Wondering if the "message" in the event ID can be parsed further?(ie have more fields) Also wondering if any regex can be applied. For exampl… --- ## [Reindexing Data](https://discuss.elastic.co/t/reindexing-data/193608)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [August 9, 2019, 12:22pm UTC](https://discuss.elastic.co/t/reindexing-data/193608 "2019-08-09T12:22:39Z")
I have about 300 indices of data, each of them consumining anywhere from a few hundred MBs to a couple tens of GBs. I'd like to reindex all of the data into rollup indices to reduce the number of indices mounted, which … --- ## [Logstash is not able connect after creating certificates for elasticsearch](https://discuss.elastic.co/t/logstash-is-not-able-connect-after-creating-certificates-for-elasticsearch/194547)
**Author:** [@akshay\_singh2](https://discuss.elastic.co/u/akshay_singh2)\ **Replies:** 3\ **Last updated:** [August 9, 2019, 11:41am UTC](https://discuss.elastic.co/t/logstash-is-not-able-connect-after-creating-certificates-for-elasticsearch/194547 "2019-08-09T11:41:16Z")
Hello Team, I have configured security for elasticsearch, but after it logstash is not able to connect elasticsearch. Getting below error: \[2019-08-08T23:13:27,353\]\[INFO \]\[org.apache.kafka.clients.consumer.internals.A… --- ## [Compare a field with a csv file and add info from csv in a new field](https://discuss.elastic.co/t/compare-a-field-with-a-csv-file-and-add-info-from-csv-in-a-new-field/191973)
**Author:** [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Replies:** 6\ **Last updated:** [August 9, 2019, 7:33am UTC](https://discuss.elastic.co/t/compare-a-field-with-a-csv-file-and-add-info-from-csv-in-a-new-field/191973 "2019-08-09T07:33:47Z")
Hello, I have a field named id in a grok filter, which I want to compare with a CSV file. For example : Index ELK : id=12 csv file : 11,home 12,garden 13,office ... How can I compare the id field with the first fi… --- ## [Logstash restart - Exception in pipelineworker](https://discuss.elastic.co/t/logstash-restart-exception-in-pipelineworker/193792)
**Author:** [@jmilot](https://discuss.elastic.co/u/jmilot)\ **Replies:** 6\ **Last updated:** [August 9, 2019, 7:48am UTC](https://discuss.elastic.co/t/logstash-restart-exception-in-pipelineworker/193792 "2019-08-09T07:48:09Z")
Hello, I have some logstash restart after this error : \[2019-08-05T14:12:13,259\]\[ERROR\]\[org.logstash.execution.WorkerLoop\] Exception in pipelineworker, the pipeline stopped processing new events, please check your filt… --- ## [Logstash plugin crashing when calling .jar](https://discuss.elastic.co/t/logstash-plugin-crashing-when-calling-jar/194375)
**Author:** [@mdobrin](https://discuss.elastic.co/u/mdobrin)\ **Replies:** 3\ **Last updated:** [August 9, 2019, 7:29am UTC](https://discuss.elastic.co/t/logstash-plugin-crashing-when-calling-jar/194375 "2019-08-09T07:29:57Z")
I am currently running Logstash 5.6.7, due to an older version of Elastic Stack being provided by our DBA (we don't expect to have an upgrade until next year). I've created a logstash plugin in jruby, which calls a jar … --- ## [Can we use stored procedure in logstash using jdbc](https://discuss.elastic.co/t/can-we-use-stored-procedure-in-logstash-using-jdbc/193497)
**Author:** [@raj.teaser](https://discuss.elastic.co/u/raj.teaser)\ **Replies:** 3\ **Last updated:** [August 9, 2019, 6:13am UTC](https://discuss.elastic.co/t/can-we-use-stored-procedure-in-logstash-using-jdbc/193497 "2019-08-09T06:13:32Z")
Hi, Can we use stored procedure in logstash using jdbc? As proc we are executing in oracle for update the data. You help will be most appreciated. Thanks & Regards, Raj --- ## [Conditional mutate of values after kv{} function](https://discuss.elastic.co/t/conditional-mutate-of-values-after-kv-function/194343)
**Author:** [@abhijitkottur](https://discuss.elastic.co/u/abhijitkottur)\ **Replies:** 3\ **Last updated:** [August 9, 2019, 5:50am UTC](https://discuss.elastic.co/t/conditional-mutate-of-values-after-kv-function/194343 "2019-08-09T05:50:46Z")
2019-08-08 08:41:47.919 INFO \[-,94a0b008e1c6714e,94a0b008e1c6714e,true\] 6 --- \[io-8164-exec-10\] a.c.f.w.r.c.controllers.VFSController : \[eventType:DELETE\_FILES\] \[storageName:playout-dr\] \[deleteResult:FILE\_NOT\_EXISTS\] \[nu… --- ## [Java Filter plugin development have hooks like register](https://discuss.elastic.co/t/java-filter-plugin-development-have-hooks-like-register/194518)
**Author:** [@Sunil\_V\_P](https://discuss.elastic.co/u/Sunil_V_P)\ **Replies:** 0\ **Last updated:** [August 9, 2019, 1:25am UTC](https://discuss.elastic.co/t/java-filter-plugin-development-have-hooks-like-register/194518 "2019-08-09T01:25:10Z")
Hi, I am new to logstash and being a java developer i wanted to write plugins in java so went ahead and explored the sample filter plugin in java. Our usecase is to enrich the events with the data which could be fetched… --- ## [Logstash Error: Unresolvable](https://discuss.elastic.co/t/logstash-error-unresolvable/194381)
**Author:** [@shenshensinian](https://discuss.elastic.co/u/shenshensinian)\ **Replies:** 3\ **Last updated:** [August 9, 2019, 12:46am UTC](https://discuss.elastic.co/t/logstash-error-unresolvable/194381 "2019-08-09T00:46:44Z")
logstash Error: Unresolvable failed to parse field \[timestamp\] of type \[date\] in document with id failed to parse date field \[Aug 8 2019 15:19:33\] with format How should I fix it? --- ## [Can we send logs to different outputs depending on filename they come from?](https://discuss.elastic.co/t/can-we-send-logs-to-different-outputs-depending-on-filename-they-come-from/192815)
**Author:** [@RITZ\_VERMA](https://discuss.elastic.co/u/RITZ_VERMA)\ **Replies:** 5\ **Last updated:** [August 8, 2019, 11:44pm UTC](https://discuss.elastic.co/t/can-we-send-logs-to-different-outputs-depending-on-filename-they-come-from/192815 "2019-08-08T23:44:39Z")
in logstash i am trying to send logs from different files to different elasticsearch depending on their filename. is this possible? --- ## [ES Input plugin gives invalid json data](https://discuss.elastic.co/t/es-input-plugin-gives-invalid-json-data/194500)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 5\ **Last updated:** [August 8, 2019, 8:23pm UTC](https://discuss.elastic.co/t/es-input-plugin-gives-invalid-json-data/194500 "2019-08-08T20:23:41Z")
Hi All I am using ES input and output as file, i get json data but this is multiline and not a valid json here is my conf file input { elasticsearch { hosts =\> "" index =\> "abcd" size =\> 10 scroll =\> "5m" … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=494) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=496)