# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=735 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 736 --- ## [One of the entries does not get indexed](https://discuss.elastic.co/t/one-of-the-entries-does-not-get-indexed/136488)
**Author:** [@Milind\_Gawde](https://discuss.elastic.co/u/Milind_Gawde)\ **Replies:** 6\ **Last updated:** [June 25, 2018, 10:28am UTC](https://discuss.elastic.co/t/one-of-the-entries-does-not-get-indexed/136488 "2018-06-25T10:28:20Z")
I have a huge JSON file, having multilined, nested and indented JSON which is about 28000 lines. I am trying to figure out how to parse this JSON under one message field in Kibana. But it seems that the message gets spli… --- ## [Logstash Index not in Kibana](https://discuss.elastic.co/t/logstash-index-not-in-kibana/136999)
**Author:** [@heskez](https://discuss.elastic.co/u/heskez)\ **Replies:** 3\ **Last updated:** [June 25, 2018, 9:36am UTC](https://discuss.elastic.co/t/logstash-index-not-in-kibana/136999 "2018-06-25T09:36:56Z")
Hi there, I've noticed that mu Indexes aren't being added to Kibana anymore. I do have changed my Logstash conf files in the meantime. But still then I would only expect GROK parse failures and not my index to dissapear… --- ## [Parsing JSON error Logstash](https://discuss.elastic.co/t/parsing-json-error-logstash/137207)
**Author:** [@SuperK\_1860](https://discuss.elastic.co/u/SuperK_1860)\ **Replies:** 4\ **Last updated:** [June 25, 2018, 8:15am UTC](https://discuss.elastic.co/t/parsing-json-error-logstash/137207 "2018-06-25T08:15:42Z")
Hi all I am trying to import a JSON file but I get the following error. filter { json { # This setting must be a string # Expected string, got \["followUpDate", "dateOfBirth", "medical.dateFinalised", "medical.dateOf… --- ## [Need help on grok filter](https://discuss.elastic.co/t/need-help-on-grok-filter/134575)
**Author:** [@retxedue](https://discuss.elastic.co/u/retxedue)\ **Replies:** 15\ **Last updated:** [June 25, 2018, 7:37am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575 "2018-06-25T07:37:46Z")
Hi guys, i would like to ask if what is the correct filter using grok syntax for below output sample.out hostname:server1.sgdc.company.net|ipaddress:1XX.XX.XXX.XX|status:ACTIVE hostname:server2.sgdc.company.net|ipadd… --- ## [Making postgresql and logstash synchronized](https://discuss.elastic.co/t/making-postgresql-and-logstash-synchronized/137196)
**Author:** [@Iremsu](https://discuss.elastic.co/u/Iremsu)\ **Replies:** 0\ **Last updated:** [June 25, 2018, 7:00am UTC](https://discuss.elastic.co/t/making-postgresql-and-logstash-synchronized/137196 "2018-06-25T07:00:32Z")
Hi all, I am new on elk stack. I am using postgresql and the thing that i am trying to do it: i have a table called testtable with two columns key and value; whenever i add something in to these columns in my database i… --- ## [\[logstash.runner \] An unexpected error occurred! {:error=\>#, :backtrace=\>](https://discuss.elastic.co/t/logstash-runner-an-unexpected-error-occurred-error-backtrace/137001)
**Author:** [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\ **Replies:** 1\ **Last updated:** [June 25, 2018, 6:41am UTC](https://discuss.elastic.co/t/logstash-runner-an-unexpected-error-occurred-error-backtrace/137001 "2018-06-25T06:41:38Z")
My Config file is like this input { file { path =\> "D:\\logstash\\logstash-6.1.0\\bin\\repoprolog.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" ignore\_older =\> 0 codec =\> multiline { patt… --- ## [Appdynamics data into elasticsearch](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 6\ **Last updated:** [June 25, 2018, 4:50am UTC](https://discuss.elastic.co/t/appdynamics-data-into-elasticsearch/136922 "2018-06-25T04:50:29Z")
Hey Guys, I have been trying to get appdynamics (APM tool) data into elasticsearch through logstash. We are using appdynamics rest api for consuming the data through GET method. Everything looks ok, but when i run the… --- ## [Converting date format in logstash](https://discuss.elastic.co/t/converting-date-format-in-logstash/137026)
**Author:** [@Melvin\_Moses](https://discuss.elastic.co/u/Melvin_Moses)\ **Replies:** 2\ **Last updated:** [June 25, 2018, 4:54am UTC](https://discuss.elastic.co/t/converting-date-format-in-logstash/137026 "2018-06-25T04:54:56Z")
I have to extract a date from a log file, need to convert it to another date format and save it to Elasticsearch. My logstash code is like this filter { grok{ patterns\_dir =\> "./patterns" match =\> \[ "message", "%{pa… --- ## [Issue with ".keyword" in ruby filter](https://discuss.elastic.co/t/issue-with-keyword-in-ruby-filter/134743)
**Author:** [@Rajkumar\_Selvakumar](https://discuss.elastic.co/u/Rajkumar_Selvakumar)\ **Replies:** 1\ **Last updated:** [June 25, 2018, 2:36am UTC](https://discuss.elastic.co/t/issue-with-keyword-in-ruby-filter/134743 "2018-06-25T02:36:52Z")
I am using ruby to perform elastic search in logstash config. The below code works fine , but its returns more than one document because of text based search. response = client.search index: 'perf\_report\_by\_audit\_and\_j… --- ## [How to filter the logs using logstash](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898)
**Author:** [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\ **Replies:** 3\ **Last updated:** [June 24, 2018, 8:53pm UTC](https://discuss.elastic.co/t/how-to-filter-the-logs-using-logstash/136898 "2018-06-24T20:53:34Z")
I am using filebeat to send the below logs to logstash. But i need to know the best way to filter the fields. Trying csv and grok. But i am not successful to filter properly. 2018-05-21 09:00:03,384 INFO 369.4163.21 swg… --- ## [XPack Pipeline groups](https://discuss.elastic.co/t/xpack-pipeline-groups/137167)
**Author:** [@fracklen](https://discuss.elastic.co/u/fracklen)\ **Replies:** 0\ **Last updated:** [June 24, 2018, 8:45pm UTC](https://discuss.elastic.co/t/xpack-pipeline-groups/137167 "2018-06-24T20:45:58Z")
Hi I'm in the process of setting up ELK. Full X-Pack. Setting up pipelines using Kibana is really nice, but I now need Ansible/Chef/Puppet to propagate the pipeline id to the logstash.yml. How about adding a group fiel… --- ## [Adding Field From MySQL](https://discuss.elastic.co/t/adding-field-from-mysql/134328)
**Author:** [@TestCandidate](https://discuss.elastic.co/u/TestCandidate)\ **Replies:** 3\ **Last updated:** [June 4, 2018, 9:37am UTC](https://discuss.elastic.co/t/adding-field-from-mysql/134328 "2018-06-04T09:37:27Z")
The setup: I have an Ubuntu 16 server. I've downloaded the ELK stack and configured filebeat to run with logstash. In other words filebeat -\> logstash -\> elastic. (mainly the configuration is that is found here) The C… --- ## [Manticore::ResolutionFailure while sending logs to elasticsearch](https://discuss.elastic.co/t/manticore-resolutionfailure-while-sending-logs-to-elasticsearch/137113)
**Author:** [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\ **Replies:** 3\ **Last updated:** [June 24, 2018, 12:36pm UTC](https://discuss.elastic.co/t/manticore-resolutionfailure-while-sending-logs-to-elasticsearch/137113 "2018-06-24T12:36:29Z")
Hi, I am trying to send large amount of log files (around 700 files) from logstash to Elasticsearch. But when I start logstash, logs are not sending to the elasticsearch. It throws error something like this, {:timestam… --- ## [Processing logs based on severity](https://discuss.elastic.co/t/processing-logs-based-on-severity/137144)
**Author:** [@klapuch](https://discuss.elastic.co/u/klapuch)\ **Replies:** 0\ **Last updated:** [June 24, 2018, 12:33pm UTC](https://discuss.elastic.co/t/processing-logs-based-on-severity/137144 "2018-06-24T12:33:48Z")
Hi, I would like to store all application exceptions from PHP, including invalid login username or some database connection issue. It would be fine, if I can mark some exceptions as less relevant without bother me anym… --- ## [Ensure delivery of events from SQL DB using logstash](https://discuss.elastic.co/t/ensure-delivery-of-events-from-sql-db-using-logstash/136398)
**Author:** [@kenneth](https://discuss.elastic.co/u/kenneth)\ **Replies:** 7\ **Last updated:** [June 24, 2018, 10:15am UTC](https://discuss.elastic.co/t/ensure-delivery-of-events-from-sql-db-using-logstash/136398 "2018-06-24T10:15:48Z")
Hi, I am currently using JDBC input in my logstash configuration. I would like to know what is the most simple setup or common approach to ensure that the records extracted from DB are delivered to Elastic? I am leaning… --- ## [Unknown setting 'mutate' for grok](https://discuss.elastic.co/t/unknown-setting-mutate-for-grok/137139)
**Author:** [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Replies:** 1\ **Last updated:** [June 24, 2018, 10:06am UTC](https://discuss.elastic.co/t/unknown-setting-mutate-for-grok/137139 "2018-06-24T10:06:37Z")
Hi, I am getting this error while starting logstash. \[2018-06-24T09:55:59,748\]\[ERROR\]\[logstash.filters.grok \] Unknown setting 'mutate' for grok \[2018-06-24T09:55:59,752\]\[ERROR\]\[logstash.agent \] Failed to e… --- ## [Logstash & Beat :index\_not\_found\_exception](https://discuss.elastic.co/t/logstash-beat-index-not-found-exception/137137)
**Author:** [@bab](https://discuss.elastic.co/u/bab)\ **Replies:** 7\ **Last updated:** [June 24, 2018, 9:20am UTC](https://discuss.elastic.co/t/logstash-beat-index-not-found-exception/137137 "2018-06-24T09:20:52Z")
Hello, I am following theLogstash tutorial . This Command curl -XGET 'localhost:9200/logstash-2018.06.24/\_search?q=response=200' returns the following error, instead of the data shown in the tutorial: { "error" : { "… --- ## [Unable to convert field to date type with date filter](https://discuss.elastic.co/t/unable-to-convert-field-to-date-type-with-date-filter/137018)
**Author:** [@ishantanu](https://discuss.elastic.co/u/ishantanu)\ **Replies:** 8\ **Last updated:** [June 23, 2018, 1:45pm UTC](https://discuss.elastic.co/t/unable-to-convert-field-to-date-type-with-date-filter/137018 "2018-06-23T13:45:20Z")
Hi, I am unable to convert a date field to its type to date with date filter. Here's what I wrote for achieving that: mutate { add\_field =\> { "end\_date\_parse" =\> "%{end\_date} %{zimbra\_proc\_end\_time}" … --- ## [Logstash Data not visible on coordinate map](https://discuss.elastic.co/t/logstash-data-not-visible-on-coordinate-map/137104)
**Author:** [@Venkatesan\_S\_Venkate](https://discuss.elastic.co/u/Venkatesan_S_Venkate)\ **Replies:** 16\ **Last updated:** [June 23, 2018, 10:25am UTC](https://discuss.elastic.co/t/logstash-data-not-visible-on-coordinate-map/137104 "2018-06-23T10:25:43Z")
I created the .conf file and executed through the log stash. It filters using the geo ip. It creates the data according to requirement. When I open Kibana and try to visualise in the coordinate map it is not visible ev… --- ## [Logstash is loading duplicates](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773)
**Author:** [@databeata](https://discuss.elastic.co/u/databeata)\ **Replies:** 7\ **Last updated:** [June 23, 2018, 5:59am UTC](https://discuss.elastic.co/t/logstash-is-loading-duplicates/136773 "2018-06-23T05:59:06Z")
Create a log file in hdfs from hue with one record. \[image\] Run nifi job to copy log file from hdfs to elk server folder. \[image\] Run logstash job. input { file { path =\> "/path/\*.log" type =\> "test" start\_posi… --- ## [Load data in Powershell Windows](https://discuss.elastic.co/t/load-data-in-powershell-windows/137027)
**Author:** [@Yong\_Rhee](https://discuss.elastic.co/u/Yong_Rhee)\ **Replies:** 0\ **Last updated:** [June 22, 2018, 1:19pm UTC](https://discuss.elastic.co/t/load-data-in-powershell-windows/137027 "2018-06-22T13:19:33Z")
Trying to load Shakespear example data into elasticsearch using Windows powershell and got this auth error. (I downloaded file on my User/Documents folder by the way) I am connected to Elasticsearch port(9200) and Kiban… --- ## [Problems with WIP Event Hubs output plugin for logstash](https://discuss.elastic.co/t/problems-with-wip-event-hubs-output-plugin-for-logstash/137083)
**Author:** [@jedthe3rd](https://discuss.elastic.co/u/jedthe3rd)\ **Replies:** 0\ **Last updated:** [June 22, 2018, 8:47pm UTC](https://discuss.elastic.co/t/problems-with-wip-event-hubs-output-plugin-for-logstash/137083 "2018-06-22T20:47:40Z")
I am currently developing an Azure Event Hubs output plugin for Logstash and I have ran into a problem where after running for around 30 minutes logstash will stop accepting flushes from telegraf for a period of time (3-… --- ## [Method to timestamp my logstash events](https://discuss.elastic.co/t/method-to-timestamp-my-logstash-events/135888)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 8\ **Last updated:** [June 22, 2018, 6:11pm UTC](https://discuss.elastic.co/t/method-to-timestamp-my-logstash-events/135888 "2018-06-22T18:11:01Z")
Hello everybody, I have this input log : INFO;0000;000001;\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* INFO;0000;000002;\* … --- ## [Issue parsing non-standard timestamp (Cisco router log)](https://discuss.elastic.co/t/issue-parsing-non-standard-timestamp-cisco-router-log/133908)
**Author:** [@AJ\_NOURI](https://discuss.elastic.co/u/AJ_NOURI)\ **Replies:** 4\ **Last updated:** [June 22, 2018, 4:16pm UTC](https://discuss.elastic.co/t/issue-parsing-non-standard-timestamp-cisco-router-log/133908 "2018-06-22T16:16:57Z")
I am trying to parse logs from Cisco router with the following format: 000022: Jan 02 15:42:11.048:%LINK-5-CHANGED: Interface Ethernet0, changed state to administratively down Notice the particular timestamp: Jan 02 1… --- ## [Grok filter question: don't understand %{COMBINEDAPACHELOG}+](https://discuss.elastic.co/t/grok-filter-question-dont-understand-combinedapachelog/136982)
**Author:** [@keli](https://discuss.elastic.co/u/keli)\ **Replies:** 1\ **Last updated:** [June 22, 2018, 3:40pm UTC](https://discuss.elastic.co/t/grok-filter-question-dont-understand-combinedapachelog/136982 "2018-06-22T15:40:49Z")
I found a very good grok filter tutorial here. Every concept is well explained except this apache grok filter configrathton. grok { match =\> \[ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra\_fields}", … --- ## [Support for VNF Event Stream (VES) data model](https://discuss.elastic.co/t/support-for-vnf-event-stream-ves-data-model/137046)
**Author:** [@marc1](https://discuss.elastic.co/u/marc1)\ **Replies:** 0\ **Last updated:** [June 22, 2018, 3:05pm UTC](https://discuss.elastic.co/t/support-for-vnf-event-stream-ves-data-model/137046 "2018-06-22T15:05:12Z")
Greetings, We are looking to use a codec for the VNF Event Stream (VES) data model from the ONAP & OPNVF communities. See link for details: https://wiki.opnfv.org/display/ves/VES+Home I am curious to know if anyone el… --- ## [Multiline Plugin - metadata missing from last line](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725)
**Author:** [@MonicaL](https://discuss.elastic.co/u/MonicaL)\ **Replies:** 3\ **Last updated:** [June 22, 2018, 2:54pm UTC](https://discuss.elastic.co/t/multiline-plugin-metadata-missing-from-last-line/136725 "2018-06-22T14:54:05Z")
Hi, I am trying to combine multiple lines into one event using the multiline codec. I also need some filename metadata. The lines are combining properly. However, the metadata of the filename is lost for the last event … --- ## [Default values for @metadata?](https://discuss.elastic.co/t/default-values-for-metadata/136303)
**Author:** [@A\_B](https://discuss.elastic.co/u/A_B)\ **Replies:** 6\ **Last updated:** [June 22, 2018, 2:23pm UTC](https://discuss.elastic.co/t/default-values-for-metadata/136303 "2018-06-22T14:23:59Z")
Hi, is there a recommended way to set some default @metadata fields in Logstash for all documents? I'm using this logstash output config to "shard" different types of logs to their own indices as ESv6 no longer supports… --- ## [Error timezone](https://discuss.elastic.co/t/error-timezone/136993)
**Author:** [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\ **Replies:** 1\ **Last updated:** [June 22, 2018, 2:16pm UTC](https://discuss.elastic.co/t/error-timezone/136993 "2018-06-22T14:16:23Z")
Hello I have a little problem that I can not solve for a while. I have a field: date: 20180612-2357 in string. I wanted to convert this field to date. I did like this (see the code after) but unfortunately, he convert… --- ## [Split multiple lines (using Grok filter) in message so they display as newlines in Kibana](https://discuss.elastic.co/t/split-multiple-lines-using-grok-filter-in-message-so-they-display-as-newlines-in-kibana/136431)
**Author:** [@Warren](https://discuss.elastic.co/u/Warren)\ **Replies:** 3\ **Last updated:** [June 22, 2018, 12:41pm UTC](https://discuss.elastic.co/t/split-multiple-lines-using-grok-filter-in-message-so-they-display-as-newlines-in-kibana/136431 "2018-06-22T12:41:16Z")
So i have a message in Kibana and it looks like this: \[2018-06-07 11:02:33\] testing.ERROR: Class 'App\\Http\\Controllers\\Deal' not found {"exception":"\[object\] (Symfony\\Component\\Debug\\Exception\\FatalThrowableError(code: … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=734) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=736)