# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=676 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 677 --- ## [ETA on Support for ubuntu 18.04 Server?](https://discuss.elastic.co/t/eta-on-support-for-ubuntu-18-04-server/150844)
**Author:** [@SpaceMoose](https://discuss.elastic.co/u/SpaceMoose)\ **Replies:** 4\ **Last updated:** [October 3, 2018, 1:52pm UTC](https://discuss.elastic.co/t/eta-on-support-for-ubuntu-18-04-server/150844 "2018-10-03T13:52:47Z")
Hi... I've had a go at getting ELK up and running on ubuntu 18.04 Server (with Java 8). I installed elasticsearch and logstash a day apart and got elasticsearch 6.4.1 and logstash 6.4.2 as a result. Both installed usi… --- ## [Grok match error](https://discuss.elastic.co/t/grok-match-error/150843)
**Author:** [@erkant](https://discuss.elastic.co/u/erkant)\ **Replies:** 2\ **Last updated:** [October 3, 2018, 1:08pm UTC](https://discuss.elastic.co/t/grok-match-error/150843 "2018-10-03T13:08:17Z")
hi, ./logstash -f /etc/logstash/conf.d/logstash.conf example log: \[2018-09-26T10:04:12,150\]\[INFO \]\[o.e.e.NodeEnvironment \] \[sezen\] using \[1\] data paths, mounts \[(rootfs)\]\], net usable\_space \[43.8gb\], net total\_spac… --- ## [Add field of one grok in another grok](https://discuss.elastic.co/t/add-field-of-one-grok-in-another-grok/150304)
**Author:** [@richagautam](https://discuss.elastic.co/u/richagautam)\ **Replies:** 1\ **Last updated:** [October 3, 2018, 1:00pm UTC](https://discuss.elastic.co/t/add-field-of-one-grok-in-another-grok/150304 "2018-10-03T13:00:42Z")
Hi Team, I am using multiple grok to extract multiple pattern from the log file. Below is the sample grok filters which I am using grok { message =\> {"%{NUMBER:Filed1} %{GREEDYDATA:message}" } } grok { message =\> … --- ## [Parsing xml log problem](https://discuss.elastic.co/t/parsing-xml-log-problem/148269)
**Author:** [@motodaddo](https://discuss.elastic.co/u/motodaddo)\ **Replies:** 4\ **Last updated:** [October 3, 2018, 12:42pm UTC](https://discuss.elastic.co/t/parsing-xml-log-problem/148269 "2018-10-03T12:42:06Z")
Hi, i'm trying to parse an xml log (mysql audit log) but i'm having problem on how the otput is done. My goal is to parse an xml and write it as is, but seems that logstash does not read/write input file in it's origin… --- ## [Lohstash output file locking in Windows preventing deletion](https://discuss.elastic.co/t/lohstash-output-file-locking-in-windows-preventing-deletion/150892)
**Author:** [@motodaddo](https://discuss.elastic.co/u/motodaddo)\ **Replies:** 0\ **Last updated:** [October 3, 2018, 12:40pm UTC](https://discuss.elastic.co/t/lohstash-output-file-locking-in-windows-preventing-deletion/150892 "2018-10-03T12:40:23Z")
Hello! I'm testing logstash on Windows and i've noticed that using file as output plugin, the output file created is locked by the process (java) preventing its deletion. Is there any option to let deletion of output f… --- ## [Formatting issues with parsing JSON from MongoDB](https://discuss.elastic.co/t/formatting-issues-with-parsing-json-from-mongodb/150847)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 5\ **Last updated:** [October 3, 2018, 10:54am UTC](https://discuss.elastic.co/t/formatting-issues-with-parsing-json-from-mongodb/150847 "2018-10-03T10:54:27Z")
I am trying to parse JSON (from MongoDB) to Kibana through Logstash. I keep getting all the data in a "log\_entry" field. This field does not exist in my data but rather is being created by Logstash. In my CONF file, I h… --- ## [Optional field in dissect plugin](https://discuss.elastic.co/t/optional-field-in-dissect-plugin/149263)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 1\ **Last updated:** [October 3, 2018, 10:53am UTC](https://discuss.elastic.co/t/optional-field-in-dissect-plugin/149263 "2018-10-03T10:53:23Z")
I have a dissect plugin in my filter section of the pipeline as follows: dissect { mapping =\> { "message" =\> "%{ts} %{+ts} %{+ts} %{host} %{prog}\[%{pid}\]: %{message}" } } The pid … --- ## [Hot to concatenate custom string in aggregate map variable](https://discuss.elastic.co/t/hot-to-concatenate-custom-string-in-aggregate-map-variable/150856)
**Author:** [@pavlik](https://discuss.elastic.co/u/pavlik)\ **Replies:** 0\ **Last updated:** [October 3, 2018, 10:19am UTC](https://discuss.elastic.co/t/hot-to-concatenate-custom-string-in-aggregate-map-variable/150856 "2018-10-03T10:19:21Z")
Hi, I'm not able to write a working code for the aggregate plugin that would allow me just to add some custom strings between existing message fields. The following one is working properly: map\['dns\_result'\] = \[map\['fr… --- ## [Help with a ruby filter](https://discuss.elastic.co/t/help-with-a-ruby-filter/150156)
**Author:** [@dmccuk](https://discuss.elastic.co/u/dmccuk)\ **Replies:** 3\ **Last updated:** [October 3, 2018, 10:07am UTC](https://discuss.elastic.co/t/help-with-a-ruby-filter/150156 "2018-10-03T10:07:20Z")
Hi, I'm getting the following message in logstash while sending metricbeat data to elasticsearch: Sep 26 12:00:04 eurvlii16592 start.sh\[58299\]: \[2018-09-26T12:00:04,355\]\[WARN \]\[logstash.outputs.elasticsearch\] Could no… --- ## [Date Format is Malformed](https://discuss.elastic.co/t/date-format-is-malformed/150832)
**Author:** [@saramali](https://discuss.elastic.co/u/saramali)\ **Replies:** 4\ **Last updated:** [October 3, 2018, 8:49am UTC](https://discuss.elastic.co/t/date-format-is-malformed/150832 "2018-10-03T08:49:07Z")
I'm indexing logs from a server to elastic search. The timestamp of the logs is the format 2018-09-18 11:41:01,648160559 The grok which I am using to match it is %{TIMESTAMP\_ISO8601:logTimestamp} Since I want the log… --- ## [Dropped UDP packets on Logstash Inputs (3 different UDP inputs)](https://discuss.elastic.co/t/dropped-udp-packets-on-logstash-inputs-3-different-udp-inputs/146922)
**Author:** [@TheZeke](https://discuss.elastic.co/u/TheZeke)\ **Replies:** 5\ **Last updated:** [October 3, 2018, 8:20am UTC](https://discuss.elastic.co/t/dropped-udp-packets-on-logstash-inputs-3-different-udp-inputs/146922 "2018-10-03T08:20:59Z")
Netdata Alert: 1m ipv4 udp receive buffer errors | 23650 errors I have a cluster of three servers that are all part of an ELK (Elasticsearch Logstash Kibana) cluster receiving netflow/sflow/ipfix data. Everything appe… --- ## [Logstash Configuration Error: "Expected one of #,"](https://discuss.elastic.co/t/logstash-configuration-error-expected-one-of/150827)
**Author:** [@rohini\_basu](https://discuss.elastic.co/u/rohini_basu)\ **Replies:** 2\ **Last updated:** [October 3, 2018, 6:48am UTC](https://discuss.elastic.co/t/logstash-configuration-error-expected-one-of/150827 "2018-10-03T06:48:54Z")
Hello, I am using Logstash to ingest CSV files into Elasticsearch. I am using a Linux Machine (CentOS7). I have Latitude and Longitude information in my csv. Hence I have created a mapping like below: PUT mks\_locdata… --- ## [Lumberjack output plugin posts duplicate entries when there is a connection error](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615)
**Author:** [@manz](https://discuss.elastic.co/u/manz)\ **Replies:** 4\ **Last updated:** [October 3, 2018, 6:32am UTC](https://discuss.elastic.co/t/lumberjack-output-plugin-posts-duplicate-entries-when-there-is-a-connection-error/149615 "2018-10-03T06:32:12Z")
Duplicate entries are posted when the following error happens. Client write error, trying connect {:e=\>#\, :backtrace=\>\["org/jruby/ext/openssl/SSLSocket.java:950:in syswrite'", "/root/… --- ## [Logstash is not able to import CSV files](https://discuss.elastic.co/t/logstash-is-not-able-to-import-csv-files/150480)
**Author:** [@Andy\_K](https://discuss.elastic.co/u/Andy_K)\ **Replies:** 6\ **Last updated:** [October 3, 2018, 3:51am UTC](https://discuss.elastic.co/t/logstash-is-not-able-to-import-csv-files/150480 "2018-10-03T03:51:49Z")
I am trying to import a CSV file to elasticsearch using logstash. There is no error displayed on the screen after running the logstash.bat. However, there is no index created in ealsticsearch. Please help. Below is the … --- ## [Logstash service starts but underlying log error shows](https://discuss.elastic.co/t/logstash-service-starts-but-underlying-log-error-shows/150730)
**Author:** [@skagnola](https://discuss.elastic.co/u/skagnola)\ **Replies:** 1\ **Last updated:** [October 2, 2018, 9:06pm UTC](https://discuss.elastic.co/t/logstash-service-starts-but-underlying-log-error-shows/150730 "2018-10-02T21:06:04Z")
Hello! I am looking for some help with an older version of Logstash - v2.0.0. Disclaimer: I am not very familiar with Logstash customization / optomization and much of what I am encountering is new to me. The service i… --- ## [Logstash truncates message](https://discuss.elastic.co/t/logstash-truncates-message/150722)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 0\ **Last updated:** [October 2, 2018, 2:57pm UTC](https://discuss.elastic.co/t/logstash-truncates-message/150722 "2018-10-02T14:57:12Z")
One of my container outputs a long array of hashes and logstash logs that array into logstash-\* index, however due to length of the array, logstash also truncates message as well, causing my app not to work properly (as … --- ## [Logstash csv output filter](https://discuss.elastic.co/t/logstash-csv-output-filter/150711)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 1\ **Last updated:** [October 2, 2018, 2:18pm UTC](https://discuss.elastic.co/t/logstash-csv-output-filter/150711 "2018-10-02T14:18:04Z")
Hello. I am querying my ES using Logstash. I want to output the id of the record from ES into my csv file. "\_index": "data", "\_type": "default", "\_id": "vANfNGYB9XD0VZRJUFfy", "\_version": 1, "\_score": null, "\_so… --- ## [Logstash km filter issue](https://discuss.elastic.co/t/logstash-km-filter-issue/150426)
**Author:** [@gopisa](https://discuss.elastic.co/u/gopisa)\ **Replies:** 1\ **Last updated:** [October 2, 2018, 12:58pm UTC](https://discuss.elastic.co/t/logstash-km-filter-issue/150426 "2018-10-02T12:58:36Z")
Hi All, I am facing KV filter issue. any one please help me on this. I am using kv filter. In that configuration value\_split =\> ":\\s" like that i spited my log file . if the any log file contain URL(https://googlr.com\\… --- ## [RemoteTransportException/SearchContextMissingException](https://discuss.elastic.co/t/remotetransportexception-searchcontextmissingexception/150679)
**Author:** [@npontes](https://discuss.elastic.co/u/npontes)\ **Replies:** 1\ **Last updated:** [October 2, 2018, 12:30pm UTC](https://discuss.elastic.co/t/remotetransportexception-searchcontextmissingexception/150679 "2018-10-02T12:30:51Z")
Hello, I'm currently using Logstash (2.1.1) to extract data from Elasticsearch (1.4.4) to a CSV. However I'm having a few errors: {:timestamp=\>"2018-10-02T12:37:24.368000+0200", :message=\>"A plugin had an unrecoverab… --- ## [LogStash-aggregate-filter events in random order](https://discuss.elastic.co/t/logstash-aggregate-filter-events-in-random-order/150676)
**Author:** [@sialagio](https://discuss.elastic.co/u/sialagio)\ **Replies:** 0\ **Last updated:** [October 2, 2018, 10:32am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-events-in-random-order/150676 "2018-10-02T10:32:17Z")
Hello, Sorry if the is a doublon, I searched, but did not find same topic. We are working on adding information to events having MessageTemplate = HTTP {Method} {Path} {StatusCode}. Information to add would come from S… --- ## [Log file to json](https://discuss.elastic.co/t/log-file-to-json/150526)
**Author:** [@erkant](https://discuss.elastic.co/u/erkant)\ **Replies:** 3\ **Last updated:** [October 2, 2018, 10:16am UTC](https://discuss.elastic.co/t/log-file-to-json/150526 "2018-10-02T10:16:58Z")
Hi, I installed elasticsearch-6.4.1, kibana-6.4.1, logstash-6.4.1 and filebeat.6-4.1 filebeat.yml #=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most… --- ## [Logstash syntax error in configuration file](https://discuss.elastic.co/t/logstash-syntax-error-in-configuration-file/150326)
**Author:** [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Replies:** 4\ **Last updated:** [October 2, 2018, 6:09am UTC](https://discuss.elastic.co/t/logstash-syntax-error-in-configuration-file/150326 "2018-10-02T06:09:50Z")
Hello Team, I am using ELK 6.4.0 and Beats (Filebeat, Metricbeat) 6.4.0. My architecture is Beat-\>logstash-\>elasticserach-\>kibana. I am getting syntax error in logstash file. I have troubleshooted it a lot but no succe… --- ## [Sqlite plugin 3.0.4 failed to start and it seems a bug](https://discuss.elastic.co/t/sqlite-plugin-3-0-4-failed-to-start-and-it-seems-a-bug/150305)
**Author:** [@qiulang](https://discuss.elastic.co/u/qiulang)\ **Replies:** 1\ **Last updated:** [October 2, 2018, 1:08am UTC](https://discuss.elastic.co/t/sqlite-plugin-3-0-4-failed-to-start-and-it-seems-a-bug/150305 "2018-10-02T01:08:14Z")
I am kinda sure this is a bug! Because if I just use logstash-input-jdbc without using sqlite plugin I can import my sqlite data successfully! Someone opened an issue at Logstash with sqllite3 problem · Issue #9788 · … --- ## [\_Grokparsefailure even though pattern works in Grok Debugger](https://discuss.elastic.co/t/grokparsefailure-even-though-pattern-works-in-grok-debugger/149179)
**Author:** [@Samvid\_Kulkarni](https://discuss.elastic.co/u/Samvid_Kulkarni)\ **Replies:** 2\ **Last updated:** [October 1, 2018, 11:00pm UTC](https://discuss.elastic.co/t/grokparsefailure-even-though-pattern-works-in-grok-debugger/149179 "2018-10-01T23:00:48Z")
I am trying to parse IIS logs and I did test my pattern in the Grok Dubugger (inside Kibana) and it is working perfectly fine but logstash is giving me \_grokparsefailure. Here is my config file input { file { type… --- ## [Trouble importing data to Logstash using tcp input](https://discuss.elastic.co/t/trouble-importing-data-to-logstash-using-tcp-input/150612)
**Author:** [@kyle.upton](https://discuss.elastic.co/u/kyle.upton)\ **Replies:** 0\ **Last updated:** [October 1, 2018, 8:33pm UTC](https://discuss.elastic.co/t/trouble-importing-data-to-logstash-using-tcp-input/150612 "2018-10-01T20:33:03Z")
Just to give a little background about what I'm trying to do, I want to monitor my servers system metrics using NetData and then import that data into an Elastic Stack. Then, using machine learning, I want to detect anom… --- ## [Arcsight Plugin Port Unopened](https://discuss.elastic.co/t/arcsight-plugin-port-unopened/150610)
**Author:** [@Eric\_B](https://discuss.elastic.co/u/Eric_B)\ **Replies:** 0\ **Last updated:** [October 1, 2018, 8:17pm UTC](https://discuss.elastic.co/t/arcsight-plugin-port-unopened/150610 "2018-10-01T20:17:40Z")
Hi. I have the module in my logstash.yml. modules: name: arcsight var.kibana.host: "soc-broker06-prod-asb.ei.rutgers.edu:5601" var.inputs: "smartconnector" var.elasticsearch.hosts: "localhost:9200" var.input.sma… --- ## [Output to stdout of timestamp format to localtime](https://discuss.elastic.co/t/output-to-stdout-of-timestamp-format-to-localtime/147783)
**Author:** [@Michael\_Savettiere](https://discuss.elastic.co/u/Michael_Savettiere)\ **Replies:** 0\ **Last updated:** [September 7, 2018, 10:40pm UTC](https://discuss.elastic.co/t/output-to-stdout-of-timestamp-format-to-localtime/147783 "2018-09-07T22:40:16Z")
I simply want the timestamp from logstash and the message sent to it from filebeat. The time showing is in UTC but I want it displayed in localtime. How do I do this? Here is the simple pipeline configuration input { … --- ## [Get no of events in memory and back pressure details](https://discuss.elastic.co/t/get-no-of-events-in-memory-and-back-pressure-details/150595)
**Author:** [@meetdave2611997](https://discuss.elastic.co/u/meetdave2611997)\ **Replies:** 0\ **Last updated:** [October 1, 2018, 6:40pm UTC](https://discuss.elastic.co/t/get-no-of-events-in-memory-and-back-pressure-details/150595 "2018-10-01T18:40:38Z")
Hi How do I know the current no of events which are in memory queue and logstash nodes are putting back pressure on inputs? Basically I want to check my logstash node's capacity of getting and processing events so that … --- ## [Filter out lines with only a TAB](https://discuss.elastic.co/t/filter-out-lines-with-only-a-tab/150589)
**Author:** [@hangstl](https://discuss.elastic.co/u/hangstl)\ **Replies:** 0\ **Last updated:** [October 1, 2018, 5:25pm UTC](https://discuss.elastic.co/t/filter-out-lines-with-only-a-tab/150589 "2018-10-01T17:25:31Z")
I have to ingest logfiles that contain lines with only a TAB in it. Is it possible to get rid of those lines using the drop() filter? Something like this: filter { ... if \[message\] =~ /^\\s\*$/ { drop { } } ... } How d… --- ## [Can't understand an error from AWS S3 input in Logstash](https://discuss.elastic.co/t/cant-understand-an-error-from-aws-s3-input-in-logstash/150579)
**Author:** [@KIVagant](https://discuss.elastic.co/u/KIVagant)\ **Replies:** 1\ **Last updated:** [October 1, 2018, 2:48pm UTC](https://discuss.elastic.co/t/cant-understand-an-error-from-aws-s3-input-in-logstash/150579 "2018-10-01T14:48:03Z")
Hello. I have a problem with the Logstash AWS S3 input plugin. I have several \*.conf files in /etc/logstash/conf.d/ and can't even understand which of them is causing the error. When I run each of them separately with D… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=675) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=677)