# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=634 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 635 --- ## [Works in Grok Debugger but not in logstash config file](https://discuss.elastic.co/t/works-in-grok-debugger-but-not-in-logstash-config-file/160112)
**Author:** [@jamespb](https://discuss.elastic.co/u/jamespb)\ **Replies:** 1\ **Last updated:** [December 11, 2018, 6:51am UTC](https://discuss.elastic.co/t/works-in-grok-debugger-but-not-in-logstash-config-file/160112 "2018-12-11T06:51:16Z")
my filter works in Grok Debugger but not in logstash config file. pattern: (?\\[0-9\]+-\[0-9\]+-\[0-9\]+) (?\\[0-9\]+:\[0-9\]+:\[0-9\]+) (?\\[0-9\]+-\[0-9\]+) %{WORD:message\_type} "(?(.+?):)… --- ## [Logstash insert only first row](https://discuss.elastic.co/t/logstash-insert-only-first-row/160288)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 1\ **Last updated:** [December 11, 2018, 5:49am UTC](https://discuss.elastic.co/t/logstash-insert-only-first-row/160288 "2018-12-11T05:49:42Z")
I am getting data from Google API and I want to save it to ES using logstash. logstash conf file input { file { path =\> "/home/templetes/dt5.txt" start\_position =\> "beginning" sincedb\_path … --- ## [Can i prevent duplicate entry in logstash template?](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 2\ **Last updated:** [December 11, 2018, 5:08am UTC](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098 "2018-12-11T05:08:36Z")
I am using the template to insert data to ES. But some duplicate value is inserting, I want to prevent them baseed on reviewId in my example sample data \[ { "reviewId": "1", "displayName": "JOHN" }, { … --- ## [How to output to file is pattern format key|value in logstash](https://discuss.elastic.co/t/how-to-output-to-file-is-pattern-format-key-value-in-logstash/160281)
**Author:** [@keep](https://discuss.elastic.co/u/keep)\ **Replies:** 0\ **Last updated:** [December 11, 2018, 3:33am UTC](https://discuss.elastic.co/t/how-to-output-to-file-is-pattern-format-key-value-in-logstash/160281 "2018-12-11T03:33:41Z")
Hello I have a data where I am trying to convert The data look like this : 2018-10-25 11:59:20:037|2018-10-25 11:59:20:059|11111111111111111111|1234567890|null|/url/test-1|#test|000|20000|Success|null|Keep The rules l… --- ## [rabbitMQ Input plugin: PKIX path building failed](https://discuss.elastic.co/t/rabbitmq-input-plugin-pkix-path-building-failed/160257)
**Author:** [@Charles\_Peters](https://discuss.elastic.co/u/Charles_Peters)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 10:39pm UTC](https://discuss.elastic.co/t/rabbitmq-input-plugin-pkix-path-building-failed/160257 "2018-12-10T22:39:06Z")
Problem statement Hi guys, I receive the following error when trying to configure logstash client certificates: \[2018-12-10T22:10:32,949\]\[ERROR\]\[logstash.inputs.rabbitmq \] RabbitMQ connection error, will retry. {:erro… --- ## [Grok translation help](https://discuss.elastic.co/t/grok-translation-help/160216)
**Author:** [@David\_Moreno](https://discuss.elastic.co/u/David_Moreno)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 4:19pm UTC](https://discuss.elastic.co/t/grok-translation-help/160216 "2018-12-10T16:19:19Z")
Hello, I'm trying to set up my centralized syslog server. I'm forwarding my logs from file beats to my elk server using logstash. Im trying to get the syslog log information to be put into the fields of kibana but its sh… --- ## [No data load by logstash on sqlserver database](https://discuss.elastic.co/t/no-data-load-by-logstash-on-sqlserver-database/160213)
**Author:** [@Michel](https://discuss.elastic.co/u/Michel)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 4:15pm UTC](https://discuss.elastic.co/t/no-data-load-by-logstash-on-sqlserver-database/160213 "2018-12-10T16:15:49Z")
Hi, I installed logstash as a service on a redhat 7 server, everything seems to be correctly configured, but logstash does not load anything. I asked him to ask every minute, but nothing happens ! My logstash.conf : in… --- ## [Using HTTPS for WebHDFS Output Plugin](https://discuss.elastic.co/t/using-https-for-webhdfs-output-plugin/160202)
**Author:** [@dyongbcst](https://discuss.elastic.co/u/dyongbcst)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 3:15pm UTC](https://discuss.elastic.co/t/using-https-for-webhdfs-output-plugin/160202 "2018-12-10T15:15:48Z")
Hi, I am facing issues with WebHDFS. My organization uses WebHDFS on port 50470, which is kerberized and requires HTTPS. After following the thread in https://github.com/elastic/logstash/issues/8791, I am still facing … --- ## [Logstash doesn't do anything](https://discuss.elastic.co/t/logstash-doesnt-do-anything/160194)
**Author:** [@nu11ahnung](https://discuss.elastic.co/u/nu11ahnung)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 2:47pm UTC](https://discuss.elastic.co/t/logstash-doesnt-do-anything/160194 "2018-12-10T14:47:49Z")
I am trying to configure Logstash for my network. I try to forward logs from a windows server 2012 via winlogbeat to my centos7 servers logstash instance and from there into my elasticsearch. This somehow doesn't work. … --- ## [Logstash output influxdb configure problem](https://discuss.elastic.co/t/logstash-output-influxdb-configure-problem/159516)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 5\ **Last updated:** [December 9, 2018, 2:12pm UTC](https://discuss.elastic.co/t/logstash-output-influxdb-configure-problem/159516 "2018-12-09T14:12:08Z")
Influxdb server ip = 192.168.30.2 logstash server ip = 192.168.30.3 logstash.version"=\>"6.5.1 InfluxDB shell version: 1.6.4 cat /etc/logstash/conf.d/02-beats-input.conf input { beats { port =\> 5044 ssl =\> true t… --- ## [Date comparison in Logstash/Kibana](https://discuss.elastic.co/t/date-comparison-in-logstash-kibana/160178)
**Author:** [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 1:10pm UTC](https://discuss.elastic.co/t/date-comparison-in-logstash-kibana/160178 "2018-12-10T13:10:38Z")
Hi I am receiving an XML file as input where I have a date field in format CYYMMDD format (eg: 1181210). I need to perform a logic that if this date is less than system date then I have to show this as error. Can anyo… --- ## [Logstastash doesn't load data](https://discuss.elastic.co/t/logstastash-doesnt-load-data/160176)
**Author:** [@Jan\_Rehacek](https://discuss.elastic.co/u/Jan_Rehacek)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 1:01pm UTC](https://discuss.elastic.co/t/logstastash-doesnt-load-data/160176 "2018-12-10T13:01:54Z")
Hello, sorry to bother, but I am completely new to ES stuff. I just succesfully started ES and Kibana, but I have huge problems with LS. I am trying to simply load a csv file with a simple structure into Kibana trough L… --- ## [Logstash 6.x & type](https://discuss.elastic.co/t/logstash-6-x-type/159655)
**Author:** [@jamesl](https://discuss.elastic.co/u/jamesl)\ **Replies:** 4\ **Last updated:** [December 10, 2018, 11:28am UTC](https://discuss.elastic.co/t/logstash-6-x-type/159655 "2018-12-10T11:28:30Z")
Hello. I'm tired of the warnings in my Logstash log file and would like to fix it: \[2018-12-06T16:34:17,200\]\[WARN \]\[logstash.outputs.elasticsearch\] Detected a 6.x and above cluster: the type event field won't be used … --- ## [Logstash not discovering files](https://discuss.elastic.co/t/logstash-not-discovering-files/160155)
**Author:** [@shubh235](https://discuss.elastic.co/u/shubh235)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 11:05am UTC](https://discuss.elastic.co/t/logstash-not-discovering-files/160155 "2018-12-10T11:05:47Z")
Hello, My logstash does not discover files. I have removed the sincedb files but still no luck. The same configuration was working earlier. Below is my configuration file Could you please help me figure out what is w… --- ## [Need Help Grok filter](https://discuss.elastic.co/t/need-help-grok-filter/160141)
**Author:** [@bayoumben](https://discuss.elastic.co/u/bayoumben)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 10:12am UTC](https://discuss.elastic.co/t/need-help-grok-filter/160141 "2018-12-10T10:12:12Z")
Hi every one, i need your help to write a grok filter to this logs message: message: {"@timestamp":"2018-12-07T14:02:23.926Z","beat":{"hostname":"xxx","name":"yyyy","version":"5.4.1"},"fields":{"environment":"xxx","ip… --- ## [IndexPattern not working PLEASE HELP ME](https://discuss.elastic.co/t/indexpattern-not-working-please-help-me/160123)
**Author:** [@ebbeacar](https://discuss.elastic.co/u/ebbeacar)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 8:52am UTC](https://discuss.elastic.co/t/indexpattern-not-working-please-help-me/160123 "2018-12-10T08:52:19Z")
Hi everybody, I will try explaining my problem here and I hope someone can help me since I am trying to get a an answer for almost 2 weeks now. So I downloaded the following projects: -Elasticsearch -Logstash -Kibana … --- ## [Logstash 6.5.2 "Could not index event to Elasticsearch", Status = 400, reason failed to parse field \[host\] of type \[text\]](https://discuss.elastic.co/t/logstash-6-5-2-could-not-index-event-to-elasticsearch-status-400-reason-failed-to-parse-field-host-of-type-text/159908)
**Author:** [@MrFrost](https://discuss.elastic.co/u/MrFrost)\ **Replies:** 3\ **Last updated:** [December 10, 2018, 8:27am UTC](https://discuss.elastic.co/t/logstash-6-5-2-could-not-index-event-to-elasticsearch-status-400-reason-failed-to-parse-field-host-of-type-text/159908 "2018-12-10T08:27:32Z")
Hello, After updating ELK Stack from 6.3.0 to 6.5.2 some of my pipeline ceased to work correctly. No idea what may be wrong with this one, so some help would be great. pipeline: # Defines inputs for logs input { b… --- ## [Logstash remove @timestamp](https://discuss.elastic.co/t/logstash-remove-timestamp/160104)
**Author:** [@lovelk](https://discuss.elastic.co/u/lovelk)\ **Replies:** 1\ **Last updated:** [December 10, 2018, 7:32am UTC](https://discuss.elastic.co/t/logstash-remove-timestamp/160104 "2018-12-10T07:32:27Z")
Hello all, a logstash question: now, logs contain "time", so l want remove "@timestamp", Is there a way？ and Is there a better way? thx regards, lovelk --- ## [Suddenly index\_not\_found\_exception in logstash](https://discuss.elastic.co/t/suddenly-index-not-found-exception-in-logstash/160063)
**Author:** [@nuwancs](https://discuss.elastic.co/u/nuwancs)\ **Replies:** 0\ **Last updated:** [December 9, 2018, 7:01pm UTC](https://discuss.elastic.co/t/suddenly-index-not-found-exception-in-logstash/160063 "2018-12-09T19:01:34Z")
When I am trying a load test suddenly I got following error on logstash logs and no records process afterward. Before error occurred. 22569558 records were processed and 64.2gb space used by transactions-5k500s-2018.12… --- ## [JDBC input plugin error for cassandra for Logstash](https://discuss.elastic.co/t/jdbc-input-plugin-error-for-cassandra-for-logstash/159562)
**Author:** [@Nishad\_Nazar](https://discuss.elastic.co/u/Nishad_Nazar)\ **Replies:** 0\ **Last updated:** [December 5, 2018, 2:57pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-error-for-cassandra-for-logstash/159562 "2018-12-05T14:57:51Z")
Hi, I have followed almost every link in this community and all over, still not able to get Cassandra data to logstash using jdbc input plugin. Intially i USed cassandra jdbc driver from Maven repository and driver cla… --- ## [How to update config without logstash service stops?](https://discuss.elastic.co/t/how-to-update-config-without-logstash-service-stops/160088)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 3:56am UTC](https://discuss.elastic.co/t/how-to-update-config-without-logstash-service-stops/160088 "2018-12-10T03:56:54Z")
hi there! My necessity is to see the streaming logs in kibana. So, I use tcp input plugin in logstash confg. It works properly almost 1 year. But the problem is that I stream several apps logs to logstash now. If I … --- ## [Filebeat unable to publish events](https://discuss.elastic.co/t/filebeat-unable-to-publish-events/160083)
**Author:** [@itnazeer](https://discuss.elastic.co/u/itnazeer)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 3:03am UTC](https://discuss.elastic.co/t/filebeat-unable-to-publish-events/160083 "2018-12-10T03:03:35Z")
Filebeat which is on Centos 7 server is unable to publish events to logstash server while the filebeat on Centos 6 server is pusblishing fine with no issues. Please help on this. INFO pipeline/output.go:95 Connecti… --- ## [Error\_message=\>"undefined method \`sanitized'](https://discuss.elastic.co/t/error-message-undefined-method-sanitized/160076)
**Author:** [@haoyun](https://discuss.elastic.co/u/haoyun)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 1:53am UTC](https://discuss.elastic.co/t/error-message-undefined-method-sanitized/160076 "2018-12-10T01:53:48Z")
Hi there, I ELK version is 5.6, and logstash received the log.From log, it complains ""undefined method \`sanitized' " Every time, it's normal when it's just started. Slowly, this error will occur frequently in the log… --- ## [Breaking down a field](https://discuss.elastic.co/t/breaking-down-a-field/160023)
**Author:** [@alexsamad](https://discuss.elastic.co/u/alexsamad)\ **Replies:** 3\ **Last updated:** [December 10, 2018, 1:14am UTC](https://discuss.elastic.co/t/breaking-down-a-field/160023 "2018-12-10T01:14:52Z")
Hi so i have a grok { match =\> \[ "message", " +"(?\\[^"\]\*)" \*%{GREEDYDATA:msg}"\] } now this field could be "-" or "1234.999" I would like to end up with if possible http\_ybid to equal "-" or "1234.999" … --- ## [How to wildcard filter?](https://discuss.elastic.co/t/how-to-wildcard-filter/160075)
**Author:** [@teichae](https://discuss.elastic.co/u/teichae)\ **Replies:** 0\ **Last updated:** [December 10, 2018, 1:01am UTC](https://discuss.elastic.co/t/how-to-wildcard-filter/160075 "2018-12-10T01:01:57Z")
Hi. I have a question. I want to use a filter using a wildcard in logstash. /a.php /bphp /c.php /abc/abc.php /bcd/bcd.php I would like to drop this part because such unwanted logs as above are being collected. if "/\*… --- ## [Add new field based on another field](https://discuss.elastic.co/t/add-new-field-based-on-another-field/160064)
**Author:** [@Ivan44785372](https://discuss.elastic.co/u/Ivan44785372)\ **Replies:** 1\ **Last updated:** [December 9, 2018, 7:35pm UTC](https://discuss.elastic.co/t/add-new-field-based-on-another-field/160064 "2018-12-09T19:35:45Z")
Hello. I have such field in filebeat output docker.container.labels.app it looks like this: "docker" =\> { "container" =\> { "id" =\> "7549e137-3394-987d-f3cc-f79759851a62", "l… --- ## [Help to isolate email adresses in a single field](https://discuss.elastic.co/t/help-to-isolate-email-adresses-in-a-single-field/160049)
**Author:** [@Gabriel\_Neumann](https://discuss.elastic.co/u/Gabriel_Neumann)\ **Replies:** 0\ **Last updated:** [December 9, 2018, 2:25pm UTC](https://discuss.elastic.co/t/help-to-isolate-email-adresses-in-a-single-field/160049 "2018-12-09T14:25:08Z")
Hello, I'm new in Logstash and I need some help. I'm processing a big index with like 600 million records, that has a field called "message". This field data structure is variant, because it was created from different … --- ## [Does Grok Filter is needed when we apply Logstash Arcsight module?](https://discuss.elastic.co/t/does-grok-filter-is-needed-when-we-apply-logstash-arcsight-module/160029)
**Author:** [@phongpeeradejc](https://discuss.elastic.co/u/phongpeeradejc)\ **Replies:** 0\ **Last updated:** [December 9, 2018, 6:47am UTC](https://discuss.elastic.co/t/does-grok-filter-is-needed-when-we-apply-logstash-arcsight-module/160029 "2018-12-09T06:47:33Z")
I am trying to implement ELK as a SIEM as instructed here, where Arcsight logs are being sent as CEF format in Logstash Arcsight module. However, I understand that normally when the logs are parsing in Logstash, Grok Fi… --- ## [Trying to replace 'host' field (created by tcp input) with a hash](https://discuss.elastic.co/t/trying-to-replace-host-field-created-by-tcp-input-with-a-hash/160026)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 0\ **Last updated:** [December 9, 2018, 5:43am UTC](https://discuss.elastic.co/t/trying-to-replace-host-field-created-by-tcp-input-with-a-hash/160026 "2018-12-09T05:43:33Z")
Hi, Logstash version: 6.2.4 (RPM) What I'm trying to do is remove the "host" field which is created by the "tcp" input plugin. The reason I'm doing this is to use it to contain the host hash as specified in the ECS (ht… --- ## [Problem with .conf](https://discuss.elastic.co/t/problem-with-conf/160009)
**Author:** [@Steven\_Falcione](https://discuss.elastic.co/u/Steven_Falcione)\ **Replies:** 0\ **Last updated:** [December 8, 2018, 12:47pm UTC](https://discuss.elastic.co/t/problem-with-conf/160009 "2018-12-08T12:47:14Z")
Hello i have a problem with the conf file in logstash, when i delete the conf file that links my sql server to kibana, logstash still finds it and link them, and if i try to put another file logstash doesn't find it. Wh… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=633) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=635)