# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=483 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 484 --- ## [Elasticsearch output: LogStash::Error: timestamp field is missing](https://discuss.elastic.co/t/elasticsearch-output-logstash-timestamp-field-is-missing/197888)
**Author:** [@ouss](https://discuss.elastic.co/u/ouss)\ **Replies:** 2\ **Last updated:** [September 4, 2019, 10:00am UTC](https://discuss.elastic.co/t/elasticsearch-output-logstash-timestamp-field-is-missing/197888 "2019-09-04T10:00:02Z")
Hello, I have an elasticsearch output working without any problem elasticsearch { hosts =\> \[ "${LOGS\_ELASTICSEARCH\_CLUSTER}:443" \] ssl =\> true index =\> "cwl-%{+YYYY.MM.dd}" document\_type =\> "log" } I wanted to… --- ## [How to use ruby code to realize “i = i + 1" operation](https://discuss.elastic.co/t/how-to-use-ruby-code-to-realize-i-i-1-operation/197646)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 6\ **Last updated:** [September 4, 2019, 9:19am UTC](https://discuss.elastic.co/t/how-to-use-ruby-code-to-realize-i-i-1-operation/197646 "2019-09-04T09:19:30Z")
Hi Experts, I' m using ruby code to parse my target log. Here is the source code: if "\_xmlparsefailure" not in \[tags\]{ ruby { code =\> ' res = \[\] … --- ## [Calculate duration with ruby code](https://discuss.elastic.co/t/calculate-duration-with-ruby-code/197654)
**Author:** [@Alisa\_Faingold](https://discuss.elastic.co/u/Alisa_Faingold)\ **Replies:** 8\ **Last updated:** [September 4, 2019, 6:04am UTC](https://discuss.elastic.co/t/calculate-duration-with-ruby-code/197654 "2019-09-04T06:04:56Z")
Hi all, After several attempts to work with the "elapsed" filter to calculate the duration between events, I try to do so using Ruby code. I can't quite figure out how I can get the time of a particular event based on … --- ## [Two different log files at different location](https://discuss.elastic.co/t/two-different-log-files-at-different-location/197456)
**Author:** [@Saravana37](https://discuss.elastic.co/u/Saravana37)\ **Replies:** 2\ **Last updated:** [September 4, 2019, 5:08am UTC](https://discuss.elastic.co/t/two-different-log-files-at-different-location/197456 "2019-09-04T05:08:04Z")
Hello Techies, Please help me since i am new to ELK stack. I have two kinds of log files with different names on the different folders. I am trying to Pull the data to Logstash and providing two Grok filters match for … --- ## [Logstash output index names are coming out incorrectly: %{\[@metadata\]\[beat\]}-%{\[@metadata\]\[version\]}-%{+YYYY.MM.dd}](https://discuss.elastic.co/t/logstash-output-index-names-are-coming-out-incorrectly-metadata-beat-metadata-version-yyyy-mm-dd/197974)
**Author:** [@bpnet](https://discuss.elastic.co/u/bpnet)\ **Replies:** 0\ **Last updated:** [September 4, 2019, 4:31am UTC](https://discuss.elastic.co/t/logstash-output-index-names-are-coming-out-incorrectly-metadata-beat-metadata-version-yyyy-mm-dd/197974 "2019-09-04T04:31:59Z")
ELK stack was on version 6.3 prior to upgrading to 6.8 and then 7.2. Just to preface, I'm not very good with using ELK (I took over a previous sysadmin who set this up), so I can't tell whether there's anything wrong wit… --- ## [Cannot reconnect to new broker ID from kafka cluster if Leader Kafka fail](https://discuss.elastic.co/t/cannot-reconnect-to-new-broker-id-from-kafka-cluster-if-leader-kafka-fail/196606)
**Author:** [@vidocq](https://discuss.elastic.co/u/vidocq)\ **Replies:** 2\ **Last updated:** [September 4, 2019, 3:09am UTC](https://discuss.elastic.co/t/cannot-reconnect-to-new-broker-id-from-kafka-cluster-if-leader-kafka-fail/196606 "2019-09-04T03:09:18Z")
Dear everyone! I have a logstash get log from 3 instances apache kafka and zookeeper cluster. When I test turn off the Leader zookeeper from cluster. Logstash still connect to OLD broker ID of OLD Leader, And traffic c… --- ## [Updates not flowing from MongoDB to ES using Logstash](https://discuss.elastic.co/t/updates-not-flowing-from-mongodb-to-es-using-logstash/197399)
**Author:** [@gagz2008shop](https://discuss.elastic.co/u/gagz2008shop)\ **Replies:** 1\ **Last updated:** [September 3, 2019, 6:05pm UTC](https://discuss.elastic.co/t/updates-not-flowing-from-mongodb-to-es-using-logstash/197399 "2019-09-03T18:05:39Z")
Hello Everyone, Any updates in MongoDB are not flowing to ElasticSearch using Logstash. I am using latest versions of all the applications. Following is the logstash config file for reference. Any help is much apprecia… --- ## [Creating nested fields with Grok](https://discuss.elastic.co/t/creating-nested-fields-with-grok/197918)
**Author:** [@carson](https://discuss.elastic.co/u/carson)\ **Replies:** 1\ **Last updated:** [September 3, 2019, 5:47pm UTC](https://discuss.elastic.co/t/creating-nested-fields-with-grok/197918 "2019-09-03T17:47:54Z")
I'm trying to create fields to measure request metrics from our profiler logs (i.e. number of requests, average request time, etc). I have a filter with a grok match like such (shortened for simple understanding): matc… --- ## [Unable to import to ES / Error occurred sending a bulk request to Elasticsearch](https://discuss.elastic.co/t/unable-to-import-to-es-error-occurred-sending-a-bulk-request-to-elasticsearch/197565)
**Author:** [@Alexandre.Bernier](https://discuss.elastic.co/u/Alexandre.Bernier)\ **Replies:** 2\ **Last updated:** [September 3, 2019, 2:44pm UTC](https://discuss.elastic.co/t/unable-to-import-to-es-error-occurred-sending-a-bulk-request-to-elasticsearch/197565 "2019-09-03T14:44:03Z")
Hello community, I'm desperately trying to import data into Elasticsearch with Logstash from MS SQL Server 2012R2 but it fails all the time with the following error: \[2019-08-30T10:57:11,347\]\[ERROR\]\[logstash.outputs.el… --- ## [Logstash giving error at IF statement whereas syntax is correct as per documentation (Trying to fetch response time of API)](https://discuss.elastic.co/t/logstash-giving-error-at-if-statement-whereas-syntax-is-correct-as-per-documentation-trying-to-fetch-response-time-of-api/197770)
**Author:** [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Replies:** 2\ **Last updated:** [September 3, 2019, 1:26pm UTC](https://discuss.elastic.co/t/logstash-giving-error-at-if-statement-whereas-syntax-is-correct-as-per-documentation-trying-to-fetch-response-time-of-api/197770 "2019-09-03T13:26:34Z")
Hi all, I am trying to setup a log management system. I wanted to get response time in a single event hence i used aggregation. But this config is giving me error at the first If statement i am pretty sure other errors… --- ## [How to speed up indexing of csv file via logstash](https://discuss.elastic.co/t/how-to-speed-up-indexing-of-csv-file-via-logstash/197785)
**Author:** [@ludaca](https://discuss.elastic.co/u/ludaca)\ **Replies:** 1\ **Last updated:** [September 3, 2019, 1:15pm UTC](https://discuss.elastic.co/t/how-to-speed-up-indexing-of-csv-file-via-logstash/197785 "2019-09-03T13:15:47Z")
Hello, ES 7.3 cluster on k8s, 6 data nodes csv file: 4500000 rows, 90 fields Indexing takes an hour and a half approximately. \ logstash conf file: input { stdin { type =\> "stdin-type" } } filter { csv { sep… --- ## [Parsing process ID incorrectly](https://discuss.elastic.co/t/parsing-process-id-incorrectly/197862)
**Author:** [@Alpha\_Bravo](https://discuss.elastic.co/u/Alpha_Bravo)\ **Replies:** 0\ **Last updated:** [September 3, 2019, 12:50pm UTC](https://discuss.elastic.co/t/parsing-process-id-incorrectly/197862 "2019-09-03T12:50:52Z")
WorkFlow: Beats (7.0) \> Logstash \> Elastic Issue: The process ID shown below in Elastic is not parsed correctly, we would like it to be without commas. event.ParentProcessId : 1,739,524,678,541 Three different conf… --- ## [Logstash output to kafka server](https://discuss.elastic.co/t/logstash-output-to-kafka-server/197710)
**Author:** [@ELK\_CL](https://discuss.elastic.co/u/ELK_CL)\ **Replies:** 7\ **Last updated:** [September 3, 2019, 12:48pm UTC](https://discuss.elastic.co/t/logstash-output-to-kafka-server/197710 "2019-09-03T12:48:15Z")
Hello! I'm running Logstash 7.0.0. It is using a filebeat input to read logs from described files and output is going to kafka. I am getting my logs correctly for some time post starting logstash. Then i get below erro… --- ## [How fields can be converted](https://discuss.elastic.co/t/how-fields-can-be-converted/197803)
**Author:** [@shenshensinian](https://discuss.elastic.co/u/shenshensinian)\ **Replies:** 0\ **Last updated:** [September 3, 2019, 9:34am UTC](https://discuss.elastic.co/t/how-fields-can-be-converted/197803 "2019-09-03T09:34:03Z")
How to convert grok into time in the format below? date=2016-02-18 time=16:59:42 log\_id=20000 Like what：2016-02-18 16:59:42 --- ## [Timestamp shifting problem](https://discuss.elastic.co/t/timestamp-shifting-problem/197657)
**Author:** [@tunigas](https://discuss.elastic.co/u/tunigas)\ **Replies:** 2\ **Last updated:** [September 3, 2019, 8:22am UTC](https://discuss.elastic.co/t/timestamp-shifting-problem/197657 "2019-09-03T08:22:25Z")
Hello, I'm using these lines in my logstash.conf to configure the timestamp. grok { match =\> { "message" =\>"%{TIMESTAMP\_ISO8601:timestamp}\\s ..." } } date { match =\> \[ "timestamp", "YYYY-mm-dd HH:MM:… --- ## [MariaDB Slow log Query not retrieve in Kibana](https://discuss.elastic.co/t/mariadb-slow-log-query-not-retrieve-in-kibana/197779)
**Author:** [@Ritesh\_Pradhan](https://discuss.elastic.co/u/Ritesh_Pradhan)\ **Replies:** 0\ **Last updated:** [September 3, 2019, 7:59am UTC](https://discuss.elastic.co/t/mariadb-slow-log-query-not-retrieve-in-kibana/197779 "2019-09-03T07:59:21Z")
Hi Elasticsearch Team, I install kibana in a custom path to ubuntu 16.04 in /opt/test/ folder and we are getting all error but slow log query does not retrieve to kibana UI Dashboard and Discover. --- ## [Logstash container using AWS-S3-output-plugin not shipping logs](https://discuss.elastic.co/t/logstash-container-using-aws-s3-output-plugin-not-shipping-logs/197659)
**Author:** [@KalleDania](https://discuss.elastic.co/u/KalleDania)\ **Replies:** 2\ **Last updated:** [September 3, 2019, 6:30am UTC](https://discuss.elastic.co/t/logstash-container-using-aws-s3-output-plugin-not-shipping-logs/197659 "2019-09-03T06:30:51Z")
Hello all, i hope some friendly soul can help me debug why my setup is not working. First off, the goal: Syslogs --\> Container running Logstash --\> AWS S3 bucket I have the container up and running without errors, howe… --- ## [Encountered a retryable error. Will Retry with exponential backoff code=\>400](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-400/197364)
**Author:** [@Thilak](https://discuss.elastic.co/u/Thilak)\ **Replies:** 4\ **Last updated:** [August 30, 2019, 4:39am UTC](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-400/197364 "2019-08-30T04:39:21Z")
Hello I was using logstash 7.0.1 while running the below config file i was continuously getting retry-able error (400) input { elasticsearch { hosts =\> \["https://ful-elastic.com:443/\_bulk"\] user =\> "username" passw… --- ## [Logstash config](https://discuss.elastic.co/t/logstash-config/196916)
**Author:** [@ntlong](https://discuss.elastic.co/u/ntlong)\ **Replies:** 7\ **Last updated:** [September 3, 2019, 5:13am UTC](https://discuss.elastic.co/t/logstash-config/196916 "2019-09-03T05:13:36Z")
Hi everyone, I am newbie to elk stack so it would be so kind if you guy could help me. I am able to send log directly to elasticsearch, but when I try to use logstash, i got below error when starting: logstash --path… --- ## [How to aggregate two events？](https://discuss.elastic.co/t/how-to-aggregate-two-events/197647)
**Author:** [@wzxmt\_l](https://discuss.elastic.co/u/wzxmt_l)\ **Replies:** 0\ **Last updated:** [September 2, 2019, 6:13am UTC](https://discuss.elastic.co/t/how-to-aggregate-two-events/197647 "2019-09-02T06:13:27Z")
hi ! I need your help?How to aggregate two different events together, they only have one associated field message\_id; I want to aggregate two events with the same message\_id, but I can't understand how to configure it! … --- ## [Filebeat -\> logstash -\> rabbitmq -\> logstash -\> elasticsearch](https://discuss.elastic.co/t/filebeat-logstash-rabbitmq-logstash-elasticsearch/197695)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 8\ **Last updated:** [September 2, 2019, 4:06pm UTC](https://discuss.elastic.co/t/filebeat-logstash-rabbitmq-logstash-elasticsearch/197695 "2019-09-02T16:06:55Z")
Hello All, I have a configuration described in the subject line and also here. Logs are shipped by filebeat to a logstash which then outputs it to a rabbitmq. All is well with this part, I can intercept messages in ra… --- ## [Remove the last character of a string field](https://discuss.elastic.co/t/remove-the-last-character-of-a-string-field/197691)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 2\ **Last updated:** [September 2, 2019, 3:11pm UTC](https://discuss.elastic.co/t/remove-the-last-character-of-a-string-field/197691 "2019-09-02T15:11:47Z")
Hello, I have a string field named hostname, when the user have multiple instances I am getting an index in this field. For example hostname looks like: toto1940,toto1941,toto1942. I would like to remove the last elem… --- ## [Use a conditional into another one](https://discuss.elastic.co/t/use-a-conditional-into-another-one/197658)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 2\ **Last updated:** [September 2, 2019, 12:42pm UTC](https://discuss.elastic.co/t/use-a-conditional-into-another-one/197658 "2019-09-02T12:42:22Z")
Hello, I would like to use a conditional in my logstash.conf output. The idea is to send data to two different cluster output{ if \[theXML\]\[measValue\]\[r\]\[p\]==\[theXML\]\[measType\]\[p\]{ if "demo" in \[team\] { elasticsearch {… --- ## [Logstash failing to start as checkpoint file is missing... How to resolve?](https://discuss.elastic.co/t/logstash-failing-to-start-as-checkpoint-file-is-missing-how-to-resolve/197640)
**Author:** [@ramsateesh](https://discuss.elastic.co/u/ramsateesh)\ **Replies:** 1\ **Last updated:** [September 2, 2019, 12:20pm UTC](https://discuss.elastic.co/t/logstash-failing-to-start-as-checkpoint-file-is-missing-how-to-resolve/197640 "2019-09-02T12:20:30Z")
Hi Team, I have logstash running as a container. When I restart logstash container it's failing with below error: ERROR s.r.NativeMethodAccessorImpl \[LogStash::Runner\] Logstash failed to create queue {"exception"=\>"ja… --- ## [Collect informations not from log file but from internet](https://discuss.elastic.co/t/collect-informations-not-from-log-file-but-from-internet/197685)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 0\ **Last updated:** [September 2, 2019, 12:07pm UTC](https://discuss.elastic.co/t/collect-informations-not-from-log-file-but-from-internet/197685 "2019-09-02T12:07:54Z")
hello, I want to configure Logstash to collect information containing a "word" such as "nature" from internet (not from log file) I want to perform a search on the internet, is there this possibility and an example of co… --- ## [Understanding dissect behaviour](https://discuss.elastic.co/t/understanding-dissect-behaviour/196183)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 7\ **Last updated:** [September 2, 2019, 8:27am UTC](https://discuss.elastic.co/t/understanding-dissect-behaviour/196183 "2019-09-02T08:27:10Z")
I have the following filter. filter { dissect { mapping =\> { "message" =\> "%{msg} src=%{host-\>}:%{port} %{abc}" } } } This throws a \_dissectfailure whe… --- ## [Grokparsefailure when I add a date filter](https://discuss.elastic.co/t/grokparsefailure-when-i-add-a-date-filter/197521)
**Author:** [@bquevat](https://discuss.elastic.co/u/bquevat)\ **Replies:** 3\ **Last updated:** [September 2, 2019, 9:29am UTC](https://discuss.elastic.co/t/grokparsefailure-when-i-add-a-date-filter/197521 "2019-09-02T09:29:06Z")
Hello, I'm encoutering a strange behaviour when using a date plugin. I have the following configuration: input { file { path =\> "/var/log/clients/tpxent06s/" type =\> "logs\_yoneos\_preprod" exclude =\> ".gz" } } f… --- ## [Index overwritten by another index](https://discuss.elastic.co/t/index-overwritten-by-another-index/197362)
**Author:** [@Anthony\_BRETON](https://discuss.elastic.co/u/Anthony_BRETON)\ **Replies:** 2\ **Last updated:** [September 2, 2019, 9:22am UTC](https://discuss.elastic.co/t/index-overwritten-by-another-index/197362 "2019-09-02T09:22:10Z")
Hi, I'm a new user to ELK and I haven't experience. I have installed ELK 7.3 on Ubuntu 19.04 x64 and I want to get own Nginx and Fail2Ban log. Elasticsearch, Logstash, Filebeat and Kibana are installed on the same ser… --- ## [Logstash.inputs.jdbc Unable to connect to database](https://discuss.elastic.co/t/logstash-inputs-jdbc-unable-to-connect-to-database/197276)
**Author:** [@whalien52](https://discuss.elastic.co/u/whalien52)\ **Replies:** 1\ **Last updated:** [September 2, 2019, 8:48am UTC](https://discuss.elastic.co/t/logstash-inputs-jdbc-unable-to-connect-to-database/197276 "2019-09-02T08:48:59Z")
Hi, I am trying to connect an SQL database with Elasticsearch. So I have done all the changes in the config file of logstash and created an index in Kibana console. But I am getting error. My config file looks like th… --- ## [Parallel execution for jdbc\_page\_size](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260)
**Author:** [@mrjayarajj](https://discuss.elastic.co/u/mrjayarajj)\ **Replies:** 3\ **Last updated:** [September 2, 2019, 8:30am UTC](https://discuss.elastic.co/t/parallel-execution-for-jdbc-page-size/197260 "2019-09-02T08:30:48Z")
is it possible to run the jdbc\_page\_size execution in parallel ? say if the count is 6K and jdbc\_page\_size is 1K, can we configure log stash to execute 6 parallel pagination ? also if it get 100K , can we have max para… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=482) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=484)