# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=616 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 617 --- ## [Add a new date field from existing String](https://discuss.elastic.co/t/add-a-new-date-field-from-existing-string/164669)
**Author:** [@kimkong](https://discuss.elastic.co/u/kimkong)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 10:59am UTC](https://discuss.elastic.co/t/add-a-new-date-field-from-existing-string/164669 "2019-01-18T10:59:32Z")
Hello everybody, i am totally new in elastic and hope you can show me an easy solution.:grinning: i want to add a new date field. The date is already existing in the file-name, so its written in the source field alrea… --- ## [I can't install any logstash plugin](https://discuss.elastic.co/t/i-cant-install-any-logstash-plugin/164393)
**Author:** [@Daniel\_Gonzalez1](https://discuss.elastic.co/u/Daniel_Gonzalez1)\ **Replies:** 1\ **Last updated:** [January 18, 2019, 8:22am UTC](https://discuss.elastic.co/t/i-cant-install-any-logstash-plugin/164393 "2019-01-18T08:22:18Z")
Hello! I want to install the elapsed plugin or 1p2location for logstash, and when I enter the command to install the plugin, I see this message: Validating logstash-filter-ip2location IOError: No route to host … --- ## [Logstash-output-influxdb 5.0.5 NullPointerException](https://discuss.elastic.co/t/logstash-output-influxdb-5-0-5-nullpointerexception/164747)
**Author:** [@3291346452](https://discuss.elastic.co/u/3291346452)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 7:45am UTC](https://discuss.elastic.co/t/logstash-output-influxdb-5-0-5-nullpointerexception/164747 "2019-01-18T07:45:11Z")
Hi, I run logstash 6.5.4 on docker. It work normally before I install the plugin：logstash-output-influxdb 5.0.5 my logstash's output\_config: if "firewall\_traffic" in \[input\_type\] { if "noop" in \[trandisp\] or "snat" i… --- ## [Save arrays as csv output put](https://discuss.elastic.co/t/save-arrays-as-csv-output-put/162559)
**Author:** [@Manoj\_Hettiarachchi](https://discuss.elastic.co/u/Manoj_Hettiarachchi)\ **Replies:** 1\ **Last updated:** [January 18, 2019, 6:41am UTC](https://discuss.elastic.co/t/save-arrays-as-csv-output-put/162559 "2019-01-18T06:41:38Z")
Hello, In my system, I input a very long log using filebeat ( has about 3000 characters in a single log line) I need to use logstash to save this log in a csv file. I need to save the log as multiple comma-separated c… --- ## [\[solved\] Logstash if condition sometimes not work!](https://discuss.elastic.co/t/solved-logstash-if-condition-sometimes-not-work/164733)
**Author:** [@nils.xia](https://discuss.elastic.co/u/nils.xia)\ **Replies:** 1\ **Last updated:** [January 18, 2019, 5:43am UTC](https://discuss.elastic.co/t/solved-logstash-if-condition-sometimes-not-work/164733 "2019-01-18T05:43:01Z")
Here is my logstash's config: input { beats { port =\> 8888 } } output { if \[message\] =~ "\[ERROR\]" { exec { command =\> "echo -e %{fields} 上出现以下日志报警: '\\n\\n' %{message} '\\n\\n' 请登录 http://172.17.96.165:5601 查看日志 | m… --- ## [Parse json "message" into separate fields in Kibana](https://discuss.elastic.co/t/parse-json-message-into-separate-fields-in-kibana/164572)
**Author:** [@yash\_sachdeva](https://discuss.elastic.co/u/yash_sachdeva)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 5:22am UTC](https://discuss.elastic.co/t/parse-json-message-into-separate-fields-in-kibana/164572 "2019-01-18T05:22:47Z")
Hi Team, I am looking to parse my json message into individual fields in Kibana. I have been trying out a lot of the recommendations online and on this forum but to no avail. The conf file I have created is as follows: … --- ## [Logstash does not process multiple filebeat inputs](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712)
**Author:** [@victoravr](https://discuss.elastic.co/u/victoravr)\ **Replies:** 2\ **Last updated:** [January 17, 2019, 11:07pm UTC](https://discuss.elastic.co/t/logstash-does-not-process-multiple-filebeat-inputs/164712 "2019-01-17T23:07:29Z")
G'day! I'm new to ELK - please forgive me if it's something obvious. I'd like filebeat to ship two different type of logs to logstash, logstash to process it with different grok filters and output to elasticsearch in t… --- ## [Adding data to existing index for which dashboards are created](https://discuss.elastic.co/t/adding-data-to-existing-index-for-which-dashboards-are-created/164700)
**Author:** [@samnayak](https://discuss.elastic.co/u/samnayak)\ **Replies:** 5\ **Last updated:** [January 17, 2019, 9:58pm UTC](https://discuss.elastic.co/t/adding-data-to-existing-index-for-which-dashboards-are-created/164700 "2019-01-17T21:58:11Z")
Hi All, I am trying to add new data to an existing index in Elasticsearch by running the same config file in Logstash. I already have kibana dashboard created for that index. My doubt is, does Logstah copy again the ol… --- ## [Connection to ES nodes can only be made with Elastic Superuser 401 Error](https://discuss.elastic.co/t/connection-to-es-nodes-can-only-be-made-with-elastic-superuser-401-error/160957)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 3\ **Last updated:** [January 10, 2019, 7:49pm UTC](https://discuss.elastic.co/t/connection-to-es-nodes-can-only-be-made-with-elastic-superuser-401-error/160957 "2019-01-10T19:49:13Z")
Elastic version 6.5.1 on RPM I've been working on our encrypted Logstash to Elasticsearch connection and the only way I can get it to work is by using our elastic user (superuser roles). I've followed the instructions … --- ## [Running logstash through Cron to pull from salesforce](https://discuss.elastic.co/t/running-logstash-through-cron-to-pull-from-salesforce/164692)
**Author:** [@chandra0651](https://discuss.elastic.co/u/chandra0651)\ **Replies:** 1\ **Last updated:** [January 17, 2019, 8:17pm UTC](https://discuss.elastic.co/t/running-logstash-through-cron-to-pull-from-salesforce/164692 "2019-01-17T20:17:19Z")
Hi, My requirement is to pull the data from salesforce and output to elasticsearch. As the logstash donot have scheduler for Salesforce input plugin i am using the cron job to run the logstash for every 15 minutes. For … --- ## [Read JSON from a static file](https://discuss.elastic.co/t/read-json-from-a-static-file/164442)
**Author:** [@mrashid](https://discuss.elastic.co/u/mrashid)\ **Replies:** 4\ **Last updated:** [January 17, 2019, 7:41pm UTC](https://discuss.elastic.co/t/read-json-from-a-static-file/164442 "2019-01-17T19:41:25Z")
I have a JSON file by name "output.json" which has only one entry -- CI ID : 213 I am trying to read from the json file using the below logstash configuration using input file plugin . My aim is to include the conten… --- ## [Add duration ( seconds \> 2 digits ) to timestamp with ruby filter](https://discuss.elastic.co/t/add-duration-seconds-2-digits-to-timestamp-with-ruby-filter/164666)
**Author:** [@Isilve28](https://discuss.elastic.co/u/Isilve28)\ **Replies:** 1\ **Last updated:** [January 17, 2019, 6:43pm UTC](https://discuss.elastic.co/t/add-duration-seconds-2-digits-to-timestamp-with-ruby-filter/164666 "2019-01-17T18:43:41Z")
Hi , I checked this topic, this one and also this one and any of them resolved my problem. I have start\_time and I want to add duration (in seconds) in order to get end\_time I have duration in seconds like this : 540… --- ## [Logstash - aggregation\_fields in elasticsearch filter plugin](https://discuss.elastic.co/t/logstash-aggregation-fields-in-elasticsearch-filter-plugin/164672)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 1\ **Last updated:** [January 17, 2019, 6:14pm UTC](https://discuss.elastic.co/t/logstash-aggregation-fields-in-elasticsearch-filter-plugin/164672 "2019-01-17T18:14:06Z")
Hello, I am trying to use elasticsearch filter plugin in logstash. My logstash file: filter { elasticsearch { hosts =\> "XXX" index =\> "logs\_0" query =\> "query.json" aggregation\_fields =\> { "value\_avg" =\> "AVG"} … --- ## [Logstash - aggregate results](https://discuss.elastic.co/t/logstash-aggregate-results/164351)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 18\ **Last updated:** [January 17, 2019, 6:12pm UTC](https://discuss.elastic.co/t/logstash-aggregate-results/164351 "2019-01-17T18:12:49Z")
Hello, When I am using the aggregate filter in logstash (to get the total sales of each product, for example), is there a way to send the aggregated results to a different output and not line by line? Thank you! --- ## [Capturing group inside a custom grok pattern](https://discuss.elastic.co/t/capturing-group-inside-a-custom-grok-pattern/163384)
**Author:** [@agosmaker](https://discuss.elastic.co/u/agosmaker)\ **Replies:** 3\ **Last updated:** [January 17, 2019, 5:56pm UTC](https://discuss.elastic.co/t/capturing-group-inside-a-custom-grok-pattern/163384 "2019-01-17T17:56:02Z")
Hello, Let's imagine we have string some text aaabbbccc another text We want to extract bbb. we can use "embedded" regex (https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html Custom Patterns "… --- ## [Remove timezone in sql\_last\_run](https://discuss.elastic.co/t/remove-timezone-in-sql-last-run/162319)
**Author:** [@hinscx](https://discuss.elastic.co/u/hinscx)\ **Replies:** 1\ **Last updated:** [January 17, 2019, 5:20pm UTC](https://discuss.elastic.co/t/remove-timezone-in-sql-last-run/162319 "2019-01-17T17:20:49Z")
In JDBC input plugin, I don't want to use ISO8601 for timestamp because. How to choose another timestamp format without timezone for sql\_last\_value? --- ## [Load an xml files](https://discuss.elastic.co/t/load-an-xml-files/164647)
**Author:** [@Alaoui](https://discuss.elastic.co/u/Alaoui)\ **Replies:** 6\ **Last updated:** [January 17, 2019, 3:45pm UTC](https://discuss.elastic.co/t/load-an-xml-files/164647 "2019-01-17T15:45:40Z")
Hi, i want to load a 100 xml files in the same time using logstash, so what i should to add in my config file!. --- ## [Logstash plus filebeat pipeline](https://discuss.elastic.co/t/logstash-plus-filebeat-pipeline/162282)
**Author:** [@fefo69](https://discuss.elastic.co/u/fefo69)\ **Replies:** 6\ **Last updated:** [January 17, 2019, 2:40pm UTC](https://discuss.elastic.co/t/logstash-plus-filebeat-pipeline/162282 "2019-01-17T14:40:06Z")
Hi there, I have a api log file with the following format ... 2018.27.12 17:37:28.423 GET /api/v1/catalogManagement/productOffering 200 4 2018.27.12 17:37:28.242 GET /api/v1/addressManagement/address 200 1214 ... I… --- ## [Pipeline to parse SSL certificate data from httpd message lines?](https://discuss.elastic.co/t/pipeline-to-parse-ssl-certificate-data-from-httpd-message-lines/164533)
**Author:** [@B\_TLJ](https://discuss.elastic.co/u/B_TLJ)\ **Replies:** 5\ **Last updated:** [January 17, 2019, 1:38pm UTC](https://discuss.elastic.co/t/pipeline-to-parse-ssl-certificate-data-from-httpd-message-lines/164533 "2019-01-17T13:38:55Z")
I am new to ELK, and need to figure out a way to pull a username out of the httpd messages I am currently sending to logstash. the message lines look like this: 100.100.100.200 - - \[16/Jan/2019:19:35:55 +0000\] "GET /xx… --- ## [Logstash - unexpected behaviour with persistent queues](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395)
**Author:** [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\ **Replies:** 3\ **Last updated:** [January 17, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-unexpected-behaviour-with-persistent-queues/164395 "2019-01-17T13:36:59Z")
Hi, I'm testing persisted queues usage and I found that, after a lot of debugging, logstash is being stopped with exit code 0. Logs shows that pipeline is being stopped. No error log found. I can't find the reason but f… --- ## [GELF output with compression option?](https://discuss.elastic.co/t/gelf-output-with-compression-option/164435)
**Author:** [@piellick](https://discuss.elastic.co/u/piellick)\ **Replies:** 3\ **Last updated:** [January 17, 2019, 1:32pm UTC](https://discuss.elastic.co/t/gelf-output-with-compression-option/164435 "2019-01-17T13:32:49Z")
Hello Everyone, does logstash support option compressions for GELF over UDP ? Related to this documentation : http://docs.graylog.org/en/latest/pages/gelf.html?highlight=compression#compression I found nothing on log… --- ## [Date Field as "Date" Time as "Time" - Why this so hard to do in Logstash?](https://discuss.elastic.co/t/date-field-as-date-time-as-time-why-this-so-hard-to-do-in-logstash/164487)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 8\ **Last updated:** [January 17, 2019, 11:54am UTC](https://discuss.elastic.co/t/date-field-as-date-time-as-time-why-this-so-hard-to-do-in-logstash/164487 "2019-01-17T11:54:02Z")
Hi all, I mess arround with my csv and a date field formatted as "DD.MM.YYYY" But whatever I do Logstash Filter will ignore all --\> still "String" Even the data\_formatted plug in will not work as I expected. What I… --- ## [How to have a boolean based field migrated from MongoDB to Elasticsearch using Logstash?](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589)
**Author:** [@topgun743](https://discuss.elastic.co/u/topgun743)\ **Replies:** 20\ **Last updated:** [January 17, 2019, 11:40am UTC](https://discuss.elastic.co/t/how-to-have-a-boolean-based-field-migrated-from-mongodb-to-elasticsearch-using-logstash/164589 "2019-01-17T11:40:58Z")
I have banged my head on so many internet forums but failed to accomplish this. Here are my sample documents present in my MongoDB in test collection: { "\_id" : ObjectId("59a41f63dba6488b33268377"), "EntityID":… --- ## [If condition in logstash filter - not working](https://discuss.elastic.co/t/if-condition-in-logstash-filter-not-working/164577)
**Author:** [@gogularaja.sr](https://discuss.elastic.co/u/gogularaja.sr)\ **Replies:** 6\ **Last updated:** [January 17, 2019, 10:52am UTC](https://discuss.elastic.co/t/if-condition-in-logstash-filter-not-working/164577 "2019-01-17T10:52:01Z")
input { beats{ port =\> 5044 } } ######################################################################## filter { if \[prospector.type\] == "log" { } } ############################################################… --- ## [Logstash Pipelines using Chef](https://discuss.elastic.co/t/logstash-pipelines-using-chef/163690)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 1\ **Last updated:** [January 17, 2019, 10:46am UTC](https://discuss.elastic.co/t/logstash-pipelines-using-chef/163690 "2019-01-17T10:46:09Z")
Hi- Did anybody have tried deploying multiple Logstash pipelines using the Chef Cookbooks? We just have a thought to deploy and maintain the pipelines using the Chef. your comments on this would be really appreciated.… --- ## [Forward logs to SIEM](https://discuss.elastic.co/t/forward-logs-to-siem/162826)
**Author:** [@romanski15](https://discuss.elastic.co/u/romanski15)\ **Replies:** 2\ **Last updated:** [January 17, 2019, 9:10am UTC](https://discuss.elastic.co/t/forward-logs-to-siem/162826 "2019-01-17T09:10:30Z")
Hi Team, I would like to know if anybody has used ELK to centralise device logs (Windows, Linux, firewall, etc), before forwarding them on to a SIEM? I have a requirement to have a central log storage at each of my geo… --- ## [Report DLQ's max\_queue\_size\_in\_bytes in logstash API](https://discuss.elastic.co/t/report-dlqs-max-queue-size-in-bytes-in-logstash-api/164585)
**Author:** [@Sarfaraz\_Ahmad](https://discuss.elastic.co/u/Sarfaraz_Ahmad)\ **Replies:** 0\ **Last updated:** [January 17, 2019, 8:13am UTC](https://discuss.elastic.co/t/report-dlqs-max-queue-size-in-bytes-in-logstash-api/164585 "2019-01-17T08:13:17Z")
logstash's monitoring API only reports the bytes used in the DLQ. If it were to report the configured bytes for the DLQ like what it does with the persistent queue, then my monitoring tool can figure out what percentage… --- ## [Logstash logging to messages](https://discuss.elastic.co/t/logstash-logging-to-messages/164563)
**Author:** [@jplamadrid](https://discuss.elastic.co/u/jplamadrid)\ **Replies:** 0\ **Last updated:** [January 17, 2019, 2:15am UTC](https://discuss.elastic.co/t/logstash-logging-to-messages/164563 "2019-01-17T02:15:02Z")
Good day, Just want to know how can I control the logstash logging to /var/messages.log in Centos 7? We integrated ELK with snort and keep on receiving these kind of logs: Jan 13 03:39:04 hostname logstash: { Jan 13 0… --- ## [LogLevel field in kibana](https://discuss.elastic.co/t/loglevel-field-in-kibana/164524)
**Author:** [@shiv94](https://discuss.elastic.co/u/shiv94)\ **Replies:** 11\ **Last updated:** [January 16, 2019, 8:32pm UTC](https://discuss.elastic.co/t/loglevel-field-in-kibana/164524 "2019-01-16T20:32:15Z")
Hello, I want to get the log level field in kibana for all the logs For example, 2019-01-16 13:54:55,833 INFO \[org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean\] (default task-29) Creating Service {htt… --- ## [How to extract the time elapsed from a value of a field to another value of the same field](https://discuss.elastic.co/t/how-to-extract-the-time-elapsed-from-a-value-of-a-field-to-another-value-of-the-same-field/164375)
**Author:** [@Daniel\_Gonzalez1](https://discuss.elastic.co/u/Daniel_Gonzalez1)\ **Replies:** 8\ **Last updated:** [January 16, 2019, 8:27pm UTC](https://discuss.elastic.co/t/how-to-extract-the-time-elapsed-from-a-value-of-a-field-to-another-value-of-the-same-field/164375 "2019-01-16T20:27:08Z")
Hello! I have a workflow and I want to know how much time pass from a value to another! For example in the next image with the field "filters" I want to know how much time pass from the value "Creacion" to "Tienda" in t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=615) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=617)