# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=455 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 456 --- ## [Logstash 7.3.0 File Input Plugin not Reading Recreated Files](https://discuss.elastic.co/t/logstash-7-3-0-file-input-plugin-not-reading-recreated-files/205703)
**Author:** [@JesseDubs](https://discuss.elastic.co/u/JesseDubs)\ **Replies:** 4\ **Last updated:** [October 29, 2019, 5:05pm UTC](https://discuss.elastic.co/t/logstash-7-3-0-file-input-plugin-not-reading-recreated-files/205703 "2019-10-29T17:05:29Z")
Hi, I have a bunch of json files that I'm using logstash to read, then delete, and then those files will eventually be recreated (with either the same contents, or slightly different contents). I'd then like for logstas… --- ## [Multiples inputs and outputs jdbc](https://discuss.elastic.co/t/multiples-inputs-and-outputs-jdbc/205687)
**Author:** [@Rafael\_Almeida1](https://discuss.elastic.co/u/Rafael_Almeida1)\ **Replies:** 2\ **Last updated:** [October 29, 2019, 4:35pm UTC](https://discuss.elastic.co/t/multiples-inputs-and-outputs-jdbc/205687 "2019-10-29T16:35:37Z")
Hello, i'm trying to create a pipeline to fetch information from one table and insert / update into two other tables with only insert / update conditional on the second table if I successfully insert / update on the firs… --- ## [Geoip on custom field](https://discuss.elastic.co/t/geoip-on-custom-field/205672)
**Author:** [@hannibal](https://discuss.elastic.co/u/hannibal)\ **Replies:** 3\ **Last updated:** [October 29, 2019, 3:43pm UTC](https://discuss.elastic.co/t/geoip-on-custom-field/205672 "2019-10-29T15:43:43Z")
Hi everyone, I'm trying to use the geoIP filter on x\_forwarded\_for header. So my pipeline is quiet easy, I use an HTTP input, and try to use the field x\_forwarded\_for which are in headers parts for the geoip filtering. … --- ## [Remove the beginning of message then apply kv filter](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631)
**Author:** [@jameshanguyen](https://discuss.elastic.co/u/jameshanguyen)\ **Replies:** 6\ **Last updated:** [October 29, 2019, 3:22pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631 "2019-10-29T15:22:35Z")
Hi, I have a message like this: \<30\>device="SFW" date=2019-10-29 time=16:50:28 timezone="+07" device\_name="XG230" device\_id=C88877XHUG2BUU8 log\_id=010101600001 log\_type="Firewall" log\_component="Firewall Rule" log\_subt… --- ## [Logstash eating memory and eventually gets oom-killed](https://discuss.elastic.co/t/logstash-eating-memory-and-eventually-gets-oom-killed/205701)
**Author:** [@Peter\_Holm](https://discuss.elastic.co/u/Peter_Holm)\ **Replies:** 1\ **Last updated:** [October 29, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logstash-eating-memory-and-eventually-gets-oom-killed/205701 "2019-10-29T15:21:06Z")
I'm running logstash 7.2.0 on a Linux on z RHEL 7.4 64 bits. Everything works just fine, ie elasticsearch, logstash, kibana, for a about 1-1.5 day then logstash gets killed by the OS as a OOM. I have changed heap size in… --- ## [Ingesting nanoseconds into elasticsearch](https://discuss.elastic.co/t/ingesting-nanoseconds-into-elasticsearch/202159)
**Author:** [@st3inbeiss](https://discuss.elastic.co/u/st3inbeiss)\ **Replies:** 1\ **Last updated:** [October 29, 2019, 2:40pm UTC](https://discuss.elastic.co/t/ingesting-nanoseconds-into-elasticsearch/202159 "2019-10-29T14:40:24Z")
I'm trying to get a long of nanoseconds since Unix Epoch through logstash into a date\_nanos field in elasticsearch. Since elasticsearch has this datatype, I assumed logstash does have it aswell. However, the date-filter… --- ## [Add numeric field](https://discuss.elastic.co/t/add-numeric-field/205685)
**Author:** [@RickT](https://discuss.elastic.co/u/RickT)\ **Replies:** 2\ **Last updated:** [October 29, 2019, 2:35pm UTC](https://discuss.elastic.co/t/add-numeric-field/205685 "2019-10-29T14:35:43Z")
Hi, i simply want to create a new numeric field. But when using "add-field" command i get systematically a string ! My logstash filter code like : .... mutate { add\_field =\> { "launcher.%{company}" =\> "%{count\_comp}… --- ## [How to nest all fields in an event under a new field in logstash?](https://discuss.elastic.co/t/how-to-nest-all-fields-in-an-event-under-a-new-field-in-logstash/205688)
**Author:** [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Replies:** 1\ **Last updated:** [October 29, 2019, 2:21pm UTC](https://discuss.elastic.co/t/how-to-nest-all-fields-in-an-event-under-a-new-field-in-logstash/205688 "2019-10-29T14:21:34Z")
Hi all, How can use logstash to convert an event like this: { "@timestamp": "2019-10-29T13:33:46.378Z", "message": "p\_name=yuval;p\_surname=khalifa;p\_age=41", "host": "yuvalk", "p\_age": "41", "@version": "1", "p\_s… --- ## [If conditional not working](https://discuss.elastic.co/t/if-conditional-not-working/205547)
**Author:** [@uklipse](https://discuss.elastic.co/u/uklipse)\ **Replies:** 5\ **Last updated:** [October 29, 2019, 1:32pm UTC](https://discuss.elastic.co/t/if-conditional-not-working/205547 "2019-10-29T13:32:28Z")
I have a basic conf file that I'm trying to eventually drop dns traffic but for some reason I can't get the if conditionals to work anymore. Below is all that is in this conf file. I've also tried changing the if stateme… --- ## [Another High CPU usage with Logstash](https://discuss.elastic.co/t/another-high-cpu-usage-with-logstash/202836)
**Author:** [@mhare](https://discuss.elastic.co/u/mhare)\ **Replies:** 16\ **Last updated:** [October 29, 2019, 1:15pm UTC](https://discuss.elastic.co/t/another-high-cpu-usage-with-logstash/202836 "2019-10-29T13:15:12Z")
1st: fairly new to ElasticStack configuration I've looked at the other topics on High CPU, but I do not believe they are helping me I have an Ubuntu box with 8 cores and 8 G of Ram This machine is running Logstash, El… --- ## [Breakpoints in Ruby code](https://discuss.elastic.co/t/breakpoints-in-ruby-code/205652)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 0\ **Last updated:** [October 29, 2019, 11:34am UTC](https://discuss.elastic.co/t/breakpoints-in-ruby-code/205652 "2019-10-29T11:34:34Z")
I have setup Logstash codebase with Intellij IDEA ultimate. When I start the Logstash process in debug mode only breakpoints in Java are hit. The breakpoints in Ruby are not hit. How do I set it up for the Ruby break-poi… --- ## [How to get the domain name from a fully qualified hostname via GROK](https://discuss.elastic.co/t/how-to-get-the-domain-name-from-a-fully-qualified-hostname-via-grok/205429)
**Author:** [@aolvikash](https://discuss.elastic.co/u/aolvikash)\ **Replies:** 4\ **Last updated:** [October 29, 2019, 9:07am UTC](https://discuss.elastic.co/t/how-to-get-the-domain-name-from-a-fully-qualified-hostname-via-grok/205429 "2019-10-29T09:07:21Z")
How to get the domain name from a fully qualified hostname via GROK. I have used the pipeline to extract a couple of details from filebeat and getting one of the fields as hostname, needs to fetch the domain name from a… --- ## [Unable to parse json log to Kibana](https://discuss.elastic.co/t/unable-to-parse-json-log-to-kibana/202620)
**Author:** [@marek.vesely](https://discuss.elastic.co/u/marek.vesely)\ **Replies:** 4\ **Last updated:** [October 29, 2019, 8:02am UTC](https://discuss.elastic.co/t/unable-to-parse-json-log-to-kibana/202620 "2019-10-29T08:02:17Z")
Hello, I have json log like this: {"thread":"http-nio-8943-exec-9","level":"INFO","loggerName":"cz.direct.common.restapi.interceptor.LoggingFilter","message":{"requestNumber":140,"responseCode":200,"contentType":"appli… --- ## [Sincedb file is generating without s3 plugin](https://discuss.elastic.co/t/sincedb-file-is-generating-without-s3-plugin/205600)
**Author:** [@pami](https://discuss.elastic.co/u/pami)\ **Replies:** 0\ **Last updated:** [October 29, 2019, 6:51am UTC](https://discuss.elastic.co/t/sincedb-file-is-generating-without-s3-plugin/205600 "2019-10-29T06:51:38Z")
I have commented out the s3 plugin in logstash conf file . 1\> cleared sincedb files 2\> cleared /tmp/logstash Not sure how sincedb is getting generating ? --- ## [Can Logstash lose data?](https://discuss.elastic.co/t/can-logstash-lose-data/205535)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 3\ **Last updated:** [October 29, 2019, 6:22am UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535 "2019-10-29T06:22:38Z")
If I understand correctly, Logstash has two thread pools: input (IN), processing and output combines (OUT). When Logstash is getting input from Kafka, at what point does it send an ACK to Kafka? If the ACK is sent when t… --- ## [How to get inputs from two tables and store in the same index?](https://discuss.elastic.co/t/how-to-get-inputs-from-two-tables-and-store-in-the-same-index/204574)
**Author:** [@abhisheksoni27](https://discuss.elastic.co/u/abhisheksoni27)\ **Replies:** 2\ **Last updated:** [October 29, 2019, 5:48am UTC](https://discuss.elastic.co/t/how-to-get-inputs-from-two-tables-and-store-in-the-same-index/204574 "2019-10-29T05:48:13Z")
I have a posts table that contains some blog posts. Each post has some tags that are stored in some other table. For example, for post with id 1, that has 5 tags, the post\_tags table has the following entries: tagId, p… --- ## [Can I use environment variables with field values?](https://discuss.elastic.co/t/can-i-use-environment-variables-with-field-values/205583)
**Author:** [@Yungyoung\_Ok](https://discuss.elastic.co/u/Yungyoung_Ok)\ **Replies:** 0\ **Last updated:** [October 29, 2019, 4:57am UTC](https://discuss.elastic.co/t/can-i-use-environment-variables-with-field-values/205583 "2019-10-29T04:57:03Z")
When using environment variables in logstash, set them as follows: ${environment\_variable\_name} example) export ENV\_TAG="tag2" filter { mutate { add\_tag =\> \[ "tag1", "${ENV\_TAG}" \] } } question: Can I make … --- ## [Unable to configure plugins for \*.log](https://discuss.elastic.co/t/unable-to-configure-plugins-for-log/205022)
**Author:** [@gyana\_nayak](https://discuss.elastic.co/u/gyana_nayak)\ **Replies:** 2\ **Last updated:** [October 29, 2019, 5:08am UTC](https://discuss.elastic.co/t/unable-to-configure-plugins-for-log/205022 "2019-10-29T05:08:04Z")
I am a newbie to ELK. was trying to upload log files to Elasticsearch using logstash encounter below error message. \> Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>… --- ## [Logstash cannot connect to ES on server using proxy settings. However, curl command works fine](https://discuss.elastic.co/t/logstash-cannot-connect-to-es-on-server-using-proxy-settings-however-curl-command-works-fine/205574)
**Author:** [@Charles\_MacKay](https://discuss.elastic.co/u/Charles_MacKay)\ **Replies:** 0\ **Last updated:** [October 29, 2019, 1:30am UTC](https://discuss.elastic.co/t/logstash-cannot-connect-to-es-on-server-using-proxy-settings-however-curl-command-works-fine/205574 "2019-10-29T01:30:44Z")
logstash on server A (using the below proxy settings) cannot connect to my ES instance (times out). user@myserver:/usr/share/logstash$ env | grep -i proxy http\_proxy=http://my-proxy-server:80/ ftp\_proxy=http://my-proxy-… --- ## [Create daily indices based on local timezone](https://discuss.elastic.co/t/create-daily-indices-based-on-local-timezone/204330)
**Author:** [@tomr](https://discuss.elastic.co/u/tomr)\ **Replies:** 8\ **Last updated:** [October 29, 2019, 1:08am UTC](https://discuss.elastic.co/t/create-daily-indices-based-on-local-timezone/204330 "2019-10-29T01:08:05Z")
While trying to figure out how to make indices rollover at midnight local time, I found a bunch of people asking how to do it but no clear solution. The ruby code below generates a YYYY.MM.dd formatted string based on @… --- ## [Adding Fields and Formatting](https://discuss.elastic.co/t/adding-fields-and-formatting/205502)
**Author:** [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\ **Replies:** 5\ **Last updated:** [October 28, 2019, 5:48pm UTC](https://discuss.elastic.co/t/adding-fields-and-formatting/205502 "2019-10-28T17:48:37Z")
Hello there I would like to parse / format a log file that at some point it contains the following: 20191016160456 ,PRIORITY,COLOR,APP-NAME~hostname\_instance\_01, process not found First field is a timestamp, seco… --- ## [Ruby filter - detect null values inside structure](https://discuss.elastic.co/t/ruby-filter-detect-null-values-inside-structure/205509)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 1\ **Last updated:** [October 28, 2019, 4:48pm UTC](https://discuss.elastic.co/t/ruby-filter-detect-null-values-inside-structure/205509 "2019-10-28T16:48:27Z")
In my Logstash pipeline, I am loading JSON messages and later use a ruby filter to detect and apply special logic to fields with null values. ruby { code =\> " event.to\_hash.each { |key,value| if value ==… --- ## [Multiple pipelines not working while using YAML hierarchical form](https://discuss.elastic.co/t/multiple-pipelines-not-working-while-using-yaml-hierarchical-form/205485)
**Author:** [@coute](https://discuss.elastic.co/u/coute)\ **Replies:** 0\ **Last updated:** [October 28, 2019, 2:26pm UTC](https://discuss.elastic.co/t/multiple-pipelines-not-working-while-using-yaml-hierarchical-form/205485 "2019-10-28T14:26:39Z")
Version: 7.4.0 Operating System: Red Hat Enterprise Linux Server release 7.6 (Maipo) Steps to Reproduce: Use hierarchical yaml form in multiple pipelines definition. When I create a pipelines.yml file like thi… --- ## [Modify @timestamp](https://discuss.elastic.co/t/modify-timestamp/205471)
**Author:** [@Jan\_Kaspar](https://discuss.elastic.co/u/Jan_Kaspar)\ **Replies:** 1\ **Last updated:** [October 28, 2019, 2:15pm UTC](https://discuss.elastic.co/t/modify-timestamp/205471 "2019-10-28T14:15:50Z")
Hi, I have a question regarding timestamp. I have a lot of text logs with date and time I am parsing field %{DATE:date} %{TIME:time} and i get following format: 28.10.2019 14:10:10 Is it possible to put this value to… --- ## [Index.lifecycle.rollover\_alias does not point to index](https://discuss.elastic.co/t/index-lifecycle-rollover-alias-does-not-point-to-index/205465)
**Author:** [@forthgate](https://discuss.elastic.co/u/forthgate)\ **Replies:** 0\ **Last updated:** [October 28, 2019, 12:53pm UTC](https://discuss.elastic.co/t/index-lifecycle-rollover-alias-does-not-point-to-index/205465 "2019-10-28T12:53:55Z")
I use ELK 7.3 and i got next template : { "order":3, "index\_patterns":\[ "logstash-airflow\*" \], "settings":{ "index":{ "lifecycle":{ "name":"airflow-policy", "rollover\_alias":"airflow-r… --- ## [Windows DNS parsing](https://discuss.elastic.co/t/windows-dns-parsing/205437)
**Author:** [@Jan\_Kaspar](https://discuss.elastic.co/u/Jan_Kaspar)\ **Replies:** 2\ **Last updated:** [October 28, 2019, 1:06pm UTC](https://discuss.elastic.co/t/windows-dns-parsing/205437 "2019-10-28T13:06:16Z")
Hi All, I would like to ask you for help. I am trying to parse Microsoft DNS debug logs but I stucked. My testing environment is running on ELK version 7.2. Beats are also running on 7.2 version. Problem is I am getti… --- ## [Split header param and convert the value to a number](https://discuss.elastic.co/t/split-header-param-and-convert-the-value-to-a-number/205421)
**Author:** [@anuradhamudalige](https://discuss.elastic.co/u/anuradhamudalige)\ **Replies:** 1\ **Last updated:** [October 28, 2019, 12:30pm UTC](https://discuss.elastic.co/t/split-header-param-and-convert-the-value-to-a-number/205421 "2019-10-28T12:30:35Z")
grok { match =\> { "message" =\> "^%{TIMESTAMP\_ISO8601:timestamp} %{LOGLEVEL:loglevel}%{SPACE}%{NOTSPACE:class}%{SPACE}(%{DATA:thread})%{SPACE}%{DATA:linenumber}%{DATA:txnid}%{NUMBER:eventID}%{DATA:eventName}%{DATA:compo… --- ## [Run pipeline based on condition in another pipeline](https://discuss.elastic.co/t/run-pipeline-based-on-condition-in-another-pipeline/204025)
**Author:** [@ansamHox](https://discuss.elastic.co/u/ansamHox)\ **Replies:** 3\ **Last updated:** [October 28, 2019, 12:11pm UTC](https://discuss.elastic.co/t/run-pipeline-based-on-condition-in-another-pipeline/204025 "2019-10-28T12:11:49Z")
Need to import new data from Oracle (as soon as they start batch processing) couple of times a day. E.g., my dba team are doing some batch processes 5 times a day (random time period and number of rows (up to 1 million)… --- ## [Centralized Logging using ELK and Filebeat](https://discuss.elastic.co/t/centralized-logging-using-elk-and-filebeat/205436)
**Author:** [@Ismail\_Zam\_Zam](https://discuss.elastic.co/u/Ismail_Zam_Zam)\ **Replies:** 0\ **Last updated:** [October 28, 2019, 9:41am UTC](https://discuss.elastic.co/t/centralized-logging-using-elk-and-filebeat/205436 "2019-10-28T09:41:19Z")
Hi, I'm making centralized server-client logging on GCP vm-instances. I followed the below link: its seem that the logs are not transferring from the filebeat, which is installed on the client machine. The server lo… --- ## [HTTP filter - failover host](https://discuss.elastic.co/t/http-filter-failover-host/205433)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 0\ **Last updated:** [October 28, 2019, 9:18am UTC](https://discuss.elastic.co/t/http-filter-failover-host/205433 "2019-10-28T09:18:27Z")
In my Logstash pipeline, I am using HTTP filter. The desired solution should be robust to HTTP host failure and if HTTP request fail try to send this request to another host (can be listed in code or as a parameter). I… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=454) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=456)