# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=566 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 567 --- ## [Error in loading twitter data](https://discuss.elastic.co/t/error-in-loading-twitter-data/176730)
**Author:** [@pathri](https://discuss.elastic.co/u/pathri)\ **Replies:** 0\ **Last updated:** [April 13, 2019, 10:25am UTC](https://discuss.elastic.co/t/error-in-loading-twitter-data/176730 "2019-04-13T10:25:36Z")
Hi All, i am loading the data from twitter in es using logstash but getting below error. i have tried with 3 new generated APIs but still not able to resolve it. Please help! \[2019-04-13T15:50:55,252\]\[WARN \]\[logstash… --- ## [Logstash Not Starting 7.0](https://discuss.elastic.co/t/logstash-not-starting-7-0/176720)
**Author:** [@lakshykar](https://discuss.elastic.co/u/lakshykar)\ **Replies:** 3\ **Last updated:** [April 13, 2019, 6:56am UTC](https://discuss.elastic.co/t/logstash-not-starting-7-0/176720 "2019-04-13T06:56:48Z")
Hi, I am trying to configure logstash on a server, i did fresh install but when i am starting logstash getting below error. \[2019-04-13T10:44:26,616\]\[DEBUG\]\[logstash.modules.scaffold\] Found module {:module\_name=\>"netfl… --- ## [Trim log line before JSON codec in Input?](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532)
**Author:** [@danielmcquillen](https://discuss.elastic.co/u/danielmcquillen)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 7:53pm UTC](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532 "2019-04-12T19:53:59Z")
I was using the cloudwatch\_logs plugin to read JSON-based logs from an AWS cloudwatch logstream. Those log lines were read ok, with the keys in the JSON arriving as properties on my event object and accessible in the fil… --- ## [Timeout executing grok using pipe symbol](https://discuss.elastic.co/t/timeout-executing-grok-using-pipe-symbol/176684)
**Author:** [@nowens95](https://discuss.elastic.co/u/nowens95)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 6:08pm UTC](https://discuss.elastic.co/t/timeout-executing-grok-using-pipe-symbol/176684 "2019-04-12T18:08:24Z")
Hey all, I'm getting a grok timeout warning with a very specific format of log. Here is my grok filter: grok { match =\> { "message" =\> "fs\_log: %{TIMESTAMP\_ISO8601:timestamp} \\|\\| %{UNIXPATH:filename} \\|\\| %{WORD:cal… --- ## [Dynamic Input for drop plugin](https://discuss.elastic.co/t/dynamic-input-for-drop-plugin/176649)
**Author:** [@adireddi77](https://discuss.elastic.co/u/adireddi77)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 5:53pm UTC](https://discuss.elastic.co/t/dynamic-input-for-drop-plugin/176649 "2019-04-12T17:53:13Z")
I am new to Logstash. Have a json input, I am trying to filter the content before I write the output to a file. Below is my filter -- filter { if \[msg\] != "null" { mutate { remove\_field =\> \["msg"\] } } if \[rawmsg… --- ## [Filtering incoming syslog (CSV) with columns containing JSON](https://discuss.elastic.co/t/filtering-incoming-syslog-csv-with-columns-containing-json/176512)
**Author:** [@jgardiner](https://discuss.elastic.co/u/jgardiner)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 5:22pm UTC](https://discuss.elastic.co/t/filtering-incoming-syslog-csv-with-columns-containing-json/176512 "2019-04-12T17:22:36Z")
I'm importing syslog messages from Palo Alto's Cortex Hub (specifically Traps logs) using logstash's syslog importer. That works great. The issue I'm running into is that these logs (in syslog format) contain json in 5… --- ## [Issue in parsing a custom apache log file with grok plugin](https://discuss.elastic.co/t/issue-in-parsing-a-custom-apache-log-file-with-grok-plugin/176640)
**Author:** [@Stefano\_M](https://discuss.elastic.co/u/Stefano_M)\ **Replies:** 6\ **Last updated:** [April 12, 2019, 4:56pm UTC](https://discuss.elastic.co/t/issue-in-parsing-a-custom-apache-log-file-with-grok-plugin/176640 "2019-04-12T16:56:32Z")
Hello, I'm new to Elastic stack so be patient :slight\_smile: My first purpose is to parse a custom Apache2 access log. "Custom" means that I only have added a field to the Apache2 Combined Log format. This filed, add… --- ## [Grok pattern extract data based on column position](https://discuss.elastic.co/t/grok-pattern-extract-data-based-on-column-position/176126)
**Author:** [@bruinjoe](https://discuss.elastic.co/u/bruinjoe)\ **Replies:** 3\ **Last updated:** [April 12, 2019, 4:23pm UTC](https://discuss.elastic.co/t/grok-pattern-extract-data-based-on-column-position/176126 "2019-04-12T16:23:05Z")
I have a mainframe export. I need to parse it based on column position. Does Logstash grok support extraction based on column position? Example line: M4349456 First two characters dept number: M4 Characters 3-6 mail … --- ## [Logstash jdbc static plugin staging directory use](https://discuss.elastic.co/t/logstash-jdbc-static-plugin-staging-directory-use/176653)
**Author:** [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 3:04pm UTC](https://discuss.elastic.co/t/logstash-jdbc-static-plugin-staging-directory-use/176653 "2019-04-12T15:04:13Z")
Hello Team, I am planning to use jdbc static plugin for data enrichment. could you please help me understand the use of staging directory, will it be used to load the incoming data /events for staging or will it load t… --- ## [Logstash S3 input issues](https://discuss.elastic.co/t/logstash-s3-input-issues/175306)
**Author:** [@cfinnie](https://discuss.elastic.co/u/cfinnie)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 2:26pm UTC](https://discuss.elastic.co/t/logstash-s3-input-issues/175306 "2019-04-12T14:26:45Z")
Hi there, I am having trouble trying to get Logstash to use the S3 input plugin to load log files from an S3 bucket. This is the error I get: \[INFO \]\[logstash.inputs.s3 \] S3 input: No files found in bucket {:pr… --- ## [Simple Split filter](https://discuss.elastic.co/t/simple-split-filter/175362)
**Author:** [@Lebedev](https://discuss.elastic.co/u/Lebedev)\ **Replies:** 4\ **Last updated:** [April 12, 2019, 2:10pm UTC](https://discuss.elastic.co/t/simple-split-filter/175362 "2019-04-12T14:10:08Z")
Hello! Dear collegues, need your help! I have the input log 09:00:00.023|SMSG|\*\*\*|ROUTING begin FLD\_029 \*\* VALIDITY\_FLAG T TIME\_STAT |POS\_ISO 09:00:00.024|DEBG|\*\*\*|proc\_new\_msg() FLD\_029 \*\* VALIDITY\_FLAG F TIME\_… --- ## [Testing fails with "bin/rspec spec/unit/\*\*/\*\_spec.rb" in logstash-5.6.15](https://discuss.elastic.co/t/testing-fails-with-bin-rspec-spec-unit-spec-rb-in-logstash-5-6-15/176637)
**Author:** [@jhon](https://discuss.elastic.co/u/jhon)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 1:51pm UTC](https://discuss.elastic.co/t/testing-fails-with-bin-rspec-spec-unit-spec-rb-in-logstash-5-6-15/176637 "2019-04-12T13:51:36Z")
Testing fails with "bin/rspec spec/unit/\*\*/\*\_spec.rb" in logstash-5.6.15 I am performing testing of logstash-5.6.15 and I got below error. ERROR Bundler::GemNotFound: Could not find benchmark-ips-2.7.2 in any of the so… --- ## [Issues with Logstash Conditionals inside filter](https://discuss.elastic.co/t/issues-with-logstash-conditionals-inside-filter/176322)
**Author:** [@Dilruwan\_Madubashi10](https://discuss.elastic.co/u/Dilruwan_Madubashi10)\ **Replies:** 6\ **Last updated:** [April 12, 2019, 1:48pm UTC](https://discuss.elastic.co/t/issues-with-logstash-conditionals-inside-filter/176322 "2019-04-12T13:48:00Z")
I have been trying to get a CSV file parsed into logstash and it works fine. How ever I need to add either a new tag or a new field based on certain conditions. I have validated that in the logs I have the text that mat… --- ## [Logstash.runner: CompiledPipeline.java: java.lang.IllegalStateException: Error](https://discuss.elastic.co/t/logstash-runner-compiledpipeline-java-java-lang-illegalstateexception-error/176412)
**Author:** [@usrjph](https://discuss.elastic.co/u/usrjph)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 1:39pm UTC](https://discuss.elastic.co/t/logstash-runner-compiledpipeline-java-java-lang-illegalstateexception-error/176412 "2019-04-12T13:39:10Z")
I've updated our test box to 7.0, having deleted the old Elasticsearch database first and merged any config updates. I resolved a few config deprecations, including using \_default in static mapping, but I'm now blocked b… --- ## [Issue with ports](https://discuss.elastic.co/t/issue-with-ports/176414)
**Author:** [@Farhad\_Kocharli](https://discuss.elastic.co/u/Farhad_Kocharli)\ **Replies:** 3\ **Last updated:** [April 12, 2019, 12:27pm UTC](https://discuss.elastic.co/t/issue-with-ports/176414 "2019-04-12T12:27:40Z")
Hello everyone! I have elastic stack 6.7 on one node I am trying to parse logs from cisco devices. I am using below configuration logstash stops listen on all ports (for winlogbeat and filebeat). P.S I have installed … --- ## [Ruby exception](https://discuss.elastic.co/t/ruby-exception/176557)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 12:22pm UTC](https://discuss.elastic.co/t/ruby-exception/176557 "2019-04-12T12:22:05Z")
Is there any issue with the below logstash configuration. I am trying to access the file path using event.get('path') and then split it. I get the following exception \[2019-04-12T07:19:25,192\]\[ERROR\]\[logstash.filters.ru… --- ## [Adding new fields from a match \[solved\]](https://discuss.elastic.co/t/adding-new-fields-from-a-match-solved/176197)
**Author:** [@elasticSmash](https://discuss.elastic.co/u/elasticSmash)\ **Replies:** 6\ **Last updated:** [April 12, 2019, 9:53am UTC](https://discuss.elastic.co/t/adding-new-fields-from-a-match-solved/176197 "2019-04-12T09:53:34Z")
Hi. New to logstash. One thing I don't understand is that when a you match a line with something like %{GREEDYDATA:myfield1}, does that automatically create a corresponding field in elasticsearch or do I need to add it o… --- ## [Filter "Date" not applying to @timestamp](https://discuss.elastic.co/t/filter-date-not-applying-to-timestamp/176566)
**Author:** [@Victor\_Krot](https://discuss.elastic.co/u/Victor_Krot)\ **Replies:** 2\ **Last updated:** [April 12, 2019, 9:20am UTC](https://discuss.elastic.co/t/filter-date-not-applying-to-timestamp/176566 "2019-04-12T09:20:15Z")
Hi! Seen a lot of topics about converting date from custom log's strings. None of them helped me. I have a string like this for example: 2019-04-11 07:36:27.842 - 14292 Information Interceptor Organization protection … --- ## [Logstash not able to connect to presto using logstash](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-presto-using-logstash/176368)
**Author:** [@yasin\_mohammed](https://discuss.elastic.co/u/yasin_mohammed)\ **Replies:** 4\ **Last updated:** [April 12, 2019, 8:31am UTC](https://discuss.elastic.co/t/logstash-not-able-to-connect-to-presto-using-logstash/176368 "2019-04-12T08:31:57Z")
Hi... We have a cassandra database from where we were fetching data and it used to work properly. Now we are asked to connect to presto sitting on top of cassandra. After changing the driver class and path to presto I k… --- ## [Logstash dont have any error on netflow modules](https://discuss.elastic.co/t/logstash-dont-have-any-error-on-netflow-modules/176568)
**Author:** [@kecepek](https://discuss.elastic.co/u/kecepek)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 7:53am UTC](https://discuss.elastic.co/t/logstash-dont-have-any-error-on-netflow-modules/176568 "2019-04-12T07:53:05Z")
I success to install logstash with netflow modules, but seem elasticearch seems doesnt to create index pattern. Still figuring out what is missing. Blockquote ./logstash --modules netflow Sending Logstash logs to /… --- ## [Getting error with cloudwatch input plugin](https://discuss.elastic.co/t/getting-error-with-cloudwatch-input-plugin/176341)
**Author:** [@Rajat\_Ravi](https://discuss.elastic.co/u/Rajat_Ravi)\ **Replies:** 7\ **Last updated:** [April 12, 2019, 7:51am UTC](https://discuss.elastic.co/t/getting-error-with-cloudwatch-input-plugin/176341 "2019-04-12T07:51:53Z")
\[2019-04-11T06:46:56,025\]\[ERROR\]\[logstash.agent \] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Expected one of #, { at line 352, column 7 (byte 9716) af… --- ## [Calculating the time difference between successive timestamps](https://discuss.elastic.co/t/calculating-the-time-difference-between-successive-timestamps/176552)
**Author:** [@AbhishekSharma](https://discuss.elastic.co/u/AbhishekSharma)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 7:13am UTC](https://discuss.elastic.co/t/calculating-the-time-difference-between-successive-timestamps/176552 "2019-04-12T07:13:38Z")
I have logs for services getting started or stopped. Using the timestamps from these logs I want to calculate the uptime and downtime for services. Here is what I did: Each time a service is stopped, calculate the uptim… --- ## [Differences between the logstash and metricbeat cloudwatch metrics collection](https://discuss.elastic.co/t/differences-between-the-logstash-and-metricbeat-cloudwatch-metrics-collection/176513)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 1\ **Last updated:** [April 12, 2019, 5:00am UTC](https://discuss.elastic.co/t/differences-between-the-logstash-and-metricbeat-cloudwatch-metrics-collection/176513 "2019-04-12T05:00:40Z")
Hi There, It appears that Elastic now has two supported methods of getting Cloudwatch Metrics into ES, Metricbeat and Logstash. Does the new v7 Metricbeat implementation now supersede the Logstash one? I'm about to imp… --- ## [Logstash don't have error, but not input any data!](https://discuss.elastic.co/t/logstash-dont-have-error-but-not-input-any-data/176531)
**Author:** [@levi\_munthe](https://discuss.elastic.co/u/levi_munthe)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 4:28am UTC](https://discuss.elastic.co/t/logstash-dont-have-error-but-not-input-any-data/176531 "2019-04-12T04:28:36Z")
Hello i running logstash 6.7.1. it fresh installation, but not storing any data to elasticsearch. \[2019-04-12T11:10:38,459\]\[INFO \]\[logstash.pipeline \] Starting pipeline {:pipeline\_id=\>"redemptionneeds", "pipeline… --- ## [Logstash-jdbc missing some records occasionally](https://discuss.elastic.co/t/logstash-jdbc-missing-some-records-occasionally/176523)
**Author:** [@wolfman](https://discuss.elastic.co/u/wolfman)\ **Replies:** 0\ **Last updated:** [April 12, 2019, 2:37am UTC](https://discuss.elastic.co/t/logstash-jdbc-missing-some-records-occasionally/176523 "2019-04-12T02:37:16Z")
Hi, I am using logstash-jdbc plugin for extracting records from DB to redis by tracing column updatedAt(timestamp),it worked,but i found that occasionally some updated records in DB are not pulled out,the version of t… --- ## [Help installing logstash plugins offline](https://discuss.elastic.co/t/help-installing-logstash-plugins-offline/176502)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 1\ **Last updated:** [April 11, 2019, 10:23pm UTC](https://discuss.elastic.co/t/help-installing-logstash-plugins-offline/176502 "2019-04-11T22:23:47Z")
On CentOS6.x running LS 6.7 So following directions here: https://www.elastic.co/guide/en/logstash/current/offline-plugins.html I get: # /usr/share/logstash/bin/logstash-plugin prepare-offline-pack --overwrite logst… --- ## [Logstash crashes on mutate filter](https://discuss.elastic.co/t/logstash-crashes-on-mutate-filter/176408)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 1\ **Last updated:** [April 11, 2019, 10:22pm UTC](https://discuss.elastic.co/t/logstash-crashes-on-mutate-filter/176408 "2019-04-11T22:22:47Z")
I'm running logstash 6.1.1 and I started getting a ton of these errors in my logs: \[2019-04-11T09:02:32,320\]\[WARN \]\[logstash.inputs.udp \] UDP listener died {:exception=\>java.nio.channels.ClosedSelectorException, :b… --- ## [Issues parsing JSON with deep nesting](https://discuss.elastic.co/t/issues-parsing-json-with-deep-nesting/176133)
**Author:** [@Dallas\_Toth](https://discuss.elastic.co/u/Dallas_Toth)\ **Replies:** 22\ **Last updated:** [April 11, 2019, 9:38pm UTC](https://discuss.elastic.co/t/issues-parsing-json-with-deep-nesting/176133 "2019-04-11T21:38:31Z")
the array looks like this. data.children.data.replies.data.children.data.replies.data.children.data.replies.data.children.data.replies.data.children.....ETC I have this JSON array that nests and I would like to get all… --- ## [Way to buffer logs in logstash](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334)
**Author:** [@Mudit\_Mehta](https://discuss.elastic.co/u/Mudit_Mehta)\ **Replies:** 4\ **Last updated:** [April 11, 2019, 9:02pm UTC](https://discuss.elastic.co/t/way-to-buffer-logs-in-logstash/176334 "2019-04-11T21:02:26Z")
I have configured logstash on multiple AWS instances. These logstash instances receive thousand of logs and I want a way to wait for the entire trace of logs in logstash and take action on the batch instead of a single l… --- ## [Is there an output plugin that can be used to update logfile index output?](https://discuss.elastic.co/t/is-there-an-output-plugin-that-can-be-used-to-update-logfile-index-output/176098)
**Author:** [@mgolubov55](https://discuss.elastic.co/u/mgolubov55)\ **Replies:** 13\ **Last updated:** [April 11, 2019, 8:24pm UTC](https://discuss.elastic.co/t/is-there-an-output-plugin-that-can-be-used-to-update-logfile-index-output/176098 "2019-04-11T20:24:50Z")
I am processing log files. Each line of the log files being processed contain an integer field called 'flag'. One of the lines of each log contains a value 'JobID' that is extracted in the filter section of the pipeli… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=565) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=567)