# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=607 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 608 --- ## [Got an Error - exception=\>#\,](https://discuss.elastic.co/t/got-an-error-exception-argumenterror-invalid-byte-sequence-in-utf-8/166780)
**Author:** [@shubhamblackstratus](https://discuss.elastic.co/u/shubhamblackstratus)\ **Replies:** 5\ **Last updated:** [February 1, 2019, 9:01pm UTC](https://discuss.elastic.co/t/got-an-error-exception-argumenterror-invalid-byte-sequence-in-utf-8/166780 "2019-02-01T21:01:01Z")
Kindly find the below error log over here. Feb 02 06:24:49 logstashbox logstash\[1082\]: \[2019-02-02T06:24:49,357\]\[ERROR\]\[logstash.pipeline \] Error ne\_id=\>"main", :plugin=\>"#LogStash::FilterDelegator:0x5af6ed72", :… --- ## [Help with embedded JSON flatten - Ruby Filter](https://discuss.elastic.co/t/help-with-embedded-json-flatten-ruby-filter/166442)
**Author:** [@Frrrrrunkis](https://discuss.elastic.co/u/Frrrrrunkis)\ **Replies:** 1\ **Last updated:** [February 1, 2019, 8:53pm UTC](https://discuss.elastic.co/t/help-with-embedded-json-flatten-ruby-filter/166442 "2019-02-01T20:53:51Z")
HI All I have been struggling with this json structure for some time now. Im looking for assistance in flattening this json message. I assume that ruby iteration would be needed here but all the examples I try dont … --- ## [Logstash 6.6.0 jdbc\_input plugin crashing during timezone conversion](https://discuss.elastic.co/t/logstash-6-6-0-jdbc-input-plugin-crashing-during-timezone-conversion/166772)
**Author:** [@jdmcalee](https://discuss.elastic.co/u/jdmcalee)\ **Replies:** 1\ **Last updated:** [February 1, 2019, 8:21pm UTC](https://discuss.elastic.co/t/logstash-6-6-0-jdbc-input-plugin-crashing-during-timezone-conversion/166772 "2019-02-01T20:21:59Z")
Since updating to 6.6.0, the jdbc\_input plugin is crashing with an undefined method error: \[2019-02-01T13:02:30,997\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin. Pip… --- ## [Logstash grok filter partial working](https://discuss.elastic.co/t/logstash-grok-filter-partial-working/166758)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 1\ **Last updated:** [February 1, 2019, 7:19pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-partial-working/166758 "2019-02-01T19:19:06Z")
This is just related to the post where i'm using the below grok filter to dissect the data to be visualize into kibana, below is what i'm using into my logstash conf file and working for the data as desired but today I g… --- ## [How to Apply Filter while Using Logstash Modules?](https://discuss.elastic.co/t/how-to-apply-filter-while-using-logstash-modules/166750)
**Author:** [@jwjhou](https://discuss.elastic.co/u/jwjhou)\ **Replies:** 1\ **Last updated:** [February 1, 2019, 6:30pm UTC](https://discuss.elastic.co/t/how-to-apply-filter-while-using-logstash-modules/166750 "2019-02-01T18:30:12Z")
I'm using Logstash ArcSight Module, one of Logstash Modules, to process my CEF data, which comes from Kafka. However, I'd like to do manipulation with the data, such as: Metadata from Kafka Filter "Mutate" to add mor… --- ## [BigQuery Output plugin: Error creating table. {:exception=\>java.lang.NullPointerException}](https://discuss.elastic.co/t/bigquery-output-plugin-error-creating-table-exception-java-lang-nullpointerexception/166762)
**Author:** [@ginger-tea](https://discuss.elastic.co/u/ginger-tea)\ **Replies:** 0\ **Last updated:** [February 1, 2019, 5:06pm UTC](https://discuss.elastic.co/t/bigquery-output-plugin-error-creating-table-exception-java-lang-nullpointerexception/166762 "2019-02-01T17:06:47Z")
I am seeing this type of error periodically in between logs of "publishing messages to bigquery table" \[2019-02-01T10:30:36,500\]\[ERROR\]\[logstash.outputs.googlebigquery\] Error creating table. {:exception=\>java.lang.NullP… --- ## [Different types of events using same instance of logstash](https://discuss.elastic.co/t/different-types-of-events-using-same-instance-of-logstash/166723)
**Author:** [@azam](https://discuss.elastic.co/u/azam)\ **Replies:** 3\ **Last updated:** [February 1, 2019, 2:36pm UTC](https://discuss.elastic.co/t/different-types-of-events-using-same-instance-of-logstash/166723 "2019-02-01T14:36:48Z")
Hi all I am trying to ship data from two different sources i.e. log data from the application server itself using gelf interface and the second is jmx metrics data using jmx input plugin. My question is, is it possible … --- ## [How to save and delete documents when login or logout event occurs](https://discuss.elastic.co/t/how-to-save-and-delete-documents-when-login-or-logout-event-occurs/166653)
**Author:** [@a86236df74eecab0f8a3](https://discuss.elastic.co/u/a86236df74eecab0f8a3)\ **Replies:** 3\ **Last updated:** [February 1, 2019, 2:34pm UTC](https://discuss.elastic.co/t/how-to-save-and-delete-documents-when-login-or-logout-event-occurs/166653 "2019-02-01T14:34:45Z")
Hi everyone. I want to show list the number of active users in VPN log using logstash. I distinguished VPN logs login and logout using tags filed. So I have separate records for Login and Logout. This is how I think. … --- ## [Logstash Crashes on Ruby filter when I change a field name](https://discuss.elastic.co/t/logstash-crashes-on-ruby-filter-when-i-change-a-field-name/166409)
**Author:** [@ChrisW](https://discuss.elastic.co/u/ChrisW)\ **Replies:** 4\ **Last updated:** [February 1, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-crashes-on-ruby-filter-when-i-change-a-field-name/166409 "2019-02-01T14:05:29Z")
Logstash version: 6.4.2 I have this code for a ruby in filter {...}: ruby { code =\> 'event.set("logstash\_time\_difference", (((event.get("\[received\_at\]").to\_f - event.get("\[@timestamp\]").to\_f).abs)\*1000)… --- ## [Want to parse fields in apache logs, where some fields are optional (may or may not be part of logs)](https://discuss.elastic.co/t/want-to-parse-fields-in-apache-logs-where-some-fields-are-optional-may-or-may-not-be-part-of-logs/166467)
**Author:** [@sandy6](https://discuss.elastic.co/u/sandy6)\ **Replies:** 3\ **Last updated:** [February 1, 2019, 1:47pm UTC](https://discuss.elastic.co/t/want-to-parse-fields-in-apache-logs-where-some-fields-are-optional-may-or-may-not-be-part-of-logs/166467 "2019-02-01T13:47:28Z")
Hi, i am parsing apache logs from 2 apache servers. Both have different log formats. i want to parse logs and extract all the fields. Some fields are available in server-1 logs, which are not present in server-2 logs a… --- ## [Removes ::ffff: from ip address field](https://discuss.elastic.co/t/removes-from-ip-address-field/166706)
**Author:** [@Thibaut603](https://discuss.elastic.co/u/Thibaut603)\ **Replies:** 3\ **Last updated:** [February 1, 2019, 1:09pm UTC](https://discuss.elastic.co/t/removes-from-ip-address-field/166706 "2019-02-01T13:09:51Z")
Hello, i want to remove ::ffff: from ip address field (like this) so in logstash i've made this # Removes ::ffff from IP address filter { if "winlogbeat" in \[tags\] { mutate { gsub =\> \["\[event\_data\]\[IpAddres… --- ## [Cant parse .log to json](https://discuss.elastic.co/t/cant-parse-log-to-json/166548)
**Author:** [@erdo](https://discuss.elastic.co/u/erdo)\ **Replies:** 12\ **Last updated:** [February 1, 2019, 1:05pm UTC](https://discuss.elastic.co/t/cant-parse-log-to-json/166548 "2019-02-01T13:05:44Z")
My log file 2019-01-18 14:03:07,666 - Request - ..................... - http://......................................................................................... - getOpenInvoices - \ **Author:** [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\ **Replies:** 0\ **Last updated:** [February 1, 2019, 11:00am UTC](https://discuss.elastic.co/t/how-to-extract-the-end-point-and-parameters-from-logs/166716 "2019-02-01T11:00:50Z") Hi All, Just now I install elastic stack 6.6.0 and trying to write grok everything seems good. But I need to extract more data from the log like: 10.121.123.104 - - \[01/Nov/2012:21:01:04 +0100\] "GET /cluster HTTP/1.1" … --- ## [Multiple IF conditions - Logstash](https://discuss.elastic.co/t/multiple-if-conditions-logstash/166186)
**Author:** [@Jasna\_Bencic](https://discuss.elastic.co/u/Jasna_Bencic)\ **Replies:** 13\ **Last updated:** [February 1, 2019, 9:40am UTC](https://discuss.elastic.co/t/multiple-if-conditions-logstash/166186 "2019-02-01T09:40:21Z")
Hi, I am fixing bigger logstash config file where I have custom grok patterns but that is just tip of the iceberg regarding my problems. In Filebeat I have multiple log files and some of them (their log events) are vis… --- ## [Logstash twitter](https://discuss.elastic.co/t/logstash-twitter/166700)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 0\ **Last updated:** [February 1, 2019, 9:32am UTC](https://discuss.elastic.co/t/logstash-twitter/166700 "2019-02-01T09:32:27Z")
i am getting this error when running a logstash twitter conf Twitter client error {:message=\>"No message available", :exception=\>"IOError", :backtrace=\>\["org/jruby/ext/openssl/SSLSocket.java:805:in sysread'", "uri:class… --- ## [Make logstash run on two nodes](https://discuss.elastic.co/t/make-logstash-run-on-two-nodes/166688)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 5\ **Last updated:** [February 1, 2019, 8:30am UTC](https://discuss.elastic.co/t/make-logstash-run-on-two-nodes/166688 "2019-02-01T08:30:15Z")
How can I make same logstash instance run on 2 nodes. For example I have a normal configuration input a csv file, csv filter and output to elastic search. Can I make the 2 nodes process the same file for faster indexing … --- ## [Unable to start Logstash via systemctl](https://discuss.elastic.co/t/unable-to-start-logstash-via-systemctl/166691)
**Author:** [@Ricardo](https://discuss.elastic.co/u/Ricardo)\ **Replies:** 0\ **Last updated:** [February 1, 2019, 8:24am UTC](https://discuss.elastic.co/t/unable-to-start-logstash-via-systemctl/166691 "2019-02-01T08:24:34Z")
Hi together, after I installed Search Guard Plugin Ichanged the hosts =\> \["\] in my own pipeline-config Output Section from: hosts =\> \["http://localhost:9200"\] to hosts =\> \["https://localhost:9200"\] but now I'm unabl… --- ## [Error Reloading pipelines.yml While Logstash Is Running](https://discuss.elastic.co/t/error-reloading-pipelines-yml-while-logstash-is-running/166672)
**Author:** [@henrytran026](https://discuss.elastic.co/u/henrytran026)\ **Replies:** 1\ **Last updated:** [February 1, 2019, 6:50am UTC](https://discuss.elastic.co/t/error-reloading-pipelines-yml-while-logstash-is-running/166672 "2019-02-01T06:50:01Z")
Version: 6.5.4 Operating System: Running with Docker Config File: pipelines.yml - pipeline.id: test-app path.config: "pipeline/test-app-pipeline.conf" - pipeline.id: test-app1 path.config: "pipeline/test-app1-pi… --- ## [A logstash input plugin to ingest RDF/graph data into elasticsearch?](https://discuss.elastic.co/t/a-logstash-input-plugin-to-ingest-rdf-graph-data-into-elasticsearch/166248)
**Author:** [@vnktsh](https://discuss.elastic.co/u/vnktsh)\ **Replies:** 5\ **Last updated:** [February 1, 2019, 2:04am UTC](https://discuss.elastic.co/t/a-logstash-input-plugin-to-ingest-rdf-graph-data-into-elasticsearch/166248 "2019-02-01T02:04:04Z")
Hello guys, Recently I integrated Postgres data into Elasticsearch through Logstash via JDBC plugin. I have a graph DB (Apache Jena) which stores rdf data in a triplestore. Is there a plugin which can connect to this gr… --- ## [Accidentally Getting duplicate values in field](https://discuss.elastic.co/t/accidentally-getting-duplicate-values-in-field/166656)
**Author:** [@tadgh](https://discuss.elastic.co/u/tadgh)\ **Replies:** 1\ **Last updated:** [February 1, 2019, 1:41am UTC](https://discuss.elastic.co/t/accidentally-getting-duplicate-values-in-field/166656 "2019-02-01T01:41:43Z")
Hey all! I'm having a weird issue where my pattern seems to be triggering and adding a field to the event twice in logstash. Have a look at a single event below: TLDR: It looks as though after grok successfully matches… --- ## [Can not be updated Lostash plugin if it is via a proxy](https://discuss.elastic.co/t/can-not-be-updated-lostash-plugin-if-it-is-via-a-proxy/166334)
**Author:** [@taku333](https://discuss.elastic.co/u/taku333)\ **Replies:** 6\ **Last updated:** [February 1, 2019, 1:27am UTC](https://discuss.elastic.co/t/can-not-be-updated-lostash-plugin-if-it-is-via-a-proxy/166334 "2019-02-01T01:27:20Z")
I try Logstash update 6.5.4 → 6.6.0.(Via proxy) The following error message will fail and fail. ErrorMessage Error Bundler::HTTPError, retrying 1/10 Could not fetch specs from https://rubygems.org/ But It will succee… --- ## [Xml filter does not work](https://discuss.elastic.co/t/xml-filter-does-not-work/166047)
**Author:** [@Necus](https://discuss.elastic.co/u/Necus)\ **Replies:** 12\ **Last updated:** [January 31, 2019, 8:45pm UTC](https://discuss.elastic.co/t/xml-filter-does-not-work/166047 "2019-01-31T20:45:54Z")
Hi, Do you have any idea why this xml filter does not work? :slight\_smile: filter{ if \[type\] == "abc" { dissect { mapping =\> { message =\> '\<%{abc}\>%{syslog\_timestamp} %{+syslog\_timestamp} %{+syslog\_timestamp} %{lo… --- ## [Config.reload.automatic ignored on Logstash Docker container](https://discuss.elastic.co/t/config-reload-automatic-ignored-on-logstash-docker-container/166399)
**Author:** [@shirakaba](https://discuss.elastic.co/u/shirakaba)\ **Replies:** 1\ **Last updated:** [January 31, 2019, 8:20pm UTC](https://discuss.elastic.co/t/config-reload-automatic-ignored-on-logstash-docker-container/166399 "2019-01-31T20:20:22Z")
Setup Docker details Docker Desktop (Community) for Mac Version 2.0.0.2 (30215), engine 18.09.1 Image: docker.elastic.co/logstash/logstash:6.3.2. I'm using the https://github.com/deviantony/docker-elk repo to compose t… --- ## [How to generate OAUTH2 token and use REST API to pull logs from Cloud?](https://discuss.elastic.co/t/how-to-generate-oauth2-token-and-use-rest-api-to-pull-logs-from-cloud/166624)
**Author:** [@Mohan\_K](https://discuss.elastic.co/u/Mohan_K)\ **Replies:** 3\ **Last updated:** [January 31, 2019, 8:08pm UTC](https://discuss.elastic.co/t/how-to-generate-oauth2-token-and-use-rest-api-to-pull-logs-from-cloud/166624 "2019-01-31T20:08:33Z")
Hi Experts, I have below requirements . Could you please help me to configure it. In short: I am trying to use OAUTH2 to generate token ,use that token to establish connection with SAAS portal to pull logs using REST … --- ## [Grok filter compile error](https://discuss.elastic.co/t/grok-filter-compile-error/166599)
**Author:** [@fefo69](https://discuss.elastic.co/u/fefo69)\ **Replies:** 2\ **Last updated:** [January 31, 2019, 8:07pm UTC](https://discuss.elastic.co/t/grok-filter-compile-error/166599 "2019-01-31T20:07:58Z")
Hi, I have the following log pattern 2019-01-31 01:27:48 10.24.32.4 POST 200 932 0.059 "394" - - - - - - /api/tbapi/services/oe/productinstances/findP… --- ## [Logstash conditions not working as expected](https://discuss.elastic.co/t/logstash-conditions-not-working-as-expected/166625)
**Author:** [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\ **Replies:** 4\ **Last updated:** [January 31, 2019, 7:47pm UTC](https://discuss.elastic.co/t/logstash-conditions-not-working-as-expected/166625 "2019-01-31T19:47:09Z")
I am trying to determine why the last part of my outputs is failing (when not remarked out). Here is my configuration. Would someone help me figure out why it's failing? It's saying I am missing output { if \[type\] ==… --- ## [Issue with conditional in output definition](https://discuss.elastic.co/t/issue-with-conditional-in-output-definition/166612)
**Author:** [@wanexa](https://discuss.elastic.co/u/wanexa)\ **Replies:** 9\ **Last updated:** [January 31, 2019, 6:37pm UTC](https://discuss.elastic.co/t/issue-with-conditional-in-output-definition/166612 "2019-01-31T18:37:29Z")
trying to put two conditions on my output for grok or geopip failure I tried : if ("\_grokparsefailure" not in \[tags\] or "\_geoip\_lookup\_failure" not in \[tags\]) if "\_grokparsefailure" not in \[tags\] or "\_geoip\_lookup\_fai… --- ## [Logstash configuration for new Version 6.5.4](https://discuss.elastic.co/t/logstash-configuration-for-new-version-6-5-4/166572)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 5\ **Last updated:** [January 31, 2019, 4:40pm UTC](https://discuss.elastic.co/t/logstash-configuration-for-new-version-6-5-4/166572 "2019-01-31T16:40:03Z")
Hello Experts, I know this may look a basic question but its just brain draining for me to understand new logstash config as i'm getting below error: \[2019-01-29T21:27:30,230\]\[WARN \]\[logstash.outputs.elasticsearch\] Cou… --- ## [How Can I parse logs that are already indexed in elasticsearch with logstash?](https://discuss.elastic.co/t/how-can-i-parse-logs-that-are-already-indexed-in-elasticsearch-with-logstash/166578)
**Author:** [@Daniel\_Gonzalez1](https://discuss.elastic.co/u/Daniel_Gonzalez1)\ **Replies:** 3\ **Last updated:** [January 31, 2019, 3:24pm UTC](https://discuss.elastic.co/t/how-can-i-parse-logs-that-are-already-indexed-in-elasticsearch-with-logstash/166578 "2019-01-31T15:24:53Z")
Hello! How can I parse logs that are already indexed? For example one of my log files has a lot of info and all the time shows up new ideas to extract the info in many different ways for many different use cases. My qu… --- ## [Memory logstash](https://discuss.elastic.co/t/memory-logstash/166579)
**Author:** [@Alaoui](https://discuss.elastic.co/u/Alaoui)\ **Replies:** 3\ **Last updated:** [January 31, 2019, 3:22pm UTC](https://discuss.elastic.co/t/memory-logstash/166579 "2019-01-31T15:22:38Z")
hi , im uploading an xml files , everything is ok with kibana , but i need to load them each day ... it s normal !!!! [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=606) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=608)