# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=730 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 731 --- ## [Parsing avro files](https://discuss.elastic.co/t/parsing-avro-files/138307)
**Author:** [@alex3](https://discuss.elastic.co/u/alex3)\ **Replies:** 0\ **Last updated:** [July 3, 2018, 5:44am UTC](https://discuss.elastic.co/t/parsing-avro-files/138307 "2018-07-03T05:44:08Z")
I'd like to ingest an avro file with an embedded schema (.avro). What's the best way to go about doing this? I was thinking of extracting the schema from the .avro file and making a separate .avsc file, but not sure wher… --- ## [Move content of sub fields as top fields](https://discuss.elastic.co/t/move-content-of-sub-fields-as-top-fields/138245)
**Author:** [@prodrg](https://discuss.elastic.co/u/prodrg)\ **Replies:** 2\ **Last updated:** [July 3, 2018, 1:10am UTC](https://discuss.elastic.co/t/move-content-of-sub-fields-as-top-fields/138245 "2018-07-03T01:10:41Z")
Hi, Let's say I have the following example: "entry": { "id": "1234567", "name": "Aryam", "age": 28, "city": "Yerevan" } Is there a plugin that can move the subfields from entry (id, name, age a… --- ## [Private networks with GeoIP](https://discuss.elastic.co/t/private-networks-with-geoip/138099)
**Author:** [@marcelo](https://discuss.elastic.co/u/marcelo)\ **Replies:** 2\ **Last updated:** [July 2, 2018, 11:01pm UTC](https://discuss.elastic.co/t/private-networks-with-geoip/138099 "2018-07-02T23:01:44Z")
Continuing the discussion from Creating geoip data for internal networks: I've been studying a way to have my private networks with GeoIP. I tried to edit the GeoLite2-City.mmdb file and include my networks in it, howe… --- ## [SSL rabbitmq input](https://discuss.elastic.co/t/ssl-rabbitmq-input/137850)
**Author:** [@kraymond](https://discuss.elastic.co/u/kraymond)\ **Replies:** 9\ **Last updated:** [July 2, 2018, 9:04pm UTC](https://discuss.elastic.co/t/ssl-rabbitmq-input/137850 "2018-07-02T21:04:33Z")
I have looked and seen different settings but when input any setting dealing with ssl other than ssl =\> true and verify\_ssl =\> true it states setting unkown. How does logstash know where the certs are if there are no op… --- ## [Basic API Help with http output plugin](https://discuss.elastic.co/t/basic-api-help-with-http-output-plugin/138258)
**Author:** [@terrik](https://discuss.elastic.co/u/terrik)\ **Replies:** 4\ **Last updated:** [July 2, 2018, 8:29pm UTC](https://discuss.elastic.co/t/basic-api-help-with-http-output-plugin/138258 "2018-07-02T20:29:18Z")
Hi all, I am completely new to Logstash, so please forgive my simple question. I am having a little trouble understanding how to use the http output plugin to send some apache logs to a SaaS cloud based log aggregation … --- ## [Problem Ingesting CSV with quoted description field](https://discuss.elastic.co/t/problem-ingesting-csv-with-quoted-description-field/137718)
**Author:** [@EmmaGS](https://discuss.elastic.co/u/EmmaGS)\ **Replies:** 10\ **Last updated:** [July 2, 2018, 8:01pm UTC](https://discuss.elastic.co/t/problem-ingesting-csv-with-quoted-description-field/137718 "2018-07-02T20:01:15Z")
Hello everyone I'm having problems with a csv file with information like this F18AP00207,,,SAI EXEMPT,83641.00,77059.00,160700,0.00,0.00,0.00,0.00,2018-05-29,2018-05-29,2021-06-30,014,DEPARTMENT OF THE INTERIOR (DOI),1… --- ## [Grok Timezone/DateTime parsing issues](https://discuss.elastic.co/t/grok-timezone-datetime-parsing-issues/137500)
**Author:** [@alilland](https://discuss.elastic.co/u/alilland)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 7:47pm UTC](https://discuss.elastic.co/t/grok-timezone-datetime-parsing-issues/137500 "2018-07-02T19:47:37Z")
I have logs that are spitting out in the following format: \[2018-06-24 07:00:03 -0700\] DEBUG - CrewManagement::API - backbone:sync\_tokens - {\\"description\\":\\"Sync local tokens with Backbone-Auth - 114\\",\\"success\\":tru… --- ## [Merging CSV files based on a common key column](https://discuss.elastic.co/t/merging-csv-files-based-on-a-common-key-column/137868)
**Author:** [@Rakesh5216](https://discuss.elastic.co/u/Rakesh5216)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 5:13pm UTC](https://discuss.elastic.co/t/merging-csv-files-based-on-a-common-key-column/137868 "2018-07-02T17:13:19Z")
Hello All, I have been reading the documentation and various posts on elastic community, but still difficult to find solution to this problem. I have 2 csv files, I need to merge into single csv file and inject the data… --- ## [Malformed logstash message](https://discuss.elastic.co/t/malformed-logstash-message/138205)
**Author:** [@Labibme](https://discuss.elastic.co/u/Labibme)\ **Replies:** 7\ **Last updated:** [July 2, 2018, 4:52pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205 "2018-07-02T16:52:42Z")
Hello Everyone, I'm working in a new logstash plateform built from kafka output, archtiecture is as shown below: " filebeat -\> kafka -\> logstash ( first site SSL encryption) -\> logstash ( second site ssl decryption) -\> … --- ## [Mapper\_parsing\_exception error on logstash](https://discuss.elastic.co/t/mapper-parsing-exception-error-on-logstash/138012)
**Author:** [@mksavic](https://discuss.elastic.co/u/mksavic)\ **Replies:** 2\ **Last updated:** [July 2, 2018, 3:54pm UTC](https://discuss.elastic.co/t/mapper-parsing-exception-error-on-logstash/138012 "2018-07-02T15:54:30Z")
Hello all, I have a machine that is currently running ELK (v6.3), and some other machines that are running Metricbeat (v6.3). Currently, Metricbeat is sending information over to Logstash, which in turn sends logs to El… --- ## [Change "null" to null value](https://discuss.elastic.co/t/change-null-to-null-value/137855)
**Author:** [@lelwar](https://discuss.elastic.co/u/lelwar)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 3:34pm UTC](https://discuss.elastic.co/t/change-null-to-null-value/137855 "2018-07-02T15:34:19Z")
Hello, I have an application that sends through the rest API a Json to logstash and this is filtered and outputted to an Elasticsearch cluster. The json is nested, so I have flattened it using the "split" codec in logst… --- ## [Logstash.conf from 5.4 no longer working in 6.3](https://discuss.elastic.co/t/logstash-conf-from-5-4-no-longer-working-in-6-3/137794)
**Author:** [@andrew\_michael](https://discuss.elastic.co/u/andrew_michael)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 3:32pm UTC](https://discuss.elastic.co/t/logstash-conf-from-5-4-no-longer-working-in-6-3/137794 "2018-07-02T15:32:45Z")
Working on my logstash.conf grok statement. It seems the part that is not working is the final bit that says: if \[tags\] == \[\] { mutate { remove\_field =\> \["tags"\] } } This is logstash 6.3. Any idea where I've gone… --- ## [How to implement failover with merely Syslog over UDP](https://discuss.elastic.co/t/how-to-implement-failover-with-merely-syslog-over-udp/138187)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 2\ **Last updated:** [July 2, 2018, 3:02pm UTC](https://discuss.elastic.co/t/how-to-implement-failover-with-merely-syslog-over-udp/138187 "2018-07-02T15:02:55Z")
Hi We have the following setup: Syslog/UDP stream from all of our devices sent to two nginx-loadbalancers which forwards the UDP stream to two logstash nodes. This works great for load-balancing but not for failover. I… --- ## [Logstash to Logstash - invalid certificate?](https://discuss.elastic.co/t/logstash-to-logstash-invalid-certificate/138227)
**Author:** [@FatalGlitch](https://discuss.elastic.co/u/FatalGlitch)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 2:53pm UTC](https://discuss.elastic.co/t/logstash-to-logstash-invalid-certificate/138227 "2018-07-02T14:53:54Z")
Trying to implement logstash to logstash via the instructions at "Logstash-to-Logstash Communication | Logstash Reference \[6.7\] | Elastic" but running into several issues. I'm using a certificate that both upstream and … --- ## [Logstash output failed retrying](https://discuss.elastic.co/t/logstash-output-failed-retrying/138231)
**Author:** [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Replies:** 0\ **Last updated:** [July 2, 2018, 2:34pm UTC](https://discuss.elastic.co/t/logstash-output-failed-retrying/138231 "2018-07-02T14:34:59Z")
Hi all, I am running ELK 6.3 on ubuntu 16.04. I have noticed an error in my logs I am not sure about. Do I not have enough resources dedicated? \[2018-07-02T10:27:20,269\]\[INFO \]\[logstash.outputs.elasticsearch\] retryin… --- ## [Xpack configuration settings not accepted ELK 6.3](https://discuss.elastic.co/t/xpack-configuration-settings-not-accepted-elk-6-3/138221)
**Author:** [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Replies:** 0\ **Last updated:** [July 2, 2018, 2:03pm UTC](https://discuss.elastic.co/t/xpack-configuration-settings-not-accepted-elk-6-3/138221 "2018-07-02T14:03:25Z")
Not sure what is going on. Here is my logstash.yml xpack.security.enabled: false xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.url: "http://localhost:9200" xpack.watcher.enabled: true xpack.reportin… --- ## [Winlogbeat being ingested in both winlog-\* and syslog-\*](https://discuss.elastic.co/t/winlogbeat-being-ingested-in-both-winlog-and-syslog/138218)
**Author:** [@erikjsmith](https://discuss.elastic.co/u/erikjsmith)\ **Replies:** 2\ **Last updated:** [July 2, 2018, 1:56pm UTC](https://discuss.elastic.co/t/winlogbeat-being-ingested-in-both-winlog-and-syslog/138218 "2018-07-02T13:56:33Z")
My Windows Event Collector has Winlogbeat 6.3 on it with it configured to ship to logstash on 5044. I have my logstash listening on both 5044(for winlogbeat) and 5514 (for syslog). My WEC doesn't have filebeat or any ot… --- ## [Possible to loadbalance Logstash via DNS?](https://discuss.elastic.co/t/possible-to-loadbalance-logstash-via-dns/138133)
**Author:** [@rcoundon](https://discuss.elastic.co/u/rcoundon)\ **Replies:** 3\ **Last updated:** [July 2, 2018, 1:16pm UTC](https://discuss.elastic.co/t/possible-to-loadbalance-logstash-via-dns/138133 "2018-07-02T13:16:42Z")
I have a number of remote instances of Logstash that send logs over https to an Elasticsearch cluster. I'm looking to find a way to loadbalance the requests across the nodes in the cluster ideally without having to spec… --- ## [Error with JDK](https://discuss.elastic.co/t/error-with-jdk/137984)
**Author:** [@sunil1](https://discuss.elastic.co/u/sunil1)\ **Replies:** 7\ **Last updated:** [July 2, 2018, 12:31pm UTC](https://discuss.elastic.co/t/error-with-jdk/137984 "2018-07-02T12:31:29Z")
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N OpenJDK 64-Bit Serv… --- ## [Weird Behaviour of Elapsed Filter Plugin](https://discuss.elastic.co/t/weird-behaviour-of-elapsed-filter-plugin/138131)
**Author:** [@rohit831](https://discuss.elastic.co/u/rohit831)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 11:36am UTC](https://discuss.elastic.co/t/weird-behaviour-of-elapsed-filter-plugin/138131 "2018-07-02T11:36:33Z")
I am using elapsed filter plugin for calculating time difference between multiple start/end events for a particular id. if \[StepName\] == "Step1" and \[StepStatus\] == "start" { mutate { add\_tag =\> "Step1\_start" } } el… --- ## [I have around 20 different types of custom generated logs. and want to parse them each based on start and end point as "=-=-=-=-=-"](https://discuss.elastic.co/t/i-have-around-20-different-types-of-custom-generated-logs-and-want-to-parse-them-each-based-on-start-and-end-point-as/138188)
**Author:** [@Avinash\_Kumar](https://discuss.elastic.co/u/Avinash_Kumar)\ **Replies:** 0\ **Last updated:** [July 2, 2018, 11:27am UTC](https://discuss.elastic.co/t/i-have-around-20-different-types-of-custom-generated-logs-and-want-to-parse-them-each-based-on-start-and-end-point-as/138188 "2018-07-02T11:27:48Z")
=-=-=-=-=-=-=-=- Timestamp: Thursday, June 07, 2018 6:38:07 AM Message: HandlingInstanceID: c7ef73de-3169-4220-8bcc-2a235537b5d2 An exception of type 'System.Net.Sockets.SocketException' occurred and was caught. 06/07… --- ## [Filewatch.tailmode.handlers.createinitial open\_file OPEN\_WARN\_INTERVAL is 300](https://discuss.elastic.co/t/filewatch-tailmode-handlers-createinitial-open-file-open-warn-interval-is-300/138175)
**Author:** [@usrjph](https://discuss.elastic.co/u/usrjph)\ **Replies:** 0\ **Last updated:** [July 2, 2018, 10:42am UTC](https://discuss.elastic.co/t/filewatch-tailmode-handlers-createinitial-open-file-open-warn-interval-is-300/138175 "2018-07-02T10:42:19Z")
Hi, Logstash has started outputting the following message sometimes: \[2018-07-02T\*\*10:19\*\*:11,155\]\[WARN \]\[filewatch.tailmode.handlers.createinitial\] open\_file OPEN\_WARN\_INTERVAL is '300' ... \[20… --- ## [Install Logstash without internet connection](https://discuss.elastic.co/t/install-logstash-without-internet-connection/138132)
**Author:** [@PolterFox](https://discuss.elastic.co/u/PolterFox)\ **Replies:** 4\ **Last updated:** [July 2, 2018, 10:11am UTC](https://discuss.elastic.co/t/install-logstash-without-internet-connection/138132 "2018-07-02T10:11:32Z")
Hey guys, I must install logstash on a server which has no active internet connection. But as I know the server needs a connection to resolve the repositories online, is there a possibility to install it without the int… --- ## [Logstash -f is failed to create index - windows environment](https://discuss.elastic.co/t/logstash-f-is-failed-to-create-index-windows-environment/137898)
**Author:** [@lax](https://discuss.elastic.co/u/lax)\ **Replies:** 2\ **Last updated:** [July 2, 2018, 9:02am UTC](https://discuss.elastic.co/t/logstash-f-is-failed-to-create-index-windows-environment/137898 "2018-07-02T09:02:23Z")
logstash version - 6.3.0 running on windows 7 - 64 machine my config file input{ file{ path =\> "c:/lax/station.csv" start\_position =\> "beginning" sincedb\_path =\> "c:/lax" } } filter { csv { separator =\> "… --- ## [Can I restart logstash while beats and ES are running?](https://discuss.elastic.co/t/can-i-restart-logstash-while-beats-and-es-are-running/138137)
**Author:** [@fRzzy](https://discuss.elastic.co/u/fRzzy)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 8:24am UTC](https://discuss.elastic.co/t/can-i-restart-logstash-while-beats-and-es-are-running/138137 "2018-07-02T08:24:13Z")
Currently Beats are pushing logs to LS then LS push processed data to ES, can I safely restart LS while both of them are running and everything will be re-connected without losing any log event? --- ## [\[Manticore::ResolutionFailure\] elasticsearch.k8s: Name or service not known"](https://discuss.elastic.co/t/manticore-resolutionfailure-elasticsearch-k8s-name-or-service-not-known/138136)
**Author:** [@aclowkey](https://discuss.elastic.co/u/aclowkey)\ **Replies:** 0\ **Last updated:** [July 2, 2018, 7:37am UTC](https://discuss.elastic.co/t/manticore-resolutionfailure-elasticsearch-k8s-name-or-service-not-known/138136 "2018-07-02T07:37:10Z")
I'm running Logstash 6.3.0, and Elasticsearch 6.2.0 on a minikube cluster with 1 pod for each. I'm using CoreDNS to have domain aliases, with the following DNS entry: rewrite name elasticsearch.k8s elasticsearch.defaul… --- ## [How to filter on execbeat exec.dtdout in logstash?](https://discuss.elastic.co/t/how-to-filter-on-execbeat-exec-dtdout-in-logstash/138079)
**Author:** [@mathias](https://discuss.elastic.co/u/mathias)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 7:11am UTC](https://discuss.elastic.co/t/how-to-filter-on-execbeat-exec-dtdout-in-logstash/138079 "2018-07-02T07:11:20Z")
Hi, Trying execbeat is see that data is imported to logstash as exec.command, exec.stdout and exec.exitCode. I wonder how I can add tags in logstash for further analysis. In this example I would like to extract the nu… --- ## [Logstash changing date received from filebeat](https://discuss.elastic.co/t/logstash-changing-date-received-from-filebeat/137972)
**Author:** [@kskubala](https://discuss.elastic.co/u/kskubala)\ **Replies:** 3\ **Last updated:** [July 2, 2018, 7:04am UTC](https://discuss.elastic.co/t/logstash-changing-date-received-from-filebeat/137972 "2018-07-02T07:04:52Z")
Hi ! I am using filebeat to send logs to elasticsearch thru logstash. Inside logstash I am using grok to match the pattern where I extract date string. grok { match =\> {"message" =\> \["%{GREEDYDATA:log\_time} \\\[… --- ## [Logstash find pattern help](https://discuss.elastic.co/t/logstash-find-pattern-help/137832)
**Author:** [@rohitnrl](https://discuss.elastic.co/u/rohitnrl)\ **Replies:** 3\ **Last updated:** [July 2, 2018, 6:57am UTC](https://discuss.elastic.co/t/logstash-find-pattern-help/137832 "2018-07-02T06:57:49Z")
Hi, I have a log as following 2018-05-07 11:38:56,748 \[15\] ERROR Library.Logger.Log4NetLogger \[(null)\] - {"Component":"Service","Message":"Exception in AuthorizationBll.CheckAuthorization.Exception:No Roles were found … --- ## [Result different between grok debugger and grok logstash](https://discuss.elastic.co/t/result-different-between-grok-debugger-and-grok-logstash/137917)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 5:10am UTC](https://discuss.elastic.co/t/result-different-between-grok-debugger-and-grok-logstash/137917 "2018-07-02T05:10:57Z")
Hey everybody, I try to parse this log line : \* /data/EDT/batchs/files/logs/E-0001MT-20180423-20180424-0050.log \* My grok : (?\E-\[0-9A-Z\]+)-%{BASE16NUM:DATE\_JOURNEE\_BATCH}-%{BASE16NUM:DATE\_EX… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=729) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=731)