# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1002 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1003 --- ## [Get the right timestamp for old log files](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943)
**Author:** [@simonrisberg](https://discuss.elastic.co/u/simonrisberg)\ **Replies:** 66\ **Last updated:** [April 11, 2017, 2:28pm UTC](https://discuss.elastic.co/t/get-the-right-timestamp-for-old-log-files/24943 "2017-04-11T14:28:02Z")
Hi! I've been trying to get logstash to read old log files for a few days now. I didn't think it worked but the truth is it did. Although the timestamp that kibana 4 makes is the timestamp for when the actual log file … --- ## [\_dateparsefailure for date filtering on database date column value](https://discuss.elastic.co/t/-dateparsefailure-for-date-filtering-on-database-date-column-value/81918)
**Author:** [@ggajanan](https://discuss.elastic.co/u/ggajanan)\ **Replies:** 2\ **Last updated:** [April 11, 2017, 1:58pm UTC](https://discuss.elastic.co/t/-dateparsefailure-for-date-filtering-on-database-date-column-value/81918 "2017-04-11T13:58:12Z")
Hi, I am using jdbc input and want to match @timestamp with the updatedttm column value extracted from the query execution. It is not working and also failing with \_dateparsefailure. Following is the config from log… --- ## [Messages truncated](https://discuss.elastic.co/t/messages-truncated/81869)
**Author:** [@craftytech1](https://discuss.elastic.co/u/craftytech1)\ **Replies:** 3\ **Last updated:** [April 11, 2017, 1:57pm UTC](https://discuss.elastic.co/t/messages-truncated/81869 "2017-04-11T13:57:01Z")
Hello Folks: I have filebeat configured to send the wildfly logs to logstash. They're coming in but I see that the messages themselves, are being truncated. So far the index is using default mapping. I set the tru… --- ## [Parsing McAfee ePO Logs](https://discuss.elastic.co/t/parsing-mcafee-epo-logs/81880)
**Author:** [@julemik](https://discuss.elastic.co/u/julemik)\ **Replies:** 2\ **Last updated:** [April 11, 2017, 1:44pm UTC](https://discuss.elastic.co/t/parsing-mcafee-epo-logs/81880 "2017-04-11T13:44:47Z")
McAfee has recently supported forwarding messages via syslog, and after a bit of groking. I'm left with the meat and potatoes portion of the message that is xml. Here is how I'm parsing it... xml { source =\> "e… --- ## [Filebeat - Logstash connection reset by peer](https://discuss.elastic.co/t/filebeat-logstash-connection-reset-by-peer/81829)
**Author:** [@Jiri\_Petak](https://discuss.elastic.co/u/Jiri_Petak)\ **Replies:** 3\ **Last updated:** [April 11, 2017, 12:51pm UTC](https://discuss.elastic.co/t/filebeat-logstash-connection-reset-by-peer/81829 "2017-04-11T12:51:21Z")
Hi, i am facing problem with filebeat sending logs to logstash. My architecture: filebeat (versions: 5.3.0 and 5.2.2 - tried both) -\> logstash-broker (versions: 5.2.2 and 5.3.0) -\> redis-cache -\> logstash-indexer -\> elas… --- ## [Any plans to support GeoIP2 ISP database in logstash?](https://discuss.elastic.co/t/any-plans-to-support-geoip2-isp-database-in-logstash/81982)
**Author:** [@madkoala](https://discuss.elastic.co/u/madkoala)\ **Replies:** 0\ **Last updated:** [April 11, 2017, 12:32pm UTC](https://discuss.elastic.co/t/any-plans-to-support-geoip2-isp-database-in-logstash/81982 "2017-04-11T12:32:21Z")
Legacy ISP database was supported in logstash 2.x. It was very helpful. Do you have any plan to support GeoIP2 ISP database in logstash 5.x? --- ## [I want to aggregate the JSON log events,How do i filter it in the logstash](https://discuss.elastic.co/t/i-want-to-aggregate-the-json-log-events-how-do-i-filter-it-in-the-logstash/81743)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 2\ **Last updated:** [April 11, 2017, 11:12am UTC](https://discuss.elastic.co/t/i-want-to-aggregate-the-json-log-events-how-do-i-filter-it-in-the-logstash/81743 "2017-04-11T11:12:37Z")
i want to aggregate the JSON log events,from the filebeats to logstash. i want to filter the warn ,error,info separately from the Log events How do i filter it in the logstash. Can anyone please guide me.. --- ## [Data missed in Logstash!](https://discuss.elastic.co/t/data-missed-in-logstash/81904)
**Author:** [@zhaochl](https://discuss.elastic.co/u/zhaochl)\ **Replies:** 4\ **Last updated:** [April 11, 2017, 10:03am UTC](https://discuss.elastic.co/t/data-missed-in-logstash/81904 "2017-04-11T10:03:22Z")
Data missed a lot in logstash ,version 5.0, is it a serous bug ? --- ## [Logstash 5.2 service doesn't start](https://discuss.elastic.co/t/logstash-5-2-service-doesnt-start/81947)
**Author:** [@paolo1](https://discuss.elastic.co/u/paolo1)\ **Replies:** 2\ **Last updated:** [April 11, 2017, 9:57am UTC](https://discuss.elastic.co/t/logstash-5-2-service-doesnt-start/81947 "2017-04-11T09:57:03Z")
Hi, Is anyone having problems starting logstash as a service (sudo systemctl start logstash) rather than using (sudo bin/logstash --path.settings /etc/logstash)? It works using the latter, but not the former. Output … --- ## [How to import the old event from file to es use logstash](https://discuss.elastic.co/t/how-to-import-the-old-event-from-file-to-es-use-logstash/81905)
**Author:** [@zhaochl](https://discuss.elastic.co/u/zhaochl)\ **Replies:** 2\ **Last updated:** [April 11, 2017, 9:40am UTC](https://discuss.elastic.co/t/how-to-import-the-old-event-from-file-to-es-use-logstash/81905 "2017-04-11T09:40:21Z")
start\_position =\> "beginning" ignore\_older =\> 2592000 #7 day before ignore I use this config ,but find data missing, and I am not sure ,when to use start\_position =\> "beginning" and when to use start\_position =\> "end"… --- ## [How to Upload data into elastic search from oracle database using logstash by creating multiple indexes](https://discuss.elastic.co/t/how-to-upload-data-into-elastic-search-from-oracle-database-using-logstash-by-creating-multiple-indexes/81910)
**Author:** [@Sachin\_Maurya](https://discuss.elastic.co/u/Sachin_Maurya)\ **Replies:** 1\ **Last updated:** [April 11, 2017, 8:15am UTC](https://discuss.elastic.co/t/how-to-upload-data-into-elastic-search-from-oracle-database-using-logstash-by-creating-multiple-indexes/81910 "2017-04-11T08:15:43Z")
I am trying to upload data into ES using logstash. I want to store the data by creating multiple indexes into ES with the help of a field( containing some integer values) in my database. I am trying to use filter but not… --- ## [Tag with pattern used](https://discuss.elastic.co/t/tag-with-pattern-used/81868)
**Author:** [@lask001](https://discuss.elastic.co/u/lask001)\ **Replies:** 2\ **Last updated:** [April 11, 2017, 6:08am UTC](https://discuss.elastic.co/t/tag-with-pattern-used/81868 "2017-04-11T06:08:11Z")
Below is a basic example of what I'm using for my setup: I have a patterns file that's defined and works correctly. patterns %{pattern1}|%{pattern2}|%{pattern3} My filter is similar to the following: filter { i… --- ## [DataLoss in Logstash!](https://discuss.elastic.co/t/dataloss-in-logstash/27970)
**Author:** [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Replies:** 11\ **Last updated:** [April 11, 2017, 3:13am UTC](https://discuss.elastic.co/t/dataloss-in-logstash/27970 "2017-04-11T03:13:16Z")
Hi, I'm using ELK for Centralized logging and i'm facing DataLoss while processing 5lakh logs(kibana hits), (Logstash (Shipper&Indexer same instance)) Machine 1 -\> (ElasticSearch -\> Kibana) Machine 2 Scenario for Da… --- ## [Converting columns in SQL table to nested datatype using Logstash JDBC plugin](https://discuss.elastic.co/t/converting-columns-in-sql-table-to-nested-datatype-using-logstash-jdbc-plugin/81871)
**Author:** [@Rory](https://discuss.elastic.co/u/Rory)\ **Replies:** 0\ **Last updated:** [April 10, 2017, 8:56pm UTC](https://discuss.elastic.co/t/converting-columns-in-sql-table-to-nested-datatype-using-logstash-jdbc-plugin/81871 "2017-04-10T20:56:07Z")
Hi, I have a table in SQL Server which looks like this: The data is in an un-pivoted format: CustomerID, AddressNumber, AddressValue 1, 1, a 1, 2, b 1, 3, c 1, 4, d 1, 5, e 2, 1, f 2, 2, g 2, 2, h 2, 2, i 2, 2, k .… --- ## [Unauthorized logstash role for bulk writes](https://discuss.elastic.co/t/unauthorized-logstash-role-for-bulk-writes/79318)
**Author:** [@addyk](https://discuss.elastic.co/u/addyk)\ **Replies:** 10\ **Last updated:** [April 10, 2017, 8:10pm UTC](https://discuss.elastic.co/t/unauthorized-logstash-role-for-bulk-writes/79318 "2017-04-10T20:10:15Z")
Here are my current role that is attached to the user logstash I know create\_index and create aren't necessary but I figured I would give that at try. I had originally followed this guide https://www.elastic.co/guid… --- ## [How to split a Json filed into two different words based on a certain character?](https://discuss.elastic.co/t/how-to-split-a-json-filed-into-two-different-words-based-on-a-certain-character/81714)
**Author:** [@rajesh](https://discuss.elastic.co/u/rajesh)\ **Replies:** 2\ **Last updated:** [April 10, 2017, 8:00pm UTC](https://discuss.elastic.co/t/how-to-split-a-json-filed-into-two-different-words-based-on-a-certain-character/81714 "2017-04-10T20:00:14Z")
I have a Json object as below, {"environment":"xxx", "feature":"first\_attempt", "scenario":"second scenario", "project":"first"} For the field "feature", I am looking for splitting the word into two separated by '\_'. B… --- ## [Logstash Monitoring Overview incorrect](https://discuss.elastic.co/t/logstash-monitoring-overview-incorrect/81831)
**Author:** [@\_Ian](https://discuss.elastic.co/u/_Ian)\ **Replies:** 1\ **Last updated:** [April 10, 2017, 6:53pm UTC](https://discuss.elastic.co/t/logstash-monitoring-overview-incorrect/81831 "2017-04-10T18:53:29Z")
While using monitoring in 5.3 we've noticed that the Logstash overview appears to be summarizing the EPS incorrectly. It appears to be displaying a single logstash node EPS count rather than a summary of all active Logst… --- ## [Logstash-elasticsearch integration after installing x-pack](https://discuss.elastic.co/t/logstash-elasticsearch-integration-after-installing-x-pack/81851)
**Author:** [@harshavardan25000](https://discuss.elastic.co/u/harshavardan25000)\ **Replies:** 1\ **Last updated:** [April 10, 2017, 5:46pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-integration-after-installing-x-pack/81851 "2017-04-10T17:46:54Z")
I installed x pack 5.2.0 on logstash, elasticsearch and kibana and left all the pieces untouched. Created logstash user "logstash\_internal " as per the instructions. But no luck when i tried to integrate logstash with el… --- ## [Importing data from Graylog to Logstash](https://discuss.elastic.co/t/importing-data-from-graylog-to-logstash/81859)
**Author:** [@rohithmen](https://discuss.elastic.co/u/rohithmen)\ **Replies:** 1\ **Last updated:** [April 10, 2017, 5:32pm UTC](https://discuss.elastic.co/t/importing-data-from-graylog-to-logstash/81859 "2017-04-10T17:32:50Z")
Hi All, I have been working on vendor project helping analyze log data. We have currently setup graylog server for our log management but feel restricted with visualizing some of these data. We were wondering if we cou… --- ## [How to consider multiple lines(starts with Timestamp all the time) as a single event?](https://discuss.elastic.co/t/how-to-consider-multiple-lines-starts-with-timestamp-all-the-time-as-a-single-event/81132)
**Author:** [@rajesh](https://discuss.elastic.co/u/rajesh)\ **Replies:** 3\ **Last updated:** [April 10, 2017, 5:26pm UTC](https://discuss.elastic.co/t/how-to-consider-multiple-lines-starts-with-timestamp-all-the-time-as-a-single-event/81132 "2017-04-10T17:26:12Z")
20170315:08:00:34:022156 xxxxx-\[INFO\]:-Dump status report 20170315:08:00:34:022156 xxxxx-\[INFO\]:---------------------------------------------------- 20170315:08:00:34:022156 xxxxx-\[INFO\]:-Target database … --- ## [Access field name with space in config?](https://discuss.elastic.co/t/access-field-name-with-space-in-config/64383)
**Author:** [@glagidse](https://discuss.elastic.co/u/glagidse)\ **Replies:** 2\ **Last updated:** [April 10, 2017, 4:37pm UTC](https://discuss.elastic.co/t/access-field-name-with-space-in-config/64383 "2017-04-10T16:37:38Z")
I assign field names in from a csv into columns, one of them is called "Primary Key" I've tried a lot of combinations, but I can't figure out how to assign it to the document index. document\_id =\> "%{Primary Key}" j… --- ## [ELK 5.2.1 - still can't get geoip working](https://discuss.elastic.co/t/elk-5-2-1-still-cant-get-geoip-working/76420)
**Author:** [@paka](https://discuss.elastic.co/u/paka)\ **Replies:** 17\ **Last updated:** [April 10, 2017, 4:29pm UTC](https://discuss.elastic.co/t/elk-5-2-1-still-cant-get-geoip-working/76420 "2017-04-10T16:29:27Z")
Hello, I was already asking about this on Jan 12, but with no solution. Meanwhile I have upgraded to ELK 5.2.1 so I tried again, but still can't get geoip working. The logstash filter is minimalistic: … --- ## [Lost gelf logs](https://discuss.elastic.co/t/lost-gelf-logs/79924)
**Author:** [@jazz](https://discuss.elastic.co/u/jazz)\ **Replies:** 4\ **Last updated:** [April 10, 2017, 2:12pm UTC](https://discuss.elastic.co/t/lost-gelf-logs/79924 "2017-04-10T14:12:02Z")
We have an ELK cluster with 3 hosts on Ubuntu 14.04 (logstash 2.4.1+ES 2.4.1 on each host in Docker containers), with logstash configured to receive gelf and syslog/tcp logs. Sometimes some gelf logs are lost. With … --- ## [Logstash substring a field using position](https://discuss.elastic.co/t/logstash-substring-a-field-using-position/32569)
**Author:** [@samnik60](https://discuss.elastic.co/u/samnik60)\ **Replies:** 5\ **Last updated:** [April 10, 2017, 1:58pm UTC](https://discuss.elastic.co/t/logstash-substring-a-field-using-position/32569 "2017-04-10T13:58:44Z")
hi guys, if i want to extract first 3 characters from a field, how do i go about it. Thanks, sam --- ## [Where to have logic to run everytime a new event occurs](https://discuss.elastic.co/t/where-to-have-logic-to-run-everytime-a-new-event-occurs/81662)
**Author:** [@shabirmean](https://discuss.elastic.co/u/shabirmean)\ **Replies:** 2\ **Last updated:** [April 10, 2017, 1:30pm UTC](https://discuss.elastic.co/t/where-to-have-logic-to-run-everytime-a-new-event-occurs/81662 "2017-04-10T13:30:01Z")
Hi, I'm writing a new input plugin. I have a need to run some logic everytime a new event is captured by Logstash. Based on the outcome of the logic I want to decide whether the event will be further processed or will … --- ## [Convert UNIX epoch to local time](https://discuss.elastic.co/t/convert-unix-epoch-to-local-time/81754)
**Author:** [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Replies:** 5\ **Last updated:** [April 10, 2017, 11:45am UTC](https://discuss.elastic.co/t/convert-unix-epoch-to-local-time/81754 "2017-04-10T11:45:17Z")
Hi, Hope I'm in the right section and will form the right question. So my situation. I'm using logstash+elasticsearch+kibana for parsin some logs, CSV. Parsing works perfectly. I've got two date fieds, submission… --- ## [Installing Gem for Ruby Filter Plugin](https://discuss.elastic.co/t/installing-gem-for-ruby-filter-plugin/79994)
**Author:** [@uschtwill](https://discuss.elastic.co/u/uschtwill)\ **Replies:** 10\ **Last updated:** [April 10, 2017, 10:57am UTC](https://discuss.elastic.co/t/installing-gem-for-ruby-filter-plugin/79994 "2017-04-10T10:57:23Z")
Hello there, I am trying to install the jdbc-mysql gem in a dockerized Logstash (after I miserably failed to set Xcext.enabled=true and use the mysql2 gem). In the Dockerfile I am first running this RUN env GEM\_HOM… --- ## [Event.sprintf question for use in logstash-filter-elasticsearch](https://discuss.elastic.co/t/event-sprintf-question-for-use-in-logstash-filter-elasticsearch/81758)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 1\ **Last updated:** [April 10, 2017, 10:51am UTC](https://discuss.elastic.co/t/event-sprintf-question-for-use-in-logstash-filter-elasticsearch/81758 "2017-04-10T10:51:41Z")
Hi, Is there any way to get event.sprintf to print the entire event, not just individual fields? I'm trying to build a percolator query that contains the entire logstash event. What I'd like my percolator query to … --- ## [Add\_tag doesn't work in translate filter in some circumstances](https://discuss.elastic.co/t/add-tag-doesnt-work-in-translate-filter-in-some-circumstances/81802)
**Author:** [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\ **Replies:** 0\ **Last updated:** [April 10, 2017, 10:38am UTC](https://discuss.elastic.co/t/add-tag-doesnt-work-in-translate-filter-in-some-circumstances/81802 "2017-04-10T10:38:17Z")
I am rewriting a field using translate. Here is the config: translate { add\_tag =\> \[ "monitoring" \] field =\> "container" regex =\> true exact =\> true dictionary =\> \[ 'sensu-client-\*', "sensu-client",… --- ## [Logstash output-elasticsearch action "update" not work 4 me](https://discuss.elastic.co/t/logstash-output-elasticsearch-action-update-not-work-4-me/81799)
**Author:** [@tonyjames3212](https://discuss.elastic.co/u/tonyjames3212)\ **Replies:** 0\ **Last updated:** [April 10, 2017, 10:17am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-action-update-not-work-4-me/81799 "2017-04-10T10:17:18Z")
version:v5.3.0 https://www.elastic.co/guide/en/logstash/5.2/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action And my out put: elasticsearch { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1001) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1003)