# Can you stream all contents of the log file to ELK? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 20, 2016, 5:32pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687 "2016-01-20T17:32:45Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 20, 2016, 5:32pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/1 "2016-01-20T17:32:45Z")
If I forward everything in the log to ELK without creating index first, would I be able to create charts on every entry in the log. Is this recommended? Sincere there are many servers and apps, creating index could become cumbersome. What do you recommend to for this type cases? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 20, 2016, 7:39pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/2 "2016-01-20T19:39:47Z")
Elasticsearch (ES) stores all data in indexes. Indexes in ES can roughly be compared to databases in the relational database world. Hence, what you're saying doesn't quite make sense. Forwarding all logs to ES is a standard use case and nothing unusual. ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 20, 2016, 7:52pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/3 "2016-01-20T19:52:04Z")
what was I trying to say is that, I have multiple log files in all different applications. Should I go though my files first to see what I would need to capture and create indeces and fields on based on what I need first. Or, should I start forwarding the logs to ES and have the ES create the indeces? can you tell me the best way? I need to be able to catpure all data with limited effort and be able to chart all data. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 20, 2016, 9:03pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/4 "2016-01-20T21:03:20Z")
There's no point in pre-creating any indexes or fields—they will be created automatically based on the events that are sent from Logstash—but you may want to filter out some less interesting events in Logstash. ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 21, 2016, 5:30pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/5 "2016-01-21T17:30:32Z")
Hi Magnus, The problem is that we dont have the ELK version of the logstash client since it is Solaris env. Therefore, streaming live data from logs not working for us. Do you know of any client for Solaris or any work around? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 21, 2016, 6:11pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/6 "2016-01-21T18:11:51Z")
Perhaps [Logstash on Solaris - stat.st\_gid unsupported or native support failed to load [SOLVED]](https://discuss.elastic.co/t/logstash-on-solaris-stat-st-gid-unsupported-or-native-support-failed-to-load-solved/24166) is helpful. You can probably also obtain a Go compiler that runs on Solaris, and that would allow you to build Filebeat. NXLog could be yet another option. ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 21, 2016, 6:15pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/7 "2016-01-21T18:15:15Z")
Hi Magnus, If I push unstructured data to ES with these clients, would I be able to create charts, table etc on them? When data comes to ES, how does ES creates fields/indices automatically? I'd rather push everything to ES and let ES create the indices and fields. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 21, 2016, 6:21pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/8 "2016-01-21T18:21:37Z")
> If I push unstructured data to ES with these clients, would I be able to create charts, table etc on them? To some extent, but to be really useful you need to write Logstash filters to process the raw data. > When data comes to ES, how does ES creates fields/indices automatically? I'd rather push everything to ES and let ES create the indices and fields. Elasticsearch won't extract any fields from unstructured data on its own. You need Logstash for that. ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 21, 2016, 6:23pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/9 "2016-01-21T18:23:50Z")
Basically, I'd need to know what I need to extract from the logs. What if the logs changes? The solution needs to be dynamic. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 21, 2016, 6:58pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/10 "2016-01-21T18:58:25Z")
You can write Logstash filters that support multiple log formats, but in the end there is no magic going on. The filters need to support the log file formats you'll encounter. ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 21, 2016, 7:24pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/11 "2016-01-21T19:24:51Z")
Hi Magnus, I need to set up a call with ELK people to understand all of these different snerios, any ideas? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 21, 2016, 7:26pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/12 "2016-01-21T19:26:58Z")
Use [Elastic's contact form](https://www.elastic.co/contact) to get in touch with them? ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 21, 2016, 7:39pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/13 "2016-01-21T19:39:09Z")
Magnus, Thank you for responding to my questions. I have one more question. Let's say we're sending data based on some filters to ES. Let's say the entries in the logs change order in the app log, is there a way that you know we can update our filters dynamically to account for changes in the logs. for example timestamp, cpu usage, memory usage, load averages but this format in the log file changes to this: timestamp, memory usage, load averages, cpu usage Any thoughts? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 21, 2016, 8:37pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/14 "2016-01-21T20:37:50Z")
If the changes are reasonably predictable you could supply multiple patterns to parse the input messages (so that Logstash tries one after another until it gets a match), but there's no built-in intelligence to dynamically guess log formats. ---
**Author:** ![gsaray101](https://avatars.discourse-cdn.com/v4/letter/g/df788c/32.png) [@gsaray101](https://discuss.elastic.co/u/gsaray101)\ **Post date:** [January 21, 2016, 8:44pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/15 "2016-01-21T20:44:34Z")
would I be able to apply a script to fields to order them. Actual content does not change but it looks liek the order of the fields change. Before aggregating the data, would I be able to execute a script, order the fields then send them to ES? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 21, 2016, 9:35pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/16 "2016-01-21T21:35:15Z")
Yes, you could do that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:22pm UTC](https://discuss.elastic.co/t/can-you-stream-all-contents-of-the-log-file-to-elk/39687/17 "2017-07-05T23:22:23Z")