# \[sincedb creation\]

**URL:** <https://discuss.elastic.co/t/sincedb-creation/34851>\
**Category:** Logstash\
**Created:** [November 17, 2015, 9:00pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851 "2015-11-17T21:00:13Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 1, 2015, 4:55am UTC](https://discuss.elastic.co/t/sincedb-creation/34851/21 "2015-12-01T04:55:41Z")

</div>

What needs to be clarified? The more specific question you ask the more specific answer you'll get. Right now I have no idea what you're asking or even what the problem is.

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 3, 2015, 6:05pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/22 "2015-12-03T18:05:00Z")

</div>

Hello magnus Sorry for any confusion .  
Actually I have modified certain things to speed up logging things .

This is my code to generate more than 1 Gb file. Now I am not using Nlogger anymore  
static void Main(string[] args)  
{  
Stopwatch sw = Stopwatch.StartNew();  
string folderName = @"D:\Log";  
string jobId = System.Guid.NewGuid().ToString();  
jobId = jobId + "\MIPS";  
string pathString = System.IO.Path.Combine(folderName, jobId);  
System.IO.Directory.CreateDirectory(pathString);

```
        string fileName = "Mipslogger.log";
        pathString = System.IO.Path.Combine(pathString, fileName);
        using (System.IO.FileStream fs = System.IO.File.Create(pathString))
        {
           
        }
        using (System.IO.StreamWriter file =
       new System.IO.StreamWriter(pathString))
        {
            for (var line = 0; line < (200000000 + ( 100)); line++)
            {
                file.WriteLine("MIP : " + line);
            }
        }
        sw.Stop();
        Console.WriteLine("Time taken: {0}ms", sw.Elapsed.TotalMilliseconds);
        
    }

```

This is my config file .for logstatsh  
input {  
file{  
path =\> "D:/Log/_/MIPS/_"  
start\_position =\> "beginning"  
sincedb\_write\_interval =\> 0  
}

}

filter{  
mutate{  
gsub =\> ["path","D:/Log/", ""]   
}

}

output {  
stdout {  
codec =\> line {  
format =\> "%{path}"  
}  
}

}

Below are the timing info I am getting  
30752ms (without logstash running with log generation)  
340478ms (when log is being generated parallel with logstash running)

And I am seeing one the logger file generated logstasth keeps on running to do its filitering .I am facing two issue

You can see sincedb\_write\_interval is 0 but I am not seeing since db creating for a very long time .  
Is there any way I can speed up the logstash processing .I am seeing log file is generated with couple of minutes but logstash keeps on running until i pressed ctrl-c.

Please provide your provide .

Thanks and Regards  
Varun

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 3, 2015, 7:16pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/23 "2015-12-03T19:16:56Z")

</div>

> Below are the timing info I am getting  
> 30752ms (without logstash running with log generation)  
> 340478ms (when log is being generated parallel with logstash running)

_Ten times_ slower just because Logstash is monitoring the file? I find that very hard to believe. Wait... you have `sincedb_write_interval` set to zero. That would probably cause Logstash to rewrite the file after every single read operation. What if you comment out that line?

Again, use Process Monitor to record what going on.

> I am seeing log file is generated with couple of minutes but logstash keeps on running until i pressed ctrl-c.

That's by design. Logstash's file input is tailing the file and waiting for more lines to be added. Right now the only way of knowing when Logstash is done with a file is to monitor the sincedb file.

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 3, 2015, 9:07pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/24 "2015-12-03T21:07:23Z")

</div>

Hello Magnus ,  
I agree with your point no 2. But my doubt is sincedb file was not created I have tried keeping sincedb\_write\_interval 0 and by default also even though I am seeing logstash keeps on running .I am seeing this issue recently only when I am not generating logs with Nlog rather I am generating log and folder with the C# FILE File write operation which I have mentioned above .Please have a look on that thing if possible .and Here are my results when I have removed sinc\_db\_write file interval from my config file

3Gb File Size  
72767.4751ms (without logstash )  
96086.1531ms(using with logstash )

Following are the snippets from process monitor

 ![](https://us1.discourse-cdn.com/elastic/original/2X/e/e7875bc40ec3a33528b75fd3324461e0b655d666.png)

 ![](https://us1.discourse-cdn.com/elastic/original/2X/4/442acdb2e20e1558967eb7891032cd495c62559e.png)

 ![](https://us1.discourse-cdn.com/elastic/original/2X/d/df0bf5bc91bfc23f40919258b8ffb8fc2896fdd6.png)

 ![](https://us1.discourse-cdn.com/elastic/original/2X/f/f04441d6341efbd36cc912e41c2139b1f942a488.png)

I am seeing there is some delay added by the logstash .

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 4, 2015, 6:47am UTC](https://discuss.elastic.co/t/sincedb-creation/34851/25 "2015-12-04T06:47:23Z")

</div>

> I am seeing there is some delay added by the logstash .

What do you mean? The Process Monitor screen shots you posted show the ingestion rate of Logstash (about 142 kB/s). But what we're interested in is the speed at which ConsoleApplication24.exe can write to the log file, right? In that case the interesting metrics are

a) the duration of each WriteFile request and  
b) the time between each Writefile request

with and without Logstash running. Plotting each metric in a histogram would be interesting—is there a tiny slowdown in each write operation, are there outlier latencies, or can we learn anything else?

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 8, 2015, 6:14pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/26 "2015-12-08T18:14:29Z")

</div>

Hello magnus ,  
After a heavy discussion and debugging with Nlog people finally we got some fix for their performance issue .Looking for similar kind of cooperation for logstash also.We have two issues currently .

Can you please check from your side also that does the performance of logging affected when logstash is simultaneously running

The time taken by logstash for parsing is slightly high even though logging has stopped.Logstash keep on running for a long time

Kindly get me resolve this issue with high priority

Thanks and Regards  
Varun

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 8, 2015, 6:29pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/27 "2015-12-08T18:29:19Z")

</div>

I'm afraid I don't have the time and interest to debug Logstash performance on Windows. Seeing the histograms I talked about in my previous post would still be interesting though. Filing a GitHub issue with some concrete data might be a way forward for you.

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 8, 2015, 6:59pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/28 "2015-12-08T18:59:48Z")

</div>

Hello magnus ,  
I have raised a case on github but wont u mind if u can just confirm that have seen this kind pf performance issue earlier or is that i am doing some mistake from my side .

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 8, 2015, 7:08pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/29 "2015-12-08T19:08:01Z")

</div>

> I have raised a case on github

Indeed: [[Logstash Performance Issue] · Issue #4318 · elastic/logstash · GitHub](https://github.com/elastic/logstash/issues/4318)

If you want to get any traction with it I suggest you add additional details and benchmarks. I added a link to this thread but it's a good idea to summarize it and, again, supply data.

> but wont u mind if u can just confirm that have seen this kind pf performance issue earlier or is that i am doing some mistake from my side .

I've never seen any such performance problems but I've also never run Logstash on Windows.

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 8, 2015, 7:16pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/30 "2015-12-08T19:16:37Z")

</div>

Hello magnus ,  
Thanks for your support.Really appreciate the kind of feedback you have given to me .

Thanks and Regards  
Varun

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 9, 2015, 8:24pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/31 "2015-12-09T20:24:03Z")

</div>

hello magnus ,  
I am not sure this is teh right question to ask to you or not but can you tell me In logstash config file I want to redirect in a output in a file whose location is on shared .I am the shared location path in this way  
file{  
path =\> "\shared-01\Share\Temporary\_Shared\_Space\varun"  
message\_format =\> "%{message}"  
flush\_interval =\> 0  
}

But its showing the error

2b0f2125-1775-4ad4-bd84-7ccc720a09b4/MIP/MipLogger.log  
Errno::ENOENT: No such file or directory - \shared-01  
mkdir at org/jruby/RubyDir.java:458

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 9, 2015, 8:59pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/32 "2015-12-09T20:59:46Z")

</div>

For new questions please start a new thread.

Try forward slashes instead of backslashes (and make sure you start with a double slash).

---

<div class="post-metadata">

**Author:** ![varun\_kumar1](https://avatars.discourse-cdn.com/v4/letter/v/ac91a4/32.png) [@varun\_kumar1](https://discuss.elastic.co/u/varun_kumar1)\
**Post date:** [December 9, 2015, 9:19pm UTC](https://discuss.elastic.co/t/sincedb-creation/34851/33 "2015-12-09T21:19:02Z")

</div>

> [@\[Logstash configuration when dumping the result file in shared network location\]](https://discuss.elastic.co/t/logstash-configuration-when-dumping-the-result-file-in-shared-network-location/36802):
>
> Hello, I am trying to parse the input log file and sending the result to a file in a shared network location .Below is my configuration file{ path =\> "\optym-01/Share/Temporary\_Shared\_Space/varun/Logs/%{host}/%{path}" message\_format =\> "%{message}" flush\_interval =\> 0 But i am getting the below error Errno::ENOENT: No such file or directory - \optym-01 mkdir at org/jruby/RubyDir.java:458 fu\_mkdir at C:/Users/varunk/Downloads/logstash-2.0.0/logstash-2.0.0/vendo r/jruby/lib/ruby/1.9/file…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:19am UTC](https://discuss.elastic.co/t/sincedb-creation/34851/34 "2017-07-06T05:19:15Z")

</div>



[Previous page](https://discuss.elastic.co/t/sincedb-creation/34851.md?page=1)