Running Logstash from .NET Service

Elastic Stack Logstash
1

Hi there,

I am working on large scale data ingestion into Elastic Search indexes.

We have a number of database records that we want to ingest into ES after we have detected delta changes in the database. We have .NET Services that run and works out delta changes these delta changes are created as CSV files along with matching config files.

These pair of files are then picked up and passed into logstash using command logstash -f myconfig.conf

We have noticed that the initial start up times are quite a lot when we call the command, is there a way or a setting to streamline the start up times?

We are running a number of data changes and have started experiencing that logstash itself becomes a bottleneck.

Any advice will be greatly appreciated.

2

Is myconfig.conf different each time you run Logstash? How do the files differ?

3

Hi Magnusbaeck,

Thanks for your response.

Yes these differ with each run, we have .NET services creating a unique set of CSV files with data, each of these files is paired with a config file that varies on type of data.

config files also have a bit of coding logic to transform the data.

Thanks

4

Okay, but then you should be able to run a single Logstash instance and just reload its configuration whenever you've created a new config file. Reloading the configuration is a lot faster than firing up a new JVM. You'll have to tag each event in the input so that you can pair them with the right filters and outputs.

5

Thanks Magnus,

Would you have an example of this ? that will help us immensely.

Also I have been going through the forums and on the web for how to detect when file input plugin has finished ingesting data, this is like a secondary trigger for us to verify that data is in elastic.

Again when i search here and also on git forums, i found all sorts of suggestions but nothing that we could utilise.

Next time i am in your town.. i will definitely buy you a drink :slight_smile:

6

Would you have an example of this ? that will help us immensely.

input {
  file {
    ...
    tags => ["some unique identifier"]
  }
}

filter {
  if "some unique identifier" in [tags] {
    ...
  }
}

output {
  if "some unique identifier" in [tags] {
    ...
  }
}

Also I have been going through the forums and on the web for how to detect when file input plugin has finished ingesting data, this is like a secondary trigger for us to verify that data is in elastic.

Your best bet it to read and parse the sincedb file.

7

Thanks Magnus

In this case would we just pass a single config file ?

I am sorry if I am asking too many questions, ideally the scenario that we have is as follows

logstash -f 1.conf

.... finishes ingestion

now run

logstash -f 2.conf

--- finsh ingestion.

Best possible solution would be that we start logstash once and then keep passing arguments into it for various config files..

so something like

logstash.exe

.... 1.conf

ingestion completed

2.conf

Any help here will be really helpful

8

Start a single Logstash process with logstash -f path/to/directory and add/remove files in that directory as necessary. Enable Logstash's autoreload of config files or explicitly ask Logstash to reload the config when you've made changes (but I don't know if that works on Windows).

9

That is such a great suggestion and can definitely work.

Would logstash allow us to delete a log file when it finishes processing it ?

10

Would logstash allow us to delete a log file when it finishes processing it ?

Logstash itself doesn't care if files are deleted but Windows might not allow the file deletion while Logstash has the file open. You may have to tune the file input's close_older option.

11

Thank you so much @magnusbaeck you have been remarkable.

12

Hi @magnusbaeck

I am back with another query it seems. I tried the approach that you suggested of running scans on a pre configured folder location.

I have noticed that once the file is pickedup by logstash we start seeing following error messages

[2017-11-14T22:14:09,135][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server
[2017-11-14T22:14:39,131][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"undefined local variable or method `bad_response_error' for #<LogStash::LicenseChecker::LicenseReader:0x7cae3ad8>", :class=>"NameError"}
[2017-11-14T22:14:39,140][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server
[2017-11-14T22:15:09,131][ERROR][logstash.licensechecker.licensemanager] Unable to retrieve license information from license server {:message=>"undefined local variable or method `bad_response_error' for #<LogStash::LicenseChecker::LicenseReader:0x7cae3ad8>", :class=>"NameError"}
[2017-11-14T22:15:09,133][WARN ][logstash.licensechecker.xpackinfo] Nil response from License Server

Any suggestions on how we configure licensing?

Thanks
Sunny

13

I am back with another query it seems.

So why not start a new topic?

Any suggestions on how we configure licensing?

No, I don't know anything about licenses.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.