Formating field dependent output

(Jon) #1

tl;dr: Can sprintf format in logstash do more than just date formatting based on @timestamp?

Long Version:

I'm in the process of moving the log collection for my application from essentially a proprietary mishmash of services to using Elastic as a centralized repository. This setup is mostly used by other developers on the team and QA. The old workflow was essentially the application reporting plain text logs to a service running on each developer's pc where they were simply dumped into flat files in a directory on the pc. For a lot of reasons, this was terrible (for instance, it was a pain to set up so only a few people used it).

Now I've migrated to having the application report directly to a central ES host so every log for every instance is saved and searchable (it rocks). However, the few developers who used the old setup want to keep their existing workflow, due to the fact that they fear change. Conveniently Logstash is very helpful here, I setup a Logstash configuration that lets people redirect their logging to Logstash, which then uses a file output to log to disk and an Elasticsearch output to forward the data on as if nothing happened.

Now to the crux of my problem: The data being logged to disk is essentially in the format of [%{[field1]}][%{[field2]}]...[%{[fieldN]}]: %{[message]} where the various fields before the message are all numbers. The old proprietary system would pad out the numbers so that all the columns lined up, which was nice for easy viewing. My logstash way doesn't do that, which has lead to much whinging because change is bad and scary.

So my question: is there any way to format the output in Logstash using it's sprintf syntax so that I can pad fields with spaces (or 0s or whatever)? Essentially so that a field with 1 in some cases and 10000 in others would be ouput as: "[ 1]" in the first case and "[10000]" in the later case. The size of the justification is predetermined (i.e. there will always be 5 digits max in this location, 10 in that one, etc).

(system) #2