Help needed for reading Logs


(Hari Baskar) #1

Hi there,
Could someone guide me on how to use pipeline(|) as separator for the Logs.
I tried gsub. But everytime I use gsub, it says non ascii character found in the log.
Any help would be appreciated..

Regards,
Hari


(Magnus Bäck) #2

Please show an example of the kind of log you want to process. Please show us what you've tried so far. It's not clear why you'd want to use gsub. It sounds like a simple csv filter would work.


(Hari Baskar) #3

Hi Magnus,

This is my sample log :

21674|glsServer.ec|ServerInit|351|2016-10-23 11:08:02|LOG|lCnt=0, service_name=ApplPref
21674|glsServer.ec|ServerInit|351|2016-10-23 11:08:02|LOG|lCnt=1, service_name=ApplList

And am trying this is filter:
filter {
mutate {
gsub => [
#Replace pipeline
"message","|"," "
]
}
grok {
match => { "message" => "%{NUMBER:col1} %{WORD:col2} %{WORD:col3} %{NUMBER:col4} %{WORD:col5} %{WORD:col6} %{WORD:col7}" }
}

Is there any direct way to make it accept the pipeline?


#4

you dont have to change do pipe(|).

you can write the pipe in the grok pattern like this: |

so you would have something like %{NUMBER:col1}|%{WORD:col2}


(Magnus Bäck) #5

Indeed, escape the | or just use the csv filter instead of the grok filter.


(Hari Baskar) #6

Hi,

Thanks much. It works. Now that I have followed the instructions and avoided the errors, I am still not able to see the logs in Kibana UI. I do not get any errors when pushing the Logs in. Any idea what could have went wrong? I tried making changes in the Log file once logstashed in. No changes in the window however.

Thanks in advance,
Hari


(Magnus Bäck) #7

What does your configuration look like? Exactly how did you change the file?


(Hari Baskar) #8

Hi there,

Sorry for my delay response. Was little busy in catching up other things. The things worked perfectly. Thanks for all your inputs. Although I am with a new query here.

I want to use URL search using current timestamp. To be precise, I want to take logs that have fallen in the last 15 mins of time.

Am looking for something like:

http://localhost:9200/_search?q="sql" AND @timestamp:"2016-11-16T17:33:19"

Can someone help what should i use in the @timestamp parameter to get the specified result


(Magnus Bäck) #9

Please post Elasticsearch questions in the Elasticsearch category.


(Hari Baskar) #10

Sure thanks


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.