File in S3 has additional timestamp and host name as prefix to each message processed

[kaushik.vankayala]

Hi There,

I am having S3 as one of the output plugins in my logstash conf. Belos is my s3 plugin configuration details;

s3
{
	id => "uat-s3-backup"
	bucket => "mt-uat-project"
	access_key_id => "BK7U8I90DGFN236BQ62A"
	secret_access_key => "uwvzbkNMSBFNSBDFKJDFKJSAHEvOX14rkZRBl5W"
	region => "ap-south-1"
	prefix => "%{+YYYY}/%{+MMM}/%{+dd}"
}

The sample content of the s3 text file is;

2019-03-14T00:11:23.679Z ZE34P2907 [2019-03-14 05:41:23.679] INFO    Audit-IN [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"APITransactionId": "2bcdfbad9abc4862ba63472178c28fdd","CorrelationId": "cd72b9a0-461b-11e9-b9b4-02ee19ab148c","Request": "GET /api/experience/member-profile/v1/fetch-profile","TimeStamp": "2019-03-14 11:11:23.679","ApplicationName": "experience-app","Environment": "Production","SourceIP": "13.229.125.192:32238","PartnerID": "Partner","PartnerUniqueID": "1100f4932232481c85910debecc71e9b"} 
2019-03-14T00:00:28.900Z ZE34P2907 [2019-03-14 05:30:28.900] INFO    Audit-IN [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"APITransactionId": "41621fd54a2c4c6081b999313f235e94","CorrelationId": "47296520-461a-11e9-b9b4-02ee19ab148c","Request": "GET /api/experience/member-profile/v1/fetch-profile","TimeStamp": "2019-03-14 11:00:28.900","ApplicationName": "experience-app","Environment": "Production","SourceIP": "13.229.125.192:31438","PartnerID": "Partner","PartnerUniqueID": "1100f4932232481c85910debecc71e9b"} 
2019-03-14T00:00:29.178Z ZE34P2907 [2019-03-14 05:30:29.178] INFO    Audit-OUT [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"APITransactionId": "41621fd54a2c4c6081b999313f235e94","StatusCode": "200","TimeStamp": "2019-03-14 11:00:29.178","PartnerID": "Partner","PartnerUniqueID": "1100f4932232481c85910debecc71e9b","Request": "/api/experience/member-profile/v1/fetch-profile","ServiceType": "Service", "ResponseTime": "327 ms","ResponseDescription": ""}
2019-03-14T00:04:34.790Z ZE34P2907 [2019-03-14 05:34:34.790] INFO    Health-Check [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"http.status": 200, "Server Status": "Online", "Environment": "Production"}
2019-03-14T00:04:36.942Z ZE34P2907 [2019-03-14 05:34:36.942] INFO    Health-Check [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"http.status": 200, "Server Status": "Online", "Environment": "Production"}
2019-03-14T00:04:37.045Z ZE34P2907 [2019-03-14 05:34:37.045] INFO    Health-Check [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"http.status": 200, "Server Status": "Online", "Environment": "Production"}
2019-03-14T00:04:37.101Z ZE34P2907 [2019-03-14 05:34:37.101] INFO    Health-Check [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"http.status": 200, "Server Status": "Online", "Environment": "Production"}
2019-03-14T00:04:37.155Z ZE34P2907 [2019-03-14 05:34:37.155] INFO    Health-Check [[project].HTTP_Experience_Listener_Configuration.worker.142]: {"http.status": 200, "Server Status": "Online", "Environment": "Production"}

The first two columns are not part of the original message and they are timestamps and the host name.

Does anyone know how to not print them in the s3 files?

Regards

Kaushik

[Badger]

The default codec for s3 is line. The default format for a line codec prepends message with @timestamp and host. Set the codec format explicitly in your output.

codec => line { format => "%{message}" }