We're updating the issue view to help you get more done. 

Challenges in SMAPI logs Parsing

Description

I have a log file like this,

1 2 3 4 5 6 7 8 9 10 11 log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext). log4j:WARN Please initialize the log4j system properly. timestamp=2014/05/26 02:43:33.367;dimeName=KeyEvent;recordId=DrsSESTermsERFE;hostName=XXX-7610586;os=Windows 7; timestamp=2014/05/26 02:43:33.383;dimeName=KeyEvent;recordId=DrsSESTermsClassifier;hostName=XXX-76186;os=Windows 7; timestamp=2014/05/26 02:43:33.491;dimeName=KeyEvent;recordId=DrsSESTerms;hostName=XXX-7610586;os=Windows 7; com.lexisnexis.csa.enrichment.exception.RequestValidationException at com.lexisnexis.csa.enrichment.validator.impl.RequestValidatorImpl.validateOptionParam(RequestValidatorImpl.java:181) at com.lexisnexis.csa.enrichment.validator.impl.RequestValidatorImpl.validateRequest(RequestValidatorImpl.java:90) at java.lang.Thread.run(Thread.java:744) timestamp=2014/05/26 02:43:34.793;dimeName=ErrorEvent;recordId=RequestValidationFailed;hostName=XXX-7610586;os=Windows 7; timestamp=2014/05/26 02:43:34.793;dimeName=InternalRecord;recordId=RequestValidationFailed;hostName=X-7610586;os=Windows 7;

Here, I am concerned about only the lines starting with timestamp.

1. The lines with recordId DrsSESTermsERFE, DrsSESClassifier & DrsSESTerms constitute one record in which both DrsSESTermsERFE, DrsSESClassifier are optional.
2. If dimeName=ErrorEvent, it should be individual record.
3. If dimeName=InternalRecord, just drop it.
4. I don't want to parse other log lines in the log file.
Patterns:

1 SMAPI_ERFE_EVENT timestamp=%{SMAPI_TIMESTAMP};dimeName=%{WORD:dime};resource=%{WORD:resource};class=%{DATA};smLog=%{WORD:log};recordId=DrsSESTermsERFE;hostName=%{HOSTNAME};os=%{DATA};

I have written the config,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 filter { grok { match => { message => [ "^\s", "^com" ] } add_tag => [ "javalog" ] } if "javalog" in [tags] { drop{} } grok { patterns_dir => "C:/nat/Install/elasticSearch/logstash-1.4.1/bin/patterns" match => { message => [ "%{SMAPI_ERFE_EVENT}", "%{SMAPI_CLASSIFY_EVENT}", "%{SMAPI_ENRMNT_EVENT}", "%{SMAPI_ERROR_EVENT}", "%{SMAPI_INT_EVENT}" ] } } if "IntervalRecord" in [dime] { drop { } } multiline { stream_identity => "%{transId}" pattern => "(recordId=DrsSESTermsERFE|recordId=DrsSESTermsClassifier|dimeName=ErrorEvent)" what => "next" } mutate { remove_field => [ "tags" ] } }

The problem here is,
1. ErrorEvent is not coming as individual record. It is merging with KeyEvent record.
2. Parsing loglines other than smapi lines also.

I am new to logstash and I don't know Ruby. Where can I get materials to learn the logstash configuration.

Environment

None

Status

Assignee

Logstash Developers

Reporter

Nataraj Ramalingam

Affects versions

1.4.0

Priority