Remove the 'syslog' input

Description

There are too many incompatible syslog message implementations and additionally too many incompatible syslog protocols, and quite frequently users report syslog problems in logstash which are simply protocol-violations from the applications writing to syslog.

In order to make user support more scalable, I propose dropping the syslog input in favor of documentation that helps users learn how to use tcp input, grok filter, and date filter to achieve roughly the same thing.

Activity

Show:
Jordan Sissel
December 17, 2012, 5:22 AM

The shortest response I can provide is that syslog isn't a thing, it's a fiction.

The long answer is follows -

The problem is RFC3164 is an "informational" RFC, it's not a specification. As a result, there's literally nothing to comply with.
RFC5424 is an actual spec, but is incompatible with practically every syslog emitter in the wild (I've never actually seen it in my life). It's also poorly written and spread across multiple RFCs to add confusion.

I tried to support RFC3164, but it's pretty hit or miss, and as a result it fails in many common cases. These failures create confused users or generally waste time for folks trying to get things done. The domino effect of this leads people seeking support to the irc and mailling list only to find out what their vendor calls "syslog" is different than what I implemented as "syslog" - it sucks for everyone.

Instead, I'd like to provide tools and guidance for solving logs the way anyone else would if the input was a file - eyeball the format, build a filter set that processes it. If there's common formats, write a cookbook entry that describes it and the solution. Hell, if you look at the syslog input code I wrote, it's literally a tcp/udp input plus a grok and date filter inside the input - that stuff belongs in a cookbook recipe, not a whole plugin.

I am open to persuasion, but it's going to be a pretty tough sell to keep the syslog input given it only works in about 20% of the "syslog" world and that logstash itself has better tools (tcp/udp inputs, filters, etc) to handle it.

Rodrigo Menezes
March 29, 2013, 8:57 PM

Should there be a warning on the documentation page ( http://logstash.net/docs/1.1.9/inputs/syslog ) to further discourage new users from using the 'syslog' input plugin?

Tim Bunce
April 22, 2013, 10:43 AM

I'd like to see the inputs/syslog page in the docs kept and used to provide background info and ready-to-use recipies for common syslogs. That way logstash will still list "syslog" as an input and people will find the docs naturally.

Ideally most people could pick a recipe from the syslog page, based on their O/S, and copy-n-paste it.

Jordan Sissel
April 22, 2013, 7:19 PM

Yeah, I've actually been thinking about ways to some how present very common log patterns as "inputs" where there's no plugin but instead shows you the configurations necessary to process it.

I'm not sure how best to really present that information though; it feels like the logstash cookbook site (in idea, shared+tested recipes) is the right approach for where the syslog support will be moved to.

Tim Bunce
April 22, 2013, 8:44 PM

The logstash cookbook site is a great place for extended commentary on the issues, fine tuning, feedback from users etc.

When there are simple recipes for common needs, however, I think logstash and its users benefit from having them in the docs, (plus a for more information link to the relevant cookbook section if appropriate).

Or look at it this way: to go to logstash.net and not find a simple recipe for a syslog input would seem kind'a crazy!

Assignee

Logstash Developers

Reporter

Jordan Sissel

Labels

None

Fix versions

Affects versions

Configure