systemd / journald logshipper request
Description
Gliffy Diagrams
Activity
Duncan Innes June 10, 2014 at 8:35 AM
Reviewing the tickets this week, I can't figure whether it's more prudent to influence the freedesktop ticket listed above, or develop an independent interim solution.
An integrated end-solution would be better done via a journald function, but it's not clear if this is actually going anywhere. I will ask if there's any progress towards a flexible log shipper.
Duncan Innes April 17, 2014 at 7:50 PM
Having spoken at length to Red Hat Support, there appears to be little/no appetite to bring forward the development of systemd-journal-upload, despite the communication in the above ticket. I tried to push for an early inclusion into RHEL7, but got no traction. Hopefully the Logstash guys who went to Red Hat Summit made more headway.
Getting something out of journalctl and into logstash is going to be imperative to any large RHEL/logstash user. We are comfortable with the current RHEL6 setup using rsyslog to get logs off the client, but having an integrated system when we move to RHEL7 would be a huge boost. Fewer parts to stick together is always going to represent a win for large corporate customers with highly segregated VLAN's and many different log sources.
Happy to help, but my last C code was 20+ years ago, I haven't written a single line of Java, perl is rusty and python just above kindergarten.
Duncan Innes April 3, 2014 at 10:06 PM
For information:
journald bugzilla entry requesting JSON (or other) output formats, user definable extra fields and possibly client-side parsing of some log entries (IPTABLES being a good example of specific fields that can be populated)
Duncan Innes February 4, 2014 at 8:24 AM
Messing around with RHEL7 beta at the moment and a direct logshipper wouldn't just be handy - it would be a killer differentiating feature.
A direct logshipper for Journald would be quite handy, and seems like it should be possible.
Journald has two methods to pull logs out in json format:
http://www.freedesktop.org/software/systemd/man/journalctl.html
$ journalctl -o json |whatever-shipper
http://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html
$ curl -s -H 'Range: entries=s=a51c80356a1c48af95d5f1eb6...'-H 'Accept: application/json' 'http://127.0.0.1:19531/entries?follow'
Both allow you to use the cursor output to resume where you left off last. If you do not have a cursor, do not pass the param and it will start from the beginning of the log.
Many other options exist for filtering.
Sample output line in json format:
{ "_CURSOR" : "s=a51c80356a1c48af95d5f1eb655738e9;i=1;b=b6c78b0057b741b48b0562cda7d20536;m=cea36;t=4ec5c572f40f7;x=fa424faf865e6ec8", "REALTIME_TIMESTAMP" : "1385781250703607", "_MONOTONIC_TIMESTAMP" : "846390", "_BOOT_ID" : "b6c78b0057b741b48b0562cda7d20536", "PRIORITY" : "6", "_TRANSPORT" : "driver", "MESSAGE" : "Runtime journal is using 184.0K (max 49.4M, leaving 74.1M of free 494.0M, current limit 49.4M).", "MESSAGE_ID" : "ec387f577b844b8fa948f33cad9a75e6", "_PID" : "45", "_UID" : "0", "_GID" : "0", "_COMM" : "systemd-journal", "_EXE" : "/usr/lib/systemd/systemd-journald", "_CMDLINE" : "/usr/lib/systemd/systemd-journald", "_CAP_EFFECTIVE" : "4402800cf", "_SYSTEMD_CGROUP" : "/system.slice/systemd-journald.service", "_SYSTEMD_UNIT" : "systemd-journald.service", "_SYSTEMD_SLICE" : "system.slice", "_SELINUX_CONTEXT" : "kernel", "_MACHINE_ID" : "3842e8edc94b495eb12b70f55de51b05", "_HOSTNAME" : "sample.local.lan" }