json filter reports jsonparsefailure on messages containing field named "message"

Description

I moved to using the JSON filter to process incoming tcp input messages that are in JSON format; these messages are in the new event data model, and contain a field named "message". When I use the filter as such:

and a message like this comes into the interface:

I get the following error emitted to stdout (using the stdout output plugin and verbose logging):

Trouble parsing json {:source=>"message", :raw=>"This is the second test.", :exception=>#<JSON:arserError: unexpected token at 'This is the second test.'>, :level=>:warn}

I also get the _jsonparsefailure tag added to the event.

Activity

Show:
Jordan Sissel
September 12, 2013, 8:56 PM

This isn't very well documented, sorry about that

Jason Levine
September 12, 2013, 11:23 PM
Edited

Jordan, I don't understand what you're saying in your first 4:56pm comment. What behavior is expected – the jsonparsefailure?

If that's the case, what do we do? What is the proper codec, "line"? If so, how does the json filter operate on it, given that it's already a JSON-formatted logstash event? (A lot of this is the same set of questions I'm asking over in LOGSTASH-1332.)

I guess I'm just frustrated – this all worked perfectly in 1.1.13, and it's more or less broken in 1.2.1 in a way that the documentation doesn't help with at all. I say this in the vein of the general "if the user is having issues and going nuts, it's a bug" ethic that I've been so impressed with around the logstash project – this user is having major issues, and is going nuts, and I'm finding the documentation isn't helping at all. (For example, the new 1.2 docs make it clear that all the format config parameters are deprecated, but nowhere does it explain why, or what the new mechanism is for achieving what they achieved. I only discovered that codecs were the new way of doing things via lurking in the IRC channel for a few hours.) I've dug into code, I've gone to the IRC channel, and I've done more or less everything I can do to figure out how to get my formerly-working system working again, and I've more or less concluded that the current 1.2 branch of logstash isn't workable. Unfortunately, I put a not-inconsiderable amount of effort into the systems that feed into logstash to migrate their events into the new model, etc., so it's nontrivial for me to go back... but I think I either have to go back, or I have to move to another (non-tcp) input and hope that I don't discover the same gotchas surrounding the move to codecs.

Cory Marsh
November 7, 2013, 12:35 AM

I was having the same problem with the TCP input. I switched to the REDIS input and I am having an even worse time of it. Where the TCP input would parse messages correctly that were not nagled (TCP nagel) the REDIS input throws "Trouble parsing json" on all messages.

Even the logstash book does not provide any information about this. It seems like everyone is just using this software to snarf files from disk, as soon as you attempt to do something different, you are off the reservation.

If anyone has any info on how to send JSON data into logstash that would be very helpful. This is the most frustrating software I have used in years and years.

I guess it's time to roll up the sleeves and fork the code.

Philippe Weber
April 18, 2014, 12:40 PM

tcp and json codec issues

Aaron Mildenstein
February 6, 2015, 10:06 PM

In the absence of new feedback, and several fixes later on both the json and json_lines codecs, I am closing this issue.

If you need to open another issue in the future, please do so at https://github.com/logstash-plugins/logstash-codec-json or https://github.com/logstash-plugins/logstash-codec-json_lines

Assignee

Logstash Developers

Reporter

Jason Levine

Labels

Affects versions

Configure