Logstash crashs and Google BigQuery feature upgrade
Description
Gliffy Diagrams
Activity
Show:
Tal Levy February 6, 2015 at 11:18 PM
Ticket has been migrated to Github for future discussion:
https://github.com/logstash-plugins/logstash-output-google_bigquery/issues/4
Dave Stahl August 4, 2014 at 2:22 PM
Hey!
after a while of researching i've found the root cause of the problem
in all google output plugins (well google_bigquery.rb and google_cloud_storage.rb) there's a statement of deleting a file in the initialize_deleter call of File.delete(filename)
I've changed that statement to File.delete(filename) if File.exist?(filename) , so now it would only send a delete call only if the file exist, and it fixed the issue
i would recommend adding that to the source code as i've tested it in several places and it seems to work
Dave Stahl August 4, 2014 at 7:19 AM
Hi
Any update on that?
The problem exist on 1.4.2 as well
Hi
1. Running Logstash i get the following error over and over again:
IOError: closed stream
flush at org/jruby/RubyIO.java:2199
size at org/jruby/RubyFile.java:1108
method_missing at /opt/logstash/lib/logstash/outputs/google_bigquery.rb:565
receive at /opt/logstash/lib/logstash/outputs/google_bigquery.rb:194
handle at /opt/logstash/lib/logstash/outputs/base.rb:86
worker_setup at /opt/logstash/lib/logstash/outputs/base.rb:78
I'm trying to understand what am i doing wrong
i've tested my configuration and everything seems more than fine
as sometimes it works and sometimes it just fails with unknown reason
2. Google BigQuery output plugin
I think its a good idea to add the threshold attribute for allowed errors within an bq upload job, you can see example in the bigquery portal
3. Unknown bug of file control within Google BigQuery plugin
within the plugin you have two keys you can set:
deleter_interval_secs - set the interval in sec for deleting the temp file from disk
uploader_interval_secs - set the interval in sec from data collection to uploading the data to bigquery
I've noted that despite that the file was uploaded and was done been written with data, once it's deleted using the deleter_interval_secs key, Logstash craches because it can't find the file