bj and rails 2.1 - can't get bj to run jobs

[dusty]

I have been trying to setup bj with rails 2.1.0 and am having some
problems, was wondering if anyone could help. Here is what I've done
so far.

1. Basic rails install
$ rails tester

2. Install bj
$ cd tester
$ ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj

3. Setup bj
$ ./script/bj setup

4. Submit a job to the queue
$ ./script/console
$ Bj.submit 'ls'

At that point, it puts the job in the database
=> [#<Bj::Table::Job bj_job_id: 1, command: "ls", state: "pending",
priority: 0, tag: "", is_restartable: 1, submitter: "dustymini.local",
runner: nil, pid: nil, submitted_at: "2008-07-22 01:15:46",
started_at: nil, finished_at: nil, env: nil, stdin: nil, stdout: nil,
stderr: nil, exit_status: nil>]

The config setting shows this:
=> [#<Bj::Table::Config bj_config_id: 1, hostname: "dustymini.local",
key: "development.23928.pid", value: "23934", cast: "to_i">]

And, it seems to fire off the bj process to run that job

$ ps axwww | grep -i bj
23934 s000 R+ 0:01.99 /opt/local/bin/ruby /Users/dusty/tester/
script/bj run --forever --redirect=/Users/dusty/tester/log/
bj.dustymini.local.development.log --ppid=23928 --
rails_env=development --rails_root=/Users/dusty/tester

But, the job is never run, and I don't have any errors. Here is some
snippets from my logs

$ cat log/bj.dustymini.local.development.log
I, [2008-07-21T21:15:48.828069 #23934] INFO -- : STARTED

$ cat log/development.log
Bj::Table::Job Create (0.000862) INSERT INTO
"bj_job" ("is_restartable", "pid", "stdout", "stdin", "command",
"submitter", "finished_at", "env", "stderr", "submitted_at",
"priority", "tag", "runner", "started_at", "exit_status", "state")
VALUES(1, NULL, NULL, NULL, 'ls', 'dustymini.local', NULL, NULL, NULL,
'2008-07-22 01:22:37', 0, '', NULL, NULL, NULL, 'pending')
Bj::Table::Config Load (0.000479) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.no_tickle' AND
"bj_config"."hostname" = 'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.000446) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.no_tickle' AND
"bj_config"."hostname" = '*') LIMIT 1
Bj::Table::Config Load (0.001018) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.23928.pid' AND
"bj_config"."hostname" = 'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.001485) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = '23934.uri' AND "bj_config"."hostname" =
'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.000288) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = '23934.uri' AND "bj_config"."hostname" = '*')
LIMIT 1


Any ideas on how I can get this to run my job queue?

Thanks

 

[dusty]

BTW - I just resubmitted a new job and then ran all the queries that
showed up in the logs manually to see what it was getting back here is
that output.

From development.log

Bj::Table::Job Create (0.001226) INSERT INTO
"bj_job" ("is_restartable", "pid", "stdout", "stdin", "command",
"submitter", "finished_at", "env", "stderr", "submitted_at",
"priority", "tag", "runner", "started_at", "exit_status", "state")
VALUES(1, NULL, NULL, NULL, 'ls', 'dustymini.local', NULL, NULL, NULL,
'2008-07-22 01:30:09', 0, '', NULL, NULL, NULL, 'pending')
Bj::Table::Config Load (0.000502) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.no_tickle' AND
"bj_config"."hostname" = 'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.000677) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.no_tickle' AND
"bj_config"."hostname" = '*') LIMIT 1
Bj::Table::Config Load (0.000361) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.23996.pid' AND
"bj_config"."hostname" = 'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.000267) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.23996.pid' AND
"bj_config"."hostname" = '*') LIMIT 1
Bj::Table::Config Load (0.000276) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.0.pid' AND "bj_config"."hostname" =
'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.000247) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = 'development.0.pid' AND "bj_config"."hostname" =
'*') LIMIT 1
Bj::Table::Config Load (0.000254) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = '.uri' AND "bj_config"."hostname" =
'dustymini.local') LIMIT 1
Bj::Table::Config Load (0.000244) SELECT * FROM "bj_config" WHERE
("bj_config"."key" = '.uri' AND "bj_config"."hostname" = '*') LIMIT 1


Running all those commands manually, one returned an answer (between
the **)

sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" =
'development.no_tickle' AND "bj_config"."hostname" =
'dustymini.local') LIMIT 1;
sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" =
'development.no_tickle' AND "bj_config"."hostname" = '*') LIMIT 1;

**
sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" =
'development.23996.pid' AND "bj_config"."hostname" =
'dustymini.local') LIMIT 1;
1|dustymini.local|development.23996.pid|24011|to_i
**

sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" =
'development.23996.pid' AND "bj_config"."hostname" = '*') LIMIT 1;
sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" =
'development.0.pid' AND "bj_config"."hostname" = 'dustymini.local')
LIMIT 1;
sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" =
'development.0.pid' AND "bj_config"."hostname" = '*') LIMIT 1;
sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" = '.uri'
AND "bj_config"."hostname" = 'dustymini.local') LIMIT 1;
sqlite> SELECT * FROM "bj_config" WHERE ("bj_config"."key" = '.uri'
AND "bj_config"."hostname" = '*') LIMIT 1;

 

[dusty]

An update. I just ran the same setup as I had above, but ran it on
ubuntu and it worked fine. I was running on leopard previously. Any
ideas? Has anyone run bj on leopard?

 

[ara.t.howard]

An update. I just ran the same setup as I had above, but ran it on
ubuntu and it worked fine. I was running on leopard previously. Any
ideas? Has anyone run bj on leopard?

i developed it on leopard ;-)

frankly i didn't see anything wrong with your setup. bj has a bug
with timezones where it's unaware of the TZ configured in rails for AR
- is the TZ for your app the same as for your machine?

a @ http://codeforpeople.com/

 

[dusty]

It does appear to be time zone related.

My ec2 ubuntu instance has the same time zone for the machine and
rails
root@domU-12-31-39-00-7D-E5:~/tester# ./script/console
Loading development environment (Rails 2.1.0)=> Tue Jul 22 12:12:19 +0000 2008

My machine has EST local time, but Time.zone was set to UTC by
default.
[dusty@dustylaptop:~/tester] $ ./script/console
Loading development environment (Rails 2.1.0)=> Tue Jul 22 08:12:42 -0400 2008

However, changing rails to EST, doesn't seem to help. I'd guess its
because rails stores it as UTC in the db.
=> #<TimeZone:0x11d3a5c @tzinfo=#<TZInfo:ataTimezone: America/
New_York>, @name="Eastern Time (US & Canada)", @utc_offset=-18000>=> #<Bj::Table::Job bj_job_id: 4, command: "ls", state: "pending",
priority: 0, tag: "", is_restartable: 1, submitter:
"dustylaptop.local", runner: nil, pid: nil, submitted_at: "2008-07-22
12:17:16", started_at: nil, finished_at: nil, env: nil, stdin: nil,
stdout: nil, stderr: nil, exit_status: nil>
Setting my system clock to UTC, fixed it
=> #<Bj::Table::Job bj_job_id: 4, command: "ls", state: "finished",
priority: 0, tag: "", is_restartable: 1, submitter:
"dustylaptop.local", runner: "dustylaptop.local", pid: 2453,
submitted_at: "2008-07-22 12:17:16", started_at: "2008-07-22
12:22:53", finished_at: "2008-07-22 12:22:53", env: nil, stdin: nil,
stdout: "README\nRakefile\napp\nconfig\ndb\ndoc\nlib\nlog\npublic
\nsc...", stderr: "", exit_status: 0>
=> Tue Jul 22 12:24:28 +0000 2008

I would prefer to be able to keep my laptop at my local timezone,
though. Is there a workaround yet? I could also try to come up with
something. Perhaps a hint at where in the code to look around at, and
I'll see if I can come up with something.

Thanks for the response!

 

[dusty]

One more thing. I also have problems submitting via command-line.

[dusty@dustylaptop:~/tester] $ ./script/bj submit cat /etc/password
F, [2008-07-22T12:38:34.993286 #2555] FATAL -- : can't convert nil
into String (TypeError)
../script/bj:434:in `open'
../script/bj:434:in `run'
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:25:in `call'
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:25:in `run'
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:13:in `catch'
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:13:in `run'
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:436:in
`mode_run!'
/Users/dusty/tester/vendor/plugins/bj/lib/main/base.rb:10:in `run'
/Users/dusty/tester/vendor/plugins/bj/lib/main/factories.rb:11:in
`run'
/Users/dusty/tester/vendor/plugins/bj/lib/main/factories.rb:16:in
`Main'
../script/bj:6


Looks like that's failing on this line:

open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }

Any ideas on that one?

Also, is there a better place for this? I see there is a bug on this
in rubyforge, but its pretty old. Should I still submit things like
this to rubyforge instead of this mailing list?

Thanks



-------

mode "submit" do
keyword("file"){
argument :required
attr
}

def run
joblist = Bj.joblist.for argv.join(' ')

case file
when "-"
joblist.push(Bj.joblist.jobs_from_io(STDIN))
when "--", "---"
joblist.push(Bj.joblist.jobs_from_yaml(STDIN))
else
open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }
end

jobs = Bj.submit joblist, :no_tickle => true

oh = lambda{|job| OrderedHash["id", job.id, "command",
job.command]}

y jobs.map{|job| oh[job]}
end
end

 

[dusty]

Answering my question.

http://rubyforge.org/tracker/index.php?func=detail&aid=19765&group_id=1024&atid=4025

They recommended to change all instances of Time.now to Time.now.utc
in

lib/bj/runner.rb
lib/bj/table.rb
lib/systemu.rb

It appears to be working.

 

[ara.t.howard]

Answering my question.

http://rubyforge.org/tracker/index.php?func=detail&aid=19765&group_id=1024&atid=4025

They recommended to change all instances of Time.now to Time.now.utc
in

lib/bj/runner.rb
lib/bj/table.rb
lib/systemu.rb

It appears to be working.

this plus fixes to the commad line interface (which was demo only btw)
will be in svn soon. thanks for the bug reports.

a @ http://codeforpeople.com/

 

[Gustin Prudner]

I am attempting to get the env hash from within the worker process.

Seems like it should be meshed into ENV but the documentation is
unclear.

I accomplished it this way. It works because we ensure only one job with
a certain tag is run at a time but it isn't ideal.

job = Bj.table.job.findfirst, :conditions => { :tag =>
'product_publisher',
:state => 'running'
})
env = YAML.load job.env


Is there a better way??


~)o
gustin

entryway software development
http://www.entryway.net

 

[ara.t.howard]

I am attempting to get the env hash from within the worker process.

Seems like it should be meshed into ENV but the documentation is
unclear.

did you simply try it?

you should be able to access it via ENV as systemu merges that in for
you

var = ENV['var']

should be all you need, assuming you specified that var in the call
the bj submit...

I accomplished it this way. It works because we ensure only one job
with
a certain tag is run at a time but it isn't ideal.

job = Bj.table.job.findfirst, :conditions => { :tag =>
'product_publisher',
:state => 'running'
})
env = YAML.load job.env


Is there a better way??

use ENV

~)o
gustin

entryway software development
http://www.entryway.net

a @ http://codeforpeople.com/

 

[Denis Mahony]

you should be able to access it via ENV as systemu merges that in for
you

var = ENV['var']

should be all you need, assuming you specified that var in the call
the bj submit...

use ENV

I've been struggling to get this to function.. Any help would be
appreciated

While i have successfully created, by using bj, a text only PDF using
prawn/prawnto, i cannot successfully pass variables (ie an account ref)
through with :env , despite the earlier advice in this thread. Is it
possible, that in some circumstances, the full set of ENV would not be
passed to the bj background task ?

#-------------------------------------------------------
def bj_simple_prawn
Bj.submit "./script/runner ./script/simple_prawn_script.rb", :env => {
:avar => @current_user.account_id, :bvar => 27 }
end

After submitting the job..

bj_job has the following in the :env field;

---
:bvar: 27
:avar: 1

I cannot seem to access :env field/variable from bj_job table in the
worker task

# This line fails to extract the value passed in :env

@this_user = ENV['myvar']

The following works ! - if use a fixed value, 1.
@this_user = 1

@myaccount = Account.findfirst, :conditions => ["accounts.id=?",
@this_user])


Question : How does one access the values passed in the job :env ? I
suspect a syntax error on my part , but cannot pin it down
Like i say ... any help /pointer would be appreciated..

Denis

 

[denis mahony]

Update on my previous message/query...

I should clarify... The :env parameters are being passed in the ENV
and I can see
them being displayed in STDOUT in bj_job. The following line
generatess the STDOUT data

@all_of_ENV = ENV.each {|key, value| puts "#{key} is #{value}" }

It effectively outputs the complete ENV as [Key, Value] pairs in
STDOUT.
However the :env parameters I pass with the B.submit command seem to
be malformed and
not amenable to extraction from ENV.
They appears as follows as the 2nd last key/value output by the puts
function above ;

---
is
:avar: 1
is
:bvar: 42
is

followed by ;

RAILS_ENV is development

You can see the Key, Value formatting is different.
I would have expected them each to be on a single line ie ;

:avar is 1
:bvar is 2

Are the additional CR/LF elements intruding ?

Again , thanks for any help/pointers.
Regards

Denis

 

[denis mahony]

* Update II *

I abandoned the attampt to use the Bj.submit :env switch to pass
values to the background task in bj.
I just could not figure how to extract/use the values passed as YAML
(?) in ENV to the task.
I wonder if anybody has successfuly used it ? Is it broken ?

I'm now using the command line parameters, in the foreground process .
viz,

Bj.submit "./script/runner ./script/simple_prawn_script.rb #
{@this_user}"

I can now access the passed value in the worker using this
expression:

@this_user = ARGV[0].

Bj rocks

Regards

Denis