Is there a standard pattern for threaded access to a file?

[Brian Adkins]

I use the one, true ruby-talk, the (e-mail address removed) mailing list.

Ok, I guess some posts get dropped in the ruby-talk -> comp.lang.ruby
transition occasionally. Probably the other direction too I suppose.

 

[James Edward Gray II]

Ok, I guess some posts get dropped in the ruby-talk -> comp.lang.ruby
transition occasionally. Probably the other direction too I suppose.

Yes, I've written about this in the past:

http://blog.grayproductions.net/articles/what_is_the_ruby_talk_gateway

Here is the relevant header from the message you are discussing that
shows why it wasn't gated:

Content-Type: multipart/alternative; boundary="----
=_Part_28483_17627615.1192285743535"

James Edward Gray II

 

[7stud --]

i'm not, but in a real piece of code longer than 5 lines it would be


ah - 'join' should indeed be 'value' there. sorry.

basically one should use Thread.current.abort_on_exception, check the
return values, or be prepared that threads may fail and you might no
know about it (which is obviously ok sometimes)

a @ http://codeforpeople.com/

Ok, so let me get this straight:

First you post a poor example that is needlessly complex for a
beginner--and that won't even work in the op's situation.

Then, when someone points out some flaws in your code, you claim that
the proposed improvements are faulty and that your original code is
superior.

Finally, when someone pointedly asked how it's possible your original
code does the things you claim it does, you refer to some imaginary
example that you would have posted.

 

[Brian Adkins]

Yes, I've written about this in the past:

http://blog.grayproductions.net/articles/what_is_the_ruby_talk_gateway

Thanks for the info!

I wonder how many mailing list posters realize this. Do you have any
stats for the percentage of mailing list posts that don't make it to
comp.lang.ruby?

If it's common knowledge that one needs to post in text only, then I
don't mind letting the gateway act as a filter for those who can't
configure their mail client, but if the requirement is not widely
known, then I may be missing posts that I'd like to receive.

I personally much prefer usenet to mailing lists, so I'm reluctant to
switch to the mailing list for just this one group.

Also, do you have stats for the percent of messages originating from
usenet vs. from the mailing list?

 

[James Edward Gray II]

I just checked out your "What is the ruby-talk" gateway; I didn't
realize
that the gateway currently dropped multipart/alternative. That's a
shame.

To be totally clear, our gateway doesn't drop them. They are
forwarded to our Usenet host. Our host rejects them as invalid
Usenet posts.

Since I bear some responsibility for its evil popularity, I'll
volunteer to update that gateway code to extract the text-part out
of the multipart if you can send it to me...

http://blog.grayproductions.net/articles/hacking_the_gateway

http://blog.grayproductions.net/articles/mail_to_newsrb

http://blog.grayproductions.net/articles/news_to_mailrb

I have a rewrite in progress that uses TMail for message handling.
On of my goals for this was to correctly separate the text portions
of multipart/alternative. I've just been distracted with work
deadlines and other short term projects, so I haven't completed it yet.

I should point out, though, that (a) it's really not that hard
(text/plain is supposed to come first, so that even clients who
didn't understand MIME would display the right thing before
displaying the wrong thing)

I've seen some pretty crazy things in messages sent to Ruby Talk.
One of those is multipart/alternative with no text/plain component.
I don't think there's too much loss in not supporting such setups
though.

and that
(b) SpamAssassin doesn't actually assign any points for HTML e-mail
- or, more accurately, it assigns zero points.

My apologies. I thought for sure I had seen a reference to that
sometime in the past, but I've been unable to dig it up this
morning. I stand corrected.

James Edward Gray II

 

[James Edward Gray II]

Do you have any stats for the percentage of mailing list posts that
don't make it to comp.lang.ruby?

I just did a simple grep of the logs for a period of a little over
the last month. It looks like we average about eight rejected
messages a day (for an "HTML post" reason).

If it's common knowledge that one needs to post in text only, then I
don't mind letting the gateway act as a filter for those who can't
configure their mail client, but if the requirement is not widely
known, then I may be missing posts that I'd like to receive.

Well, I wrote a blog post about it and reference it whenever the
discussion comes up.

Also, do you have stats for the percent of messages originating from
usenet vs. from the mailing list?

Over the same time period, the gateway saw 1,071 posts from Usenet
and 5,126 from the mailing list.

James Edward Gray II

 

[ara.t.howard]

I'm pretty new to ruby and this is one of those areas where I can't
quite seem to turn my head inside out as the language requires

I have a log file that I want to process in parts, with multiple
threads
working from the same file, each one getting a line at a time and
doing
something with it.

I'd like something like

1. Open the file
2. Create 5 threads

Each thread should read a line of the file and process it, but no 2
threads should get the same line.

What I have looks like

open (ARGV.flags.log) do |logfile|
logfile.each do |line|

blah blah blah...

end
end

but that's inside out! How do I rubify this code?

Thanks,

Jon

*finally* i remembered where this has been abstracted. in my own
lib : alib ;-)


cfp:~ > cat a.rb
require 'alib' ### gem install alib

alib.util.threadify IO.readlines(__FILE__), n_threads=5 do |line,
lineno|

puts "#{ lineno }:#{ line }"

end


cfp:~ > ruby a.rb
0:require 'alib'
1:
2:alib.util.threadify IO.readlines(__FILE__), n_threads=5 do |line,
lineno|
3:
4: puts "#{ lineno }:#{ line }"
5:
6:end


this works for *any* enumerable thing you want to process with 'n'
backend threads.

the current (0.5.0) alib version will blow up if you give it an IO
object though, as it uses #size to calculate the return value. i'll
tweak it and release 0.5.1 today.

cheers.

a @ http://codeforpeople.com/

 

[Brian Adkins]

I just did a simple grep of the logs for a period of a little over
the last month. It looks like we average about eight rejected
messages a day (for an "HTML post" reason).


Over the same time period, the gateway saw 1,071 posts from Usenet
and 5,126 from the mailing list.

8 msgs/day * ~30 days / 5,126 is ~5%; that's a little higher than I
was hoping I'm also surprised by the mailing list : usenet ratio.

 

[James Edward Gray II]

8 msgs/day * ~30 days / 5,126 is ~5%; that's a little higher than I
was hoping

The log I used actually went back to 9-6-2007, so it was closer to 38
days, but yeah. It was higher than I would have guessed too.

James Edward Gray II

 

[James Edward Gray II]

Sounds like it'd be more useful for me to help on the TMail version
(since that's what I'd end up using anyway).. is that code posted
anywhere yet, or would you be willing to send/post it?

It's not yet online. I am happy to put it up, sure. I really need
to get through two projects before I get to that though. Please give
me a few weeks.

It's not in the Gateway topic, and your search box is, uh, broken.

It seems to work OK for me. Feel free to email me the details off-
list and I'll sure try to fix it.

Yeah, that's just totally broken. I mean, it's technically legal
MIME, but pointless.

We see quite a few broken posts pass through the gateway in quite a
few different ways. Welcome to the Internet.

James Edward Gray II

 

[Jon Handler]

Why are you doing this in the first place? Do you have a computer with
five
processors and five memory buses?

I'm stress-testing my http server... Since most of the time is spent
waiting for requests to go and come back, multiple threads on the
sending end allows for greater throughput (to a point anyway). The
number 5 was just an example of n where n > 1.

Jon

 

[Jon Handler]

Actually, the example provided won't even work in your case. You have
to do some extra things.


A Queue is a first in first out container, which means the items you
push() into one end of the Queue are the first items that pop() out the
other end. A Queue is also thread safe, which means that only one
thread can access it at the same time.

Slapping forehead... of course! Producer/Consumer = q.

Thanks!

 

[Jon Handler]

Wow!

Thanks everyone for the detailed and insightful help!

Cheers,

Jon