FPGA to solve the two most annoying problems on usenet - SuggestionsWelcome

[Charles Xavier]

As you all know, downloading files from usenet leaves you with two
sets of files.. The rar files from what you're downloading and the
par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.

Suggestions / Comments / Complaints?

 

[Atilla Filiz]

Seems like a really nice idea. Please notify your progress.

 

[Jon Beniston]

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

There are chips that will do this already. Here's one:

http://www.aspex-semi.com/q/product_features.shtml

Cheers,
Jon

 

[Charles Xavier]

There are chips that will do this already. Here's one:

http://www.aspex-semi.com/q/product_features.shtml

Cheers,
Jon

Thanks for the reply. Aspects of this design can certainly be used and
incorporated into this design, but as you know.. Their product isn't
free/open source. As this is also a learning project, I think it would
be better to build it from the ground up (using some pre-existing open
source IP if possible).

 

[PFC]

So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Why USB ? Gigabit Ethernet would be much faster
Don't tell me a Virtex5 can't handle a measly 100 MB/s

And... is it a PCI express slot that I see there ?
That's even better !

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

I was under the impression that x264 was h264 in disguise. Perhaps
tweaking the file format would be enough ?

Also you could recode the thing in a much simpler codec like MPEG2 or
something, who cares if it takes 10x the size, just stream it in realtime


Stick a DVI port on your FPGA !

 

[Charles Xavier]

I have a hard time believing that those are the "two most annoying problems
on usenet".  I've been using Usenet since 1984, and I've never personally
encountered either problem.

I've never tried to download an 8GB file from Usenet, and don't have any
clue why anyone would want to do such a thing.  There are much better
ways to distributed 8GB files.  But if I *did* want to download an 8GB
file from Usenet, I can't imagine that waiting three hours for a
reconstruction of missing pieces would really bother me that much.
There's never been a single thing that I've needed from Usenet so
urgently that it couldn't have waited three more hours.

I watch a fair number of movies, but I've never had any problem with
x.264.  It sounds like you just need better player software.  That has
nothing to do with Usenet.

Since it doesn't appear that there is any real problem here, it also
doesn't appear that there is any need for an FPGA-based "solution".

Thank you for sharing. Let your next post be in contribution to the
project.

If it doesn't apply to you and you don't see the need, why even reply?

 

[Symon]

Thank you for sharing. Let your next post be in contribution to the
project.

If it doesn't apply to you and you don't see the need, why even reply?

Hi Charles,
Maybe to share the benefit of his experience to help you avoid wasting your
time on a solution without a problem?
Good luck, Syms.

 

[Charles Xavier]

Hi Charles,
Maybe to share the benefit of his experience to help you avoid wasting your
time on a solution without a problem?
Good luck, Syms.

Engineers can often be of a single track mind, so let me restate. This
is a problem that I am having. I could certainly get a faster
computer, or try to finesse the algorithms for the decoding, but I
want to learn about this specific algorithm implementation on an FPGA,
so I'm using the FPGA to solve this problem (that is a problem for
me).

To me, this is a practical solution and an excellent gateway for
myself and others who want to get into FPGA-based hardware
acceleration of software algorithms. You give me a better way that
solves a practical problem for _me_ and I'll take it.

This reminds of the idiots at In-N-Out Burger who created the drive-
through fast food restaurant (they claim). It doesn't affect the
geriatric who only wants to eat inside because they can't imagine
driving while eating. But apparently it turned out to be a good idea.

Responding to this type of feedback is the real waste of time.

 

[Andreas Ehliar]

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

I'm not sure what the problem with x.264 is, but lets just say that
it is far from trivial to write a H.264 decoder/encoder. The decoder
is especially tricky as you need to be able to parse all parts of the
standard whereas you can select only the parts you care about for the
encoder.

If you are an expert on video codecs and a newcomer to FPGAs, this
would probably be a quite ambitiouis project to start with. If you
are not an expert on video codecs, you will have to spend a lot of
time to understand H.264 before you can even get started on the FPGA
parts.

This is written from personal experience by the way, I have written
a grayscale H.264 decoder in C to get an understanding of how the
standard works. Lets just say that it took way more time than I wanted
to...

In summary: I don't recommend a H.264 codec as a first FPGA project.

The Reed-Solomon application seems like a much more reasonable project.
You will still have a lot of things to figure out like how you are going
to communicate with your FPGA board, but the problem itself is much more
limited.

/Andreas

 

[Aiken]

I have an idea simular or not to your mind..(but I still
thinking....hope you are now starting ..your project).
my project is that

FILE COMPRESSION (zip or other)
read files from SD card, then compress it, and write a new zipped file
back to SD card.

it will not include the PC to do it.

 

[Mike Treseler]

Responding to this type of feedback is the real waste of time.

"Suggestions / Comments / Complaints?"

 

[Arash Partow]

Consider using rateless codes, reed-solomon and ldpc based codes do
provide erasure correction capabilities, however rateless codes such
those based on LT class of codes (tornado etc) ,or more generally
digital fountain codes, will have better performance due to their near
linear decoding complexity as opposed to say reed-solomon erasure only
decoding complexity which is around O(n^2) for naive implementations.



Arash Partow
__________________________________________________
Be one who knows what they don't know,
Instead of being one who knows not what they don't know,
Thinking they know everything about all things.
http://www.partow.net