Unique in-depth problem

[Idgarad]

I have a real challenge:

I need to work with, manage, and render a timeline. The problem? The
dates range, in years, from 0 to 1452652!! So I have a few hurdles:

1: Standard date libraries don't work. These dates are vastly out of
epoch ranges.
2: I have to render events on a timeline. Requirements are open but it
has to be able to handle rendering the whole damn thing at once (i.e
zooming in is ok.) but effectivly I have to either use SVG or Flash to
do this. Resolution goes to the day so:
YEAR|YEAR| etc..
MONTH|MONTH|MONTH|MONTH| etc..
1|2|3|4|etc...
-----------
XXXXXXXXXXXXXX YYYYYYYYYY ZZZZZZZZZZZZZZZZZZZZ

I originally did this via tables with colspans etc. but complicated
entries with 800+ events in a month (I was only rendering a single
year) could take up to 3 hours to display on a browser. SVG or Flash is
gonna be needed.

Dear God! What to do!? So the questions are really this:

Q1: What options do I have in Perl for handling dates of this nature
Q2: What graphical format is going to be able to handle, assuming I
dedicated 10px per day (so each year would be at leasy 3650px wide),
handling something of that scale? (Lets assume a limited viewport is
acceptable so we don't have to display the whole thing at once...)

Think of this as a History of the World Gnatt chart of sorts. I mean
what options, if any, are there for this kind of problem? I can batch
the process (It does not have to be real-time thank god!) but must be
doable within an 16 hour window utilizing a standard desktop (p4, 2ghz
lets say). This would eventually end up on a server but the constraint
is can start a friday night and must be done before Sunday 11:30pm so I
am working with Worst Case Scenario here.

 

[xhoster]

I have a real challenge:

I need to work with, manage, and render a timeline. The problem? The
dates range, in years, from 0 to 1452652!! So I have a few hurdles:

1: Standard date libraries don't work. These dates are vastly out of
epoch ranges.
....

Q1: What options do I have in Perl for handling dates of this nature

Are the events points in time or do they have durations? Is the database
dynamic or fairly static? If it is static points, how about just a sorted
list you can binary search into?

Q2: What graphical format is going to be able to handle, assuming I
dedicated 10px per day (so each year would be at leasy 3650px wide),
handling something of that scale? (Lets assume a limited viewport is
acceptable so we don't have to display the whole thing at once...)

I have no idea. This doesn't seem like much of a Perl question. Maybe
you can look at some "genome browsers" (largely written in java, it seems)
to get some ideas for dynamic zooming/panning of very large linear feature
sets.

Xho

 

[Tad McClellan]

I need to work with, manage, and render a timeline. The problem? The
dates range, in years, from 0 to 1452652!!

I wonder how you determined the age of the universe to a resolution
of only 1 year.

 

[Ted Zlatanov]

I need to work with, manage, and render a timeline. The problem? The
dates range, in years, from 0 to 1452652!! So I have a few hurdles:

1: Standard date libraries don't work. These dates are vastly out of
epoch ranges.

2: I have to render events on a timeline. Requirements are open but it
has to be able to handle rendering the whole damn thing at once (i.e
zooming in is ok.) but effectivly I have to either use SVG or Flash to
do this. Resolution goes to the day so:
YEAR|YEAR| etc..
MONTH|MONTH|MONTH|MONTH| etc..
1|2|3|4|etc...
-----------
XXXXXXXXXXXXXX YYYYYYYYYY ZZZZZZZZZZZZZZZZZZZZ

I originally did this via tables with colspans etc. but complicated
entries with 800+ events in a month (I was only rendering a single
year) could take up to 3 hours to display on a browser. SVG or Flash is
gonna be needed.

Dear God! What to do!? So the questions are really this:

Q1: What options do I have in Perl for handling dates of this nature

You can easily write a class to handle these dates.

Precompute all the dates and put them in a database (file, SQL server,
etc.).

Look at Google Maps for inspiration. They deal with lots of pieces of
a huge data set, yet manage to retrieve and show only the interesting
ones quickly.

Q2: What graphical format is going to be able to handle, assuming I
dedicated 10px per day (so each year would be at leasy 3650px wide),
handling something of that scale? (Lets assume a limited viewport is
acceptable so we don't have to display the whole thing at once...)

PNG should work.

Think of this as a History of the World Gnatt chart of sorts. I mean
what options, if any, are there for this kind of problem? I can batch
the process (It does not have to be real-time thank god!) but must be
doable within an 16 hour window utilizing a standard desktop (p4, 2ghz
lets say). This would eventually end up on a server but the constraint
is can start a friday night and must be done before Sunday 11:30pm so I
am working with Worst Case Scenario here.

s/Gnatt/Gantt/

Just generate the dates that have changed since the last build.

When dealing with large data sets, three things are always important:

1) cache data everywhere possible

2) precompute things when possible

3) buy as much RAM as possible for the server

With Perl, these are just as valid as with Java or any other language.

Also consider using something on the storage backend like a SQL server
(MySQL, Postgres, etc.) that can do efficient queries across your data.

Ted

 

[A. Sinan Unur]

I wonder how you determined the age of the universe to a resolution
of only 1 year.

Actually, he claims to be able to distinguish events by day during that
entire duration as he wants the thing to zoom to any individual day during
that period. Quite impressive. I have a sneaking suspicion someone will
claim this year was the warmest in the past 1452652 years.

Sinan

 

[Ala Qumsieh]

I originally did this via tables with colspans etc. but complicated
entries with 800+ events in a month (I was only rendering a single
year) could take up to 3 hours to display on a browser. SVG or Flash is
gonna be needed.

Or javascript. Think of google maps.

--Ala

 

[Idgarad]

The back end is SQL, for the sake of sanity I broke up the dates into
separate fields for day, month, and year. Even SQL dies a horrible
grizzley death when querying say the duration of the roman empire per
the database entries. It is sheer horror so far. I checked PNG and
sadly PNG is not viable as any bitmap format results in terabyte sized
files. Vector is going to be my only option. They want to be able to
print out wall sized images so I not only have to be able to efficently
chop the image up but be able to handle absurd format sizes (they are
planning on using a HP Designjet 42 wide printer with as much as 300
feet (yes feet) on a roll and print it out. Sending a bitmap is
possible if I segment it but the printer only has a 30gb hard drive per
batch. Ugg... As far as the perl side of this goes it doesn't seem that
there is any modules that exist to handle something of this nature so
they're going to need to bring in someone to write an epoc-less date
module, that is outside of my capabilities. I appreciate the feedback
from everyone. Thank You All!

 

[Peter J. Holzer]

[quoting corrected for better readability - please don't top-post]

I need to work with, manage, and render a timeline. The problem? The
dates range, in years, from 0 to 1452652!! So I have a few hurdles:

1: Standard date libraries don't work. These dates are vastly out of
epoch ranges.

2: I have to render events on a timeline. Requirements are open but it
has to be able to handle rendering the whole damn thing at once (i.e
zooming in is ok.) but effectivly I have to either use SVG or Flash to
do this. Resolution goes to the day so: [...]
I originally did this via tables with colspans etc. but complicated
entries with 800+ events in a month (I was only rendering a single
year) could take up to 3 hours to display on a browser. SVG or Flash is
gonna be needed.

Dear God! What to do!? So the questions are really this:

Q1: What options do I have in Perl for handling dates of this nature

You can easily write a class to handle these dates.

Precompute all the dates and put them in a database (file, SQL server,
etc.).

Look at Google Maps for inspiration. They deal with lots of pieces of
a huge data set, yet manage to retrieve and show only the interesting
ones quickly.

Q2: What graphical format is going to be able to handle, assuming I
dedicated 10px per day (so each year would be at leasy 3650px wide),
handling something of that scale? (Lets assume a limited viewport is
acceptable so we don't have to display the whole thing at once...)

PNG should work.

I checked PNG and sadly PNG is not viable as any bitmap format results
in terabyte sized files. Vector is going to be my only option. They
want to be able to print out wall sized images so I not only have to
be able to efficently chop the image up but be able to handle absurd
format sizes (they are planning on using a HP Designjet 42 wide
printer with as much as 300 feet (yes feet) on a roll and print it
out.

How do you plan to use SVG or Flash on a printer? When you were talking
about colspan, SVG or Flash, everybody was probably assuming you were
talking about showing the timeline on a browser, not on hundreds of
square meters of paper.

Since a browser can't show more than a few thousand pixels in either
direction, you would generate only the small tile currently viewed - no
need to generate terabyte-sized png files.

For the printer, you should probably generate narrow strips and print
one after the other. Since I assume that the 1.5 million years will span
the 300 feet and not the 42 inches, you can proably just print one year
after the other.

Oh, BTW, your 1.5 million years are about 530 million days. If you allot
1 mm per day (which corresponds roughly to your 10 px/day at 300 dpi),
your chart will be 530 km long. If you want to print that in 16 hours,
your printer must print at least 33 km/h. I doubt a HP designjet can do
this, never mention that you need to load one of your 300 feet paper
rolls every 11 seconds.

batch. Ugg... As far as the perl side of this goes it doesn't seem that
there is any modules that exist to handle something of this nature so
they're going to need to bring in someone to write an epoc-less date
module, that is outside of my capabilities.

The date part seems actually pretty easy, unless you have to deal with
real calendars (like the missing days at the transition between the
Julian and Gregorian Calendar or the irregular leap days in the early
Julian era - not to mention calendars like Islamic calendar which cannot
be reliably computed because they depend on astronomical observations).
Perl uses double precision floating point numbers, so at 1.5 millions
years you can still represent fractions of seconds, and the mathematics
of the Gregorian calendar (for instance) aren't that complicated (except
for Easter).

Laying out a graph that large seems to be much more problematic to me.

hp