Templating system needs help

[Tore Aursand]

Hi!

I need to develop a templating system. I _cannot_ use on of the
templating systems found on CPAN as this utility will be used on
"proprietary" (and already existing) template files. So - please don't
tell me to go to CPAN; I already know of it, and I know the templating
systems found there. Thanks.

The problem is lists. And lists in lists (in lists...). I've decided to
have the data structure for my lists like this:

$VAR1 = {
'__ROOT__' => {
'children' => [
{
'name' => 'WEEK_LIST',
'columns' => [ 'NR' ],
'values' => [
[ '1' ],
[ '2' ]
],
'children' => [
{
'name' => 'DAY_LIST',
'columns' => [ 'DAY' ],
'values' => [
[ '1' ],
[ '2' ],
[ '3' ],
[ '4' ],
[ '5' ],
[ '6' ],
[ '7' ],
]
},
{
'name' => 'DAY_LIST',
'columns' => [ 'DAY' ],
'values' => [
[ '8' ],
[ '9' ],
[ '10' ],
[ '11' ],
[ '12' ],
[ '13' ],
[ '14' ],
]
}
]
}
]
}
}

This is from Data:umper, though I had to reformat it a bit. I guess you
all get the point. The template for this data looks like this (shortened):

<%% WEEK_LIST BEGIN %%>
<%% NR %%>:<%% DAY_LIST BEGIN %%> <%% DAY %%><%% DAY_LIST END %%>
<%% WEEK_LIST END %%>

Thus - in the end - the following output should be created:

1: 1 2 3 4 5 6 7
2: 8 9 10 11 12 13 14

Now. I have absolutely _no_ idea on how to create this list (and take
height for the fact that there can be unlimited nested lists). Anyone
feel like they have the guts to try?

Thanks!

 

[Tore Aursand]

[...]
Now. I have absolutely _no_ idea on how to create this list (and take
height for the fact that there can be unlimited nested lists). Anyone
feel like they have the guts to try?

I could of course post some of the ideas I've been playing with. The
module is OO, and everything gets "wrapped up" when 'build_template' is
called;

sub build_template {
my $self = shift;

# Lists
$self->_generate_lists();

# Replacements
while ( my ($key, $value) = each %{$self->{'_replace'}} ) {
# ...
}

# Substitutions
while ( my ($key, $value) = each %{$self->{'_subst'}} ) {
# ...
}
}

sub _generate_lists {
my $self = shift;
my $node = shift || $self->{'_lists'}->{'__ROOT__'};
my $template = shift || $self->{'_template'};

# Now what?
}

 

[Gunnar Hjalmarsson]

I need to develop a templating system. I _cannot_ use on of the
templating systems found on CPAN as this utility will be used on
"proprietary" (and already existing) template files. So - please
don't tell me to go to CPAN; I already know of it, and I know the
templating systems found there. Thanks.

Err.. Go to CPAN.

Actually, I'm not sure I understand why you can't. Most modules there
are provided under the artistic license, which I thought does not
prevent you from using the modules with propriety code.

 

[Eric Bohlman]

Err.. Go to CPAN.

Actually, I'm not sure I understand why you can't. Most modules there
are provided under the artistic license, which I thought does not
prevent you from using the modules with propriety code.

I think he was trying to say that the template files already exist, and
were written in a novel template language. Unless I'm badly mistaken, each
of the several million (OK, I exaggerate) templating modules available on
CPAN has its own template language, and none of them will be compatible
with the one used by Tore's files.

 

[Tore Aursand]

I think he was trying to say that the template files already exist, and
were written in a novel template language.

That's correct. I can't use any of them - AFAIK. And - still AFAIK -
none of them handles lists (in lists) the way I'm looking for, so I can't
"borrow" some of the existing modules' code, either.

 

[Gunnar Hjalmarsson]

That's correct. I can't use any of them - AFAIK. And - still
AFAIK - none of them handles lists (in lists) the way I'm looking
for, so I can't "borrow" some of the existing modules' code,
either.

Okay. Your mentioning of the word "proprietary" gave the impression
that you preclude the use of CPAN for licensing reasons, and I just
wanted to say that that's not necessary - AFAIK. ;-)

 

[Uri Guttman]

TA> $VAR1 = {
TA> '__ROOT__' => {
TA> 'children' => [
TA> {
TA> 'name' => 'WEEK_LIST',
TA> 'columns' => [ 'NR' ],
TA> 'values' => [
TA> [ '1' ],
TA> [ '2' ]
TA> ],
TA> 'children' => [
TA> {
TA> 'name' => 'DAY_LIST',
TA> 'columns' => [ 'DAY' ],
TA> 'values' => [
TA> [ '1' ],
TA> [ '2' ],
TA> [ '3' ],
TA> [ '4' ],
TA> [ '5' ],
TA> [ '6' ],
TA> [ '7' ],
TA> ]
TA> },

TA> <%% WEEK_LIST BEGIN %%>
TA> <%% NR %%>:<%% DAY_LIST BEGIN %%> <%% DAY %%><%% DAY_LIST END %%>
TA> <%% WEEK_LIST END %%>

TA> Thus - in the end - the following output should be created:

TA> 1: 1 2 3 4 5 6 7
TA> 2: 8 9 10 11 12 13 14

TA> Now. I have absolutely _no_ idea on how to create this list (and take
TA> height for the fact that there can be unlimited nested lists). Anyone
TA> feel like they have the guts to try?

very recently i did something like this for a proof of concept. it is a
different way to look at templating but it fits your needs well. but the
code is at work so i will get it tomorrow. rather small and interesting
stuff as well.

the concept is that you preprocess the templates and extract out the
'chunks' that are delimited by BEGIN/END (and they have a name). those
chunks are then substituted elsewhere in the file where you have a
special tag with the same name. this would be inside some string (i like
here docs) and can be munged by code. so this template:

sub expand_rows {

code_to_mung( <<ROW ) ;
%ROW%
ROW

return \$munged_rows ;
}

print <<TEMPLATE ;
blah <%ROW BEGIN sub=expand_rows%>asdfghjkl<%ROW END%>
<<TEMPLATE

would get expanded to:

sub expand_rows {

code_to_mung( <<ROW ) ;
asdfghjkl
ROW

return \$munged_rows ;
}

print <<TEMPLATE ;
blah ${ expand_rows() }
<<TEMPLATE

this way the code has direct access to the chunk it wants and it can
return an expanded chunk back into the main text. but the template
separates the code from the full text.

and that can be made recursive with a little more effort so it can
handle nested data easily.

i always passed in hashes that were already stuffed with data and you
have that already. all you need to do is write the code to do the row
stuff and that can be generic to a point. the basic template
preprocessing is generic and fairly easy to write (the code i have is
less than a page). it is very powerful and yet simple, just the way i
like to design stuff.

if i ever get into it again, i would make a module for it. it is just a
short experiment so far but it does work. the hard part was figuring out
how to pass row text to a sub for runtime expansion and yet keep it in
the main text. the callback from a ${} was the key as well as moving the
chunk to the actual code.

simple values can be just replaced with an access to the tree like:

blah<%FOO%>blah

becomes

blah$tree->{FOO}blah

and that can be improved to handle deeper trees or similar stuff.

so is cpan ready for ANOTHER template system? this separates code from
text (which i like) and doesn't invent a new mini-lang (like template
toolkit does). i dunno if it can handle all the possible complex
template needs but scalars, nesting and loops are done with little
effort.

uri

 

[Randal L. Schwartz]

Tore> That's correct. I can't use any of them - AFAIK. And - still AFAIK -
Tore> none of them handles lists (in lists) the way I'm looking for, so I can't
Tore> "borrow" some of the existing modules' code, either.

Well, I bet you could do a source-to-source translator from your
magical templating language to Template Toolkit's language, or even
plugin in a different parser with relative ease.

There's still no point in starting from scratch.

print "Just another Perl hacker,"; # the first

 

[Tassilo v. Parseval]

Also sprach Tore Aursand:

I need to develop a templating system. I _cannot_ use on of the
templating systems found on CPAN as this utility will be used on
"proprietary" (and already existing) template files. So - please don't
tell me to go to CPAN; I already know of it, and I know the templating
systems found there. Thanks.

The problem is lists. And lists in lists (in lists...). I've decided to
have the data structure for my lists like this:

$VAR1 = {
'__ROOT__' => {
'children' => [
{
'name' => 'WEEK_LIST',
'columns' => [ 'NR' ],
'values' => [
[ '1' ],
[ '2' ]
],
'children' => [
{
'name' => 'DAY_LIST',
'columns' => [ 'DAY' ],
'values' => [
[ '1' ],
[ '2' ],
[ '3' ],
[ '4' ],
[ '5' ],
[ '6' ],
[ '7' ],
]
},
{
'name' => 'DAY_LIST',
'columns' => [ 'DAY' ],
'values' => [
[ '8' ],
[ '9' ],
[ '10' ],
[ '11' ],
[ '12' ],
[ '13' ],
[ '14' ],
]
}
]
}
]
}
}

This is from Data:umper, though I had to reformat it a bit. I guess you
all get the point. The template for this data looks like this (shortened):

<%% WEEK_LIST BEGIN %%>
<%% NR %%>:<%% DAY_LIST BEGIN %%> <%% DAY %%><%% DAY_LIST END %%>
<%% WEEK_LIST END %%>

Thus - in the end - the following output should be created:

1: 1 2 3 4 5 6 7
2: 8 9 10 11 12 13 14

Now. I have absolutely _no_ idea on how to create this list (and take
height for the fact that there can be unlimited nested lists). Anyone
feel like they have the guts to try?

You are best off using a proper grammar for this kind of mini-language
where things can be nested arbitrarily deeply. Depending on how complex
the templating language is you can write a custom recursive descent
parser for it. That naturally requires that your grammar would be LL(1)
which forbids left-recursiveness and common prefixes such as

rule -> pref rule1 | pref rule2

Both can be eliminated through some transformations (the latter is
solved with left factorization). You'll also need a tokenizer that
returns the next token your parser is supposed to look-ahead at.

Or you just use Parse::RecDescent to create the parser for you. This
means getting acquainted with a complex-ish module but it would spare
you some boring work such as calculating the predict sets yourself which
a recursive descent parser usually needs.

Tassilo

 

[Tore Aursand]

[...]

very recently i did something like this for a proof of concept. it is a
different way to look at templating but it fits your needs well. but the
code is at work so i will get it tomorrow. rather small and interesting
stuff as well.

I'm looking forward to have a look at it!

the concept is that you preprocess the templates and extract out the
'chunks' that are delimited by BEGIN/END (and they have a name).

Hmm. I really don't think I can do that, if I understand you correctly.
There are ways to dynamically create lists, and that is done by using a
"template tag" which is - uhm - interpolated (?) into the "real template
tag".

Don't know how I can explain this any better, but take a look at this
example template file:

This <%% WORD_1 %%> a [%% TPL_1 %%]

In my script I do something like this:

my $Template = Template->new( ... );
$Template->subst( 'WORD_1', 'is' );
$Template->subst( 'TPL_1', 'WORD_2' );
print $Template->get_template();

The output will be:

This is a <%% WORD_2 %%>

As you can see, the []-tags have been converted to <>-tags by now, which
is ideal to let the templating system create/generate templates.

I have been trying out some code since my original post, and I will post
that code very soon (when it works a bit better than it does now). I feel
that I'm getting quite close to dealing with that list thing.

 

[Uri Guttman]

[...]

the concept is that you preprocess the templates and extract out the
'chunks' that are delimited by BEGIN/END (and they have a name).

TA> Hmm. I really don't think I can do that, if I understand you correctly.
TA> There are ways to dynamically create lists, and that is done by using a
TA> "template tag" which is - uhm - interpolated (?) into the "real template
TA> tag".

it handles single tags as well. the hard part is chunks and lists. my
design assumes the data is already for use by the template so it can
handle any dynamic lists/hashes you want.

TA> Don't know how I can explain this any better, but take a look at this
TA> example template file:

TA> This <%% WORD_1 %%> a [%% TPL_1 %%]

TA> In my script I do something like this:

TA> my $Template = Template->new( ... );
TA> $Template->subst( 'WORD_1', 'is' );
TA> $Template->subst( 'TPL_1', 'WORD_2' );
TA> print $Template->get_template();

TA> The output will be:

TA> This is a <%% WORD_2 %%>

TA> As you can see, the []-tags have been converted to <>-tags by now, which
TA> is ideal to let the templating system create/generate templates.

gack!

well, that is easily done with a second pass looking for [] markers.

TA> I have been trying out some code since my original post, and I will post
TA> that code very soon (when it works a bit better than it does now). I feel
TA> that I'm getting quite close to dealing with that list thing.

yes, we need more templaters!

i have written so many little templaters for projects. i never seem to
get around to using one of the modules as they are overkill in my
experience. and writing templaters is easy.

uri

 

[Bill]

Hi!

I need to develop a templating system. I _cannot_ use on of the
templating systems found on CPAN as this utility will be used on
"proprietary" (and already existing) template files. So - please don't
tell me to go to CPAN; I already know of it, and I know the templating
systems found there. Thanks.

The problem is lists. And lists in lists (in lists...). I've decided to
have the data structure for my lists like this:
.....

This is from Data:umper, though I had to reformat it a bit. I guess you
all get the point. The template for this data looks like this (shortened):

<%% WEEK_LIST BEGIN %%>
<%% NR %%>:<%% DAY_LIST BEGIN %%> <%% DAY %%><%% DAY_LIST END %%>
<%% WEEK_LIST END %%>

Thus - in the end - the following output should be created:

1: 1 2 3 4 5 6 7
2: 8 9 10 11 12 13 14

Now. I have absolutely _no_ idea on how to create this list (and take
height for the fact that there can be unlimited nested lists). Anyone
feel like they have the guts to try?

Any chance this can just be translated to the Template.pm format:

<%% WEEK_LIST BEGIN %%>
<%% NR %%>:<%% DAY_LIST BEGIN %%> <%% DAY %%><%% DAY_LIST END %%>
<%% WEEK_LIST END %%>

becomes, for the array ref of array of hash refs to key nr and value
array ref daylist

(whitespace may need tweaking etc)

Template.pm template 'template.tp':
=============
[% FOREACH week=weeklist %]
[% week.nr %]: [% FOREACH day=week.daylist %] [% day %] [% END %]
[% END %]
==============
called with


use Template;
use strict;
use warnings;


my $template = new Template;

my $vars = {
weeklist => [
{ nr => 1, daylist => [ qw(1 2 3 4 5 6 7) ] },
{ nr => 2, daylist => [ qw(18 9 10 11 12 13 14)] },
],
};

$template->process('template.tp', $vars);

=========

Now, it seems to me that there should be a way to translate from your
templates to Template.pm's, though YMMV.

 

[Tore Aursand]

As you can see, the []-tags have been converted to <>-tags by now,
which is ideal to let the templating system create/generate templates.

gack!

Excactly. Imagine that these tags also can be combined;

template.txt
------------
<!-- [WORD_<!-- %%NR%% -->] -->

script.pl
---------
$Template->subst( 'NR', 1 );
print Template->get-template();

output
------

well, that is easily done with a second pass looking for [] markers.

Actually, yes. In my module it's the last thing that gets done when
processing the template.

AFAIT, however, this makes it impossible (?) to preprocess the template in
any form, as you never know what your tags _really_ will look like after
the user have been substituting and replacing tags.

yes, we need more templaters!

Indeed. It was very fun to write this module, until I came to the thing
with the lists (in lists...). Gah!

i have written so many little templaters for projects. i never seem to
get around to using one of the modules as they are overkill in my
experience. and writing templaters is easy.

Not only overkill, but too many of them introduces too much logic in the
template system. I feel that the logic should solely be in the scripts,
and that templates are only for "markup".

The closest thing to logic in my module is replacement tags;

template.txt
------------
<!-- %%TAG REPLACE BEGIN%% -->
This text should be replaced.
<!-- %%TAG REPLACE END%% -->

script.pl
---------
$Template->replace( 'TAG', 'something' );

I've also thought about introducing sorting (for lists), which can be
implemented directly in the template files;

<!-- %%USER_LIST BEGIN%% -->
<!-- %%_SORT:LASTNAME,STR,ASC;FIRSTNAME,STR,ASC%% -->
<!-- %%LASTNAME%% -->, <!-- %%FIRSTNAME%% -->
<!-- %%USER_LIST END%% -->

But I don't know about this one yet. Might as well keep it as clean and
simple as possible and have this done from the script instead.

 

[Tore Aursand]

Any chance this can just be translated to the Template.pm format:
[...]

It is possible, of course, but I don't want to walk that path for a number
of reasons;

* The system that generates these templates can't be messed with,
which in turn forces me to create a "gateway" (which converts
them).

* Template::Toolkit (and many of the other templating systems, as
well) are overkill (or too heavy), IMO.

* I still have a lot to learn when it comes to Perl. Creating a
templating system has forced me to come up with better ideas
for many things that I really didn't think much about before.

One small thing: Speed _will_ be an issue at some point, and my simple
benchmarks shows that my module is quite faster than Template::Toolkit and
HTML::Template (which are the templating systems that I've benchmarked it
against).

For the record: If I ever finishes this module, and it works as expected,
I will - of course - release it to CPAN. Name suggestions, anyone?


--
Tore Aursand <[email protected]>
"Have you ever had a dream, Neo, that you were so sure was real? What
if you were unable to wake from that dream? How would you know the
difference between the dream world and the real world?" (Morpheus, The
Matrix)

 

[Anno Siegel]

[...]

For the record: If I ever finishes this module, and it works as expected,
I will - of course - release it to CPAN. Name suggestions, anyone?

No. Writing modules is easy. Naming modules is hard.



Anno

 

[Uri Guttman]

TA> Excactly. Imagine that these tags also can be combined;

TA> template.txt
TA> ------------
TA> <!-- [WORD_<!-- %%NR%% -->] -->

double gack!

TA> Actually, yes. In my module it's the last thing that gets done when
TA> processing the template.

do you process code and text in the template?

TA> AFAIT, however, this makes it impossible (?) to preprocess the
TA> template in any form, as you never know what your tags _really_
TA> will look like after the user have been substituting and replacing
TA> tags.

why not? just do multiple passes seems to be the answer.

TA> Not only overkill, but too many of them introduces too much logic in the
TA> template system. I feel that the logic should solely be in the scripts,
TA> and that templates are only for "markup".

yep. that is the essence of my new design. pure perl logic and pure text
(usually in a here doc).

TA> The closest thing to logic in my module is replacement tags;

TA> template.txt
TA> ------------
TA> <!-- %%TAG REPLACE BEGIN%% -->
TA> This text should be replaced.
TA> <!-- %%TAG REPLACE END%% -->

TA> script.pl
TA> ---------
TA> $Template->replace( 'TAG', 'something' );

i supported simple if/else as it was useful to do it there. but the
condition was tested from the hash and can't be a general test. so you
must set the boolean result in the data tree first keeping the real
logic out of the text again. if i remember to get the code and examples
from work, i will show that too.

TA> I've also thought about introducing sorting (for lists), which can be
TA> implemented directly in the template files;

TA> <!-- %%USER_LIST BEGIN%% -->
TA> <!-- %%_SORT:LASTNAME,STR,ASC;FIRSTNAME,STR,ASC%% -->
TA> <!-- %%LASTNAME%% -->, <!-- %%FIRSTNAME%% -->
TA> <!-- %%USER_LIST END%% -->

but that seems to mean you can control the template syntax. i thought it
was fixed in concrete and passed to you. if you can change it some, you
can use some of my design.

uri

 

[Tore Aursand]

do you process code and text in the template?

You mean Perl code in the template file? No. The template "language" is
pure markup/tags; no logic or "code".

why not? just do multiple passes seems to be the answer.

Hmm. Maybe you're right. I need to look into this some more. But do I
really _want to_ do multiple passes if I _really_ don't have to?

but that seems to mean you can control the template syntax.

Does it? Why do you think so? Just because of that '_SORT' thing?
That's a feature I plan to _add_ to the existing syntax.

If you mean that I switch between '<%% ... %%>' and '<!-- %%TAG%% -->',
it's because the latter (which is the _real_ syntax) takes more space (and
I try to keep my line lengths below 80 characters on Usenet).

With my module you can change the start and end tag, however. 'new()'
accepts a list of arguments;

filename
tag_begin (default: '<!-- %%')
tag_end (default: '%% -->')

I have to change 'new()' though, so that it can accept either only a
filename _or_ a hash, as I want to write for example:

my $Template = Template->new(filename => 'template.thtml',
path => \@paths);

This way, I can let the module search for - and find the first occurance
of - a file named 'template.thtml' among @paths.

 

[Tore Aursand]

No. Writing modules is easy. Naming modules is hard.

Haha. Excellent point, Anno. May I add that statement to my signature
collection?

 

[David H. Adler]

Haha. Excellent point, Anno. May I add that statement to my signature
collection?

And if Tore may, may I?

dha

 

[Anno Siegel]

Haha. Excellent point, Anno. May I add that statement to my signature
collection?

Sure.

It's true, though. There are a lot of unrelated and partially
conflicting considerations that pertain to naming.

The name should be short and to the point.
The phrases "use <name>", and possibly "no <name>" should make sense.
Possible parameters should go well after "use <name> ...".
The name must fit in the existing CPAN hierarchy. That makes is long
and redundant.
The existing hierarchy is not entirely perspicuous.
There are often multiple places where a given module could go.
Choosing a place is a bit like staking out a claim in the name space,
especially if you have plans for the future.
You want to put the module where it belongs.
You want to put the module where people will find it.

The decision, once made, it hard to revise. Even before release,
a change is a coordinated action of renaming files, changing the
content of (these and other) files accordingly, and explaining
to CVS (or whatever) that you have had second thoughts. After
release the name is hewn in stone.

Anno