Creating Fixed Length Records

[Greg Lindstrom]

Hello-

I'm creating fixed-length record layouts for various record translations
I need to perform. I have been using a home-grown object,
"FixedLengthRecord" for about 4 years now and am very happy with it.
Almost. The burr under my saddle occurs when I serialize the record.
Currently, I create an empty text field and for each record in my
database (all record layouts, data types, lengths, defaults, etc. are
held in an SQL server) I concatenate the field to the text segment. I
have been led to believe this is bad form because Python will copy the
entire segment each time I add a field. Up until now, it was not a big
deal because the segments had at most 20 fields. I have just been
handed a record layout that is almost 5000 bytes long with 350 fields in
it. Gulp!! Although I could let my object churn away on this bad boy,
I'd like to know if there is a more pythonic way to serialize the record.

One thought I had, which might lead to an addition to the language, was
to use the struct module. If I could feed the pack method a format
string then a tuple of values (instead of individual values), then I
could create the format string once, then pass it a tuple with the
values for that record. Just a thought.

So, gurus, what are your suggestions to tame this record? Are there
easier ways that I'm just not seeing?

Thanks,
--greg

 

[Jeff Shannon]

[...] I concatenate the field to the text segment. I have been led to
believe this is bad form because Python will copy the entire segment
each time I add a field. Up until now, it was not a big deal because
the segments had at most 20 fields. I have just been handed a record
layout that is almost 5000 bytes long with 350 fields in it. Gulp!!
Although I could let my object churn away on this bad boy, I'd like to
know if there is a more pythonic way to serialize the record.

The standard Python way would be to read your fields into a list, and
then join() the list. Psuedocode-ish example:

values = []
for field in list_of_fields:
this_value = database.get(field) # or however you want to
retrieve it
values.append(this_value)
text = ''.join(values)

Of course, if you want to be able to reverse the process, then you will
need some kind of format definition. This could be an explicit list of
column numbers where fields break, or an implicit list in the form of a
struct module format string, or whatever. Using the struct module would
(I believe) be a reasonable way of getting data into/out of the
fixed-length block.

Jeff Shannon
Technician/Programmer
Credit International

 

[Stuart D. Gathman]

One thought I had, which might lead to an addition to the language, was
to use the struct module. If I could feed the pack method a format
string then a tuple of values (instead of individual values), then I
could create the format string once, then pass it a tuple with the
values for that record. Just a thought.

The language extension has been around since the early days. It was
originally called 'apply'. But now is called '*'.
'foo\x00\x00\x00\x00\x00bar\x00\x00\x00AA'