Add "Received:" header to email msg in correct position?

[Grant Edwards]

I'm working on a Python app that receives an e-mail message via SMTP,
does some trivial processing on it, and forwards it to another SMTP
server.

I'd like to do the polite thing and add a "Received:" header, but I
can't figure out how to get Python's email module to add it in the
correct place. It always ends up at the "bottom" of the headers below
From: To: etc. It's supposed to go at the above all the Received:
headers that where there when I received it.

 

[Ethan Furman]

I'm working on a Python app that receives an e-mail message via SMTP,
does some trivial processing on it, and forwards it to another SMTP
server.

I'd like to do the polite thing and add a "Received:" header, but I
can't figure out how to get Python's email module to add it in the
correct place. It always ends up at the "bottom" of the headers below
From: To: etc. It's supposed to go at the above all the Received:
headers that where there when I received it.

I don't know that it matters, but which Python version?

 

[Grant Edwards]

On 05/05/2014 12:51 PM, Grant Edwards wrote:
[...]

I'd like to do the polite thing and add a "Received:" header, but I
can't figure out how to get Python's email module to add it in the
correct place. It always ends up at the "bottom" of the headers below
From: To: etc. It's supposed to go at the above all the Received:
headers that where there when I received it.

I don't know that it matters, but which Python version?

Sorry, should have mentioned it: 2.7.5

 

[Tim Chase]

Sorry, should have mentioned it: 2.7.5

Looking at the stdlib source, it doesn't look like there's an easy
way to specify where it gets inserted. However, the source to
email.message.Message.add_header() is all of 9 lines of code, so it
wouldn't be too hard to subclass Message and twiddle self._headers as
you would any other list (i.e., using .insert() to specify an index).

It might look something like

class MyMessage(email.message.Message):
def insert_header(self, index, _name, _value, **_params):
parts = []
for k, v in _params.items():
if v is None:
parts.append(k.replace('_', '-'))
else:
parts.append(_formatparam(k.replace('_', '-'), v))
if _value is not None:
parts.insert(0, _value)
self._headers.insert(index, (_name, SEMISPACE.join(parts)))

You might still need to search for *where* you want to insert it, but
I'll leave that as an exercise to the reader.

-tkc






-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBCgAGBQJTZ/4KAAoJEN385Nh6n9aQmagIAKkeLRANxfbsl4kDkCkuCWn/
uYothgT82ihtFwmfSdPeMbrjIUAPnPVl3jwujU9nYcFRwT7bXBbiCTVapBSeeCD0
uInv8lPhEOLQgvATy4oABQnTqw5aQwylTi9RSgwozNu2E89xi0sYuVF+36WniOlk
ACZ0HPh5lhYWKMB0SSmNo9TmpVWDyhOXTe7W8PJjxjXRatBcgV4WwGUb2QiAXH4f
Z/eMAXhcABcJDJ6SL12OaAMGGF3ooBDN+Wm7GTzu6ka98hwM7qcxXjilBtXiMxV3
L2nIZXzaTp9ChtxaZuh0AR2OfEW0l8ditSFJgExss0edbgEaq/JI+4/im2fKHCQ=
=swTy
-----END PGP SIGNATURE-----

 

[Grant Edwards]

Looking at the stdlib source, it doesn't look like there's an easy
way to specify where it gets inserted.

Thanks.

There's is a somewhat messy way to do it by calling msg.items() to
retrieve all the current headers, removing all of them, and then
adding them all back (includeing the new one) in the order I want
them.

However, the source to email.message.Message.add_header() is all of 9
lines of code, so it wouldn't be too hard to subclass Message and
twiddle self._headers as you would any other list (i.e., using
.insert() to specify an index).

[ insert_header() method example that lets you specify an index ]

A couple other options I was thinking about:

* Just override __set_item__ so it treats 'Received' as a special
case and inserts it before any existing 'Received' headers -- or
prepends it if there aren't any.

* Add a prepend_header() method that just sticks it at the top -- in
most cases, I think that will be correct enough.

I think I like your suggestion better. Since the keys() method
preserves the order of header keys, that makes figuring out the
insertion point trivial.

 

[Steven D'Aprano]

There's is a somewhat messy way to do it by calling msg.items() to
retrieve all the current headers, removing all of them, and then adding
them all back (includeing the new one) in the order I want them.

Does msg.items() return the actual message header list, or a copy?
Because if it returns the list itself, you can just insert directly into
the list using the insert method.

 

[Chris Angelico]

Is this required or just being polite?
what I mean is does the standard state the headers must be in a
particular order or can they appear anywhere, you may be spending time
trying to resolve an issue that does not need fixing.

Yes, it's required. RFC 2821 [1] section 3.8.2 says "prepend".

ChrisA

[1] http://www.ietf.org/rfc/rfc2821.txt

 

[Chris Angelico]

On Mon, 05 May 2014 19:51:15 +0000, Grant Edwards wrote:

I'm working on a Python app that receives an e-mail message via SMTP,
does some trivial processing on it, and forwards it to another SMTP
server.

I'd like to do the polite thing and add a "Received:" header, but I
can't figure out how to get Python's email module to add it in the
correct place. It always ends up at the "bottom" of the headers below
From: To: etc. It's supposed to go at the above all the Received:
headers that where there when I received it.

Is this required or just being polite?
what I mean is does the standard state the headers must be in a
particular order or can they appear anywhere, you may be spending time
trying to resolve an issue that does not need fixing.

Yes, it's required. RFC 2821 [1] section 3.8.2 says "prepend".

ChrisA

[1] http://www.ietf.org/rfc/rfc2821.txt

oh well, so much for the easy route
This suggests the email module could do with amending so that headers can
be pretended as appended.

Even if it's special-cased as a dedicated "prepend received header"
method, that would probably do. I can't think of any other headers
where you need to specifically order them rather than appending.

ChrisA

 

[Grant Edwards]

Does msg.items() return the actual message header list, or a copy?

A shallow copy:

def items(self):
"""Get all the message's header fields and values.

These will be sorted in the order they appeared in the original
message, or were added to the message, and may contain duplicates.
Any fields deleted and re-inserted are always appended to the header
list.
"""
return self._headers[:]

 

[Grant Edwards]

Is this required or just being polite?

I couldn't find it in an RFC. But every reference I could find that
mentioned Received: headers did say that you read them from the bottom
up (most recent is on top).

what I mean is does the standard state the headers must be in a
particular order or can they appear anywhere, you may be spending
time trying to resolve an issue that does not need fixing.

I'd like to do it the right way whether it's required by the letter of
the law or not.

 

[Grant Edwards]

I couldn't find it in an RFC.

I was looking in the message format RFCs, and as has already been
pointed out, it's in the SMTP spec (RFC2821). Since there _is_ an RFC
that specifically states you MUST prepend a header line to a message,
this seems to be a hole in the email module...

 

[Chris Angelico]

I couldn't find it in an RFC. But every reference I could find that
mentioned Received: headers did say that you read them from the bottom
up (most recent is on top).

Yeah, I went looking for it in 2822 but didn't find much. After some
digging, found it in 2821, section 3.8.2. It's not dwelled on, but the
word "prepend" is used, which to my mind is a clear indication of
necessary ordering.

ChrisA

 

[Burak Arslan]

Is this required or just being polite?
what I mean is does the standard state the headers must be in a
particular order or can they appear anywhere, you may be spending time
trying to resolve an issue that does not need fixing.

Yes, it's required. RFC 2821 [1] section 3.8.2 says "prepend".

The rationale for "prepend" is to make it possible for MTAs to add their
"Received:" headers to messages without having to parse them.

So you're supposed to do the same: Just write your Received header,
followed by '\r\n', followed by the rest of the message to the socket
and you should be fine.

Best,
Burak

 

[Grant Edwards]

On Mon, 05 May 2014 19:51:15 +0000, Grant Edwards wrote:

I'm working on a Python app that receives an e-mail message via SMTP,
does some trivial processing on it, and forwards it to another SMTP
server.

I'd like to do the polite thing and add a "Received:" header, but I
can't figure out how to get Python's email module to add it in the
correct place. It always ends up at the "bottom" of the headers below
From: To: etc. It's supposed to go at the above all the Received:
headers that where there when I received it.

Is this required or just being polite?
what I mean is does the standard state the headers must be in a
particular order or can they appear anywhere, you may be spending time
trying to resolve an issue that does not need fixing.

Yes, it's required. RFC 2821 [1] section 3.8.2 says "prepend".

The rationale for "prepend" is to make it possible for MTAs to add
their "Received:" headers to messages without having to parse them.

So you're supposed to do the same: Just write your Received header,
followed by '\r\n', followed by the rest of the message to the socket
and you should be fine.

I need to check and manipulate other headers for other reasons, so I'm
using the email module for that. In order to keep things consistent
and easy to understand, I'd like to use the email module to prepend
the Received header as well. That keeps my application from having to
have any knowledge about e-mail message formatting.

 

[Terry Reedy]

I usually does these days.

Sorry, should have mentioned it: 2.7.5

email has been improved with successive 3.x versions.
https://docs.python.org/3/library/email.html#package-history

There might be an email5 backport for 2.7 on pypi.

If the prepend requirement is covered by

"The email package is a library for managing email messages, including
MIME and other RFC 2822-based message documents. It is specifically not
designed to do any sending of email messages to SMTP (RFC 2821), NNTP,
or other servers; those are functions of modules such as smtplib and
nntplib. The email package attempts to be as RFC-compliant as possible,
supporting in addition to RFC 2822, such MIME-related RFCs as RFC 2045,
RFC 2046, RFC 2047, and RFC 2231."

and the current 3.4/5 version does not prepend and there is no existing
tracker issue, then a new issue would seem to be appropriate.

 

[Chris Angelico]

If the prepend requirement is covered by

"The email package is a library for managing email messages, including MIME
and other RFC 2822-based message documents. It is specifically not designed
to do any sending of email messages to SMTP (RFC 2821), NNTP, or other
servers; those are functions of modules such as smtplib and nntplib. The
email package attempts to be as RFC-compliant as possible, supporting in
addition to RFC 2822, such MIME-related RFCs as RFC 2045, RFC 2046, RFC
2047, and RFC 2231."

and the current 3.4/5 version does not prepend and there is no existing
tracker issue, then a new issue would seem to be appropriate.

That's a bit tricky. RFC 2822 section 3.6.7 says the Received: headers
are "strictly informational, and any formal interpretation of them is
outside of the scope of this document", but it does reference RFC
2821. Should the addition of another Received header be part of
building an RFC 2822 compliant message, or should you build up a
message without that, and have one added at transport time?

I would say that, even if this isn't considered a bug (as in, failure
to comply with standards it claims to comply with), it would still be
a viable feature addition.

ChrisA

 

[Antoon Pardon]

I'm working on a Python app that receives an e-mail message via SMTP,
does some trivial processing on it, and forwards it to another SMTP
server.

I'd like to do the polite thing and add a "Received:" header, but I
can't figure out how to get Python's email module to add it in the
correct place. It always ends up at the "bottom" of the headers below
From: To: etc. It's supposed to go at the above all the Received:
headers that where there when I received it.

I thought that was the job of the SMTP servers, not of email
applications. So I'm not sure
that what you want to do is the polite thing to do.

 

[Emre Hasegeli]

I'm working on a Python app that receives an e-mail message via SMTP,

I thought that was the job of the SMTP servers, not of email
applications. So I'm not sure
that what you want to do is the polite thing to do.

This application seems like an SMTP server.

 

[Antoon Pardon]

Antoon Pardon <[email protected]


I thought that was the job of the SMTP servers, not of email
applications. So I'm not sure
that what you want to do is the polite thing to do.


This application seems like an SMTP server.

It doesn't to me. As far as I can see what he wants to do can be done
by a mail program like procmail in combination with some mail
filtering/processing.

 

[Grant Edwards]

I thought that was the job of the SMTP servers, not of email
applications.

Exactly. And the SMTP server I'm writing uses the email module to
manipulate headers as messages are processed.

So I'm not sure that what you want to do is the polite thing to do.

Why not?