cpython-lru_cache-weakref: Lib/email/_header_value

annotate Lib/email/_header_value_parser.py @ 79691:05c8928362ee

Merge __next__ method link fixes with 3.3.

author	Ezio Melotti <ezio.melotti@gmail.com>
date	Fri, 12 Oct 2012 13:46:37 +0300
parents	fb90e2ff95b7
children	908d45fdbecd

rev	line source
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1 """Header value parser implementing various email-related RFC parsing rules.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	3 The parsing methods defined in this module implement various email related
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	4 parsing rules. Principal among them is RFC 5322, which is the followon
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	5 to RFC 2822 and primarily a clarification of the former. It also implements
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	6 RFC 2047 encoded word decoding.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	7
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	8 RFC 5322 goes to considerable trouble to maintain backward compatibility with
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	9 RFC 822 in the parse phase, while cleaning up the structure on the generation
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	10 phase. This parser supports correct RFC 5322 generation by tagging white space
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	11 as folding white space only when folding is allowed in the non-obsolete rule
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	12 sets. Actually, the parser is even more generous when accepting input than RFC
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	13 5322 mandates, following the spirit of Postel's Law, which RFC 5322 encourages.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	14 Where possible deviations from the standard are annotated on the 'defects'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	15 attribute of tokens that deviate.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	16
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	17 The general structure of the parser follows RFC 5322, and uses its terminology
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	18 where there is a direct correspondence. Where the implementation requires a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	19 somewhat different structure than that used by the formal grammar, new terms
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	20 that mimic the closest existing terms are used. Thus, it really helps to have
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	21 a copy of RFC 5322 handy when studying this code.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	22
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	23 Input to the parser is a string that has already been unfolded according to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	24 RFC 5322 rules. According to the RFC this unfolding is the very first step, and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	25 this parser leaves the unfolding step to a higher level message parser, which
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	26 will have already detected the line breaks that need unfolding while
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	27 determining the beginning and end of each header.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	28
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	29 The output of the parser is a TokenList object, which is a list subclass. A
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	30 TokenList is a recursive data structure. The terminal nodes of the structure
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	31 are Terminal objects, which are subclasses of str. These do not correspond
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	32 directly to terminal objects in the formal grammar, but are instead more
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	33 practical higher level combinations of true terminals.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	34
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	35 All TokenList and Terminal objects have a 'value' attribute, which produces the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	36 semantically meaningful value of that part of the parse subtree. The value of
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	37 all whitespace tokens (no matter how many sub-tokens they may contain) is a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	38 single space, as per the RFC rules. This includes 'CFWS', which is herein
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	39 included in the general class of whitespace tokens. There is one exception to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	40 the rule that whitespace tokens are collapsed into single spaces in values: in
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	41 the value of a 'bare-quoted-string' (a quoted-string with no leading or
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	42 trailing whitespace), any whitespace that appeared between the quotation marks
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	43 is preserved in the returned value. Note that in all Terminal strings quoted
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	44 pairs are turned into their unquoted values.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	45
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	46 All TokenList and Terminal objects also have a string value, which attempts to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	47 be a "canonical" representation of the RFC-compliant form of the substring that
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	48 produced the parsed subtree, including minimal use of quoted pair quoting.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	49 Whitespace runs are not collapsed.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	50
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	51 Comment tokens also have a 'content' attribute providing the string found
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	52 between the parens (including any nested comments) with whitespace preserved.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	53
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	54 All TokenList and Terminal objects have a 'defects' attribute which is a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	55 possibly empty list all of the defects found while creating the token. Defects
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	56 may appear on any token in the tree, and a composite list of all defects in the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	57 subtree is available through the 'all_defects' attribute of any node. (For
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	58 Terminal notes x.defects == x.all_defects.)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	59
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	60 Each object in a parse tree is called a 'token', and each has a 'token_type'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	61 attribute that gives the name from the RFC 5322 grammar that it represents.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	62 Not all RFC 5322 nodes are produced, and there is one non-RFC 5322 node that
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	63 may be produced: 'ptext'. A 'ptext' is a string of printable ascii characters.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	64 It is returned in place of lists of (ctext/quoted-pair) and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	65 (qtext/quoted-pair).
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	66
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	67 XXX: provide complete list of token types.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	68 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	69
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	70 import re
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	71 import urllib # For urllib.parse.unquote
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	72 from collections import namedtuple, OrderedDict
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	73 from email import _encoded_words as _ew
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	74 from email import errors
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	75 from email import utils
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	76
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	77 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	78 # Useful constants and functions
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	79 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	80
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	81 WSP = set(' \t')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	82 CFWS_LEADER = WSP \| set('(')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	83 SPECIALS = set(r'()<>@,:;.\"[]')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	84 ATOM_ENDS = SPECIALS \| WSP
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	85 DOT_ATOM_ENDS = ATOM_ENDS - set('.')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	86 # '.', '"', and '(' do not end phrases in order to support obs-phrase
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	87 PHRASE_ENDS = SPECIALS - set('."(')
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	88 TSPECIALS = (SPECIALS \| set('/?=')) - set('.')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	89 TOKEN_ENDS = TSPECIALS \| WSP
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	90 ASPECIALS = TSPECIALS \| set("*'%")
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	91 ATTRIBUTE_ENDS = ASPECIALS \| WSP
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	92 EXTENDED_ATTRIBUTE_ENDS = ATTRIBUTE_ENDS - set('%')
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	93
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	94 def quote_string(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	95 return '"'+str(value).replace('\\', '\\\\').replace('"', r'\"')+'"'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	96
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	97 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	98 # Accumulator for header folding
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	99 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	100
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	101 class _Folded:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	102
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	103 def __init__(self, maxlen, policy):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	104 self.maxlen = maxlen
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	105 self.policy = policy
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	106 self.lastlen = 0
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	107 self.stickyspace = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	108 self.firstline = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	109 self.done = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	110 self.current = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	111
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	112 def newline(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	113 self.done.extend(self.current)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	114 self.done.append(self.policy.linesep)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	115 self.current.clear()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	116 self.lastlen = 0
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	117
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	118 def finalize(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	119 if self.current:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	120 self.newline()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	121
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	122 def __str__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	123 return ''.join(self.done)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	124
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	125 def append(self, stoken):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	126 self.current.append(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	127
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	128 def append_if_fits(self, token, stoken=None):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	129 if stoken is None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	130 stoken = str(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	131 l = len(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	132 if self.stickyspace is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	133 stickyspace_len = len(self.stickyspace)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	134 if self.lastlen + stickyspace_len + l <= self.maxlen:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	135 self.current.append(self.stickyspace)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	136 self.lastlen += stickyspace_len
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	137 self.current.append(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	138 self.lastlen += l
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	139 self.stickyspace = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	140 self.firstline = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	141 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	142 if token.has_fws:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	143 ws = token.pop_leading_fws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	144 if ws is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	145 self.stickyspace += str(ws)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	146 stickyspace_len += len(ws)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	147 token._fold(self)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	148 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	149 if stickyspace_len and l + 1 <= self.maxlen:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	150 margin = self.maxlen - l
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	151 if 0 < margin < stickyspace_len:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	152 trim = stickyspace_len - margin
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	153 self.current.append(self.stickyspace[:trim])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	154 self.stickyspace = self.stickyspace[trim:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	155 stickyspace_len = trim
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	156 self.newline()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	157 self.current.append(self.stickyspace)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	158 self.current.append(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	159 self.lastlen = l + stickyspace_len
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	160 self.stickyspace = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	161 self.firstline = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	162 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	163 if not self.firstline:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	164 self.newline()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	165 self.current.append(self.stickyspace)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	166 self.current.append(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	167 self.stickyspace = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	168 self.firstline = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	169 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	170 if self.lastlen + l <= self.maxlen:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	171 self.current.append(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	172 self.lastlen += l
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	173 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	174 if l < self.maxlen:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	175 self.newline()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	176 self.current.append(stoken)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	177 self.lastlen = l
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	178 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	179 return False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	180
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	181 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	182 # TokenList and its subclasses
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	183 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	184
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	185 class TokenList(list):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	186
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	187 token_type = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	188
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	189 def __init__(self, args, *kw):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	190 super().__init__(args, *kw)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	191 self.defects = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	192
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	193 def __str__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	194 return ''.join(str(x) for x in self)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	195
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	196 def __repr__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	197 return '{}({})'.format(self.__class__.__name__,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	198 super().__repr__())
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	199
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	200 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	201 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	202 return ''.join(x.value for x in self if x.value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	203
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	204 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	205 def all_defects(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	206 return sum((x.all_defects for x in self), self.defects)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	207
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	208 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	209 # Folding API
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	210 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	211 # parts():
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	212 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	213 # return a list of objects that constitute the "higher level syntactic
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	214 # objects" specified by the RFC as the best places to fold a header line.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	215 # The returned objects must include leading folding white space, even if
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	216 # this means mutating the underlying parse tree of the object. Each object
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	217 # is only responsible for returning its parts, and should not drill down
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	218 # to any lower level except as required to meet the leading folding white
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	219 # space constraint.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	220 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	221 # _fold(folded):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	222 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	223 # folded: the result accumulator. This is an instance of _Folded.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	224 # (XXX: I haven't finished factoring this out yet, the folding code
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	225 # pretty much uses this as a state object.) When the folded.current
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	226 # contains as much text as will fit, the _fold method should call
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	227 # folded.newline.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	228 # folded.lastlen: the current length of the test stored in folded.current.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	229 # folded.maxlen: The maximum number of characters that may appear on a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	230 # folded line. Differs from the policy setting in that "no limit" is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	231 # represented by +inf, which means it can be used in the trivially
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	232 # logical fashion in comparisons.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	233 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	234 # Currently no subclasses implement parts, and I think this will remain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	235 # true. A subclass only needs to implement _fold when the generic version
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	236 # isn't sufficient. _fold will need to be implemented primarily when it is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	237 # possible for encoded words to appear in the specialized token-list, since
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	238 # there is no generic algorithm that can know where exactly the encoded
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	239 # words are allowed. A _fold implementation is responsible for filling
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	240 # lines in the same general way that the top level _fold does. It may, and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	241 # should, call the _fold method of sub-objects in a similar fashion to that
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	242 # of the top level _fold.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	243 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	244 # XXX: I'm hoping it will be possible to factor the existing code further
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	245 # to reduce redundancy and make the logic clearer.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	246
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	247 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	248 def parts(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	249 klass = self.__class__
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	250 this = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	251 for token in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	252 if token.startswith_fws():
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	253 if this:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	254 yield this[0] if len(this)==1 else klass(this)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	255 this.clear()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	256 end_ws = token.pop_trailing_ws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	257 this.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	258 if end_ws:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	259 yield klass(this)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	260 this = [end_ws]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	261 if this:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	262 yield this[0] if len(this)==1 else klass(this)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	263
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	264 def startswith_fws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	265 return self[0].startswith_fws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	266
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	267 def pop_leading_fws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	268 if self[0].token_type == 'fws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	269 return self.pop(0)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	270 return self[0].pop_leading_fws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	271
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	272 def pop_trailing_ws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	273 if self[-1].token_type == 'cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	274 return self.pop(-1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	275 return self[-1].pop_trailing_ws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	276
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	277 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	278 def has_fws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	279 for part in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	280 if part.has_fws:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	281 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	282 return False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	283
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	284 def has_leading_comment(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	285 return self[0].has_leading_comment()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	286
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	287 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	288 def comments(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	289 comments = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	290 for token in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	291 comments.extend(token.comments)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	292 return comments
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	293
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	294 def fold(self, *, policy):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	295 # max_line_length 0/None means no limit, ie: infinitely long.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	296 maxlen = policy.max_line_length or float("+inf")
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	297 folded = _Folded(maxlen, policy)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	298 self._fold(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	299 folded.finalize()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	300 return str(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	301
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	302 def as_encoded_word(self, charset):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	303 # This works only for things returned by 'parts', which include
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	304 # the leading fws, if any, that should be used.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	305 res = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	306 ws = self.pop_leading_fws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	307 if ws:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	308 res.append(ws)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	309 trailer = self.pop(-1) if self[-1].token_type=='fws' else ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	310 res.append(_ew.encode(str(self), charset))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	311 res.append(trailer)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	312 return ''.join(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	313
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	314 def cte_encode(self, charset, policy):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	315 res = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	316 for part in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	317 res.append(part.cte_encode(charset, policy))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	318 return ''.join(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	319
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	320 def _fold(self, folded):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	321 for part in self.parts:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	322 tstr = str(part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	323 tlen = len(tstr)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	324 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	325 str(part).encode('us-ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	326 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	327 if any(isinstance(x, errors.UndecodableBytesDefect)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	328 for x in part.all_defects):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	329 charset = 'unknown-8bit'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	330 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	331 # XXX: this should be a policy setting
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	332 charset = 'utf-8'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	333 tstr = part.cte_encode(charset, folded.policy)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	334 tlen = len(tstr)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	335 if folded.append_if_fits(part, tstr):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	336 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	337 # Peel off the leading whitespace if any and make it sticky, to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	338 # avoid infinite recursion.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	339 ws = part.pop_leading_fws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	340 if ws is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	341 # Peel off the leading whitespace and make it sticky, to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	342 # avoid infinite recursion.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	343 folded.stickyspace = str(part.pop(0))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	344 if folded.append_if_fits(part):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	345 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	346 if part.has_fws:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	347 part._fold(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	348 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	349 # There are no fold points in this one; it is too long for a single
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	350 # line and can't be split...we just have to put it on its own line.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	351 folded.append(tstr)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	352 folded.newline()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	353
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	354 def pprint(self, indent=''):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	355 print('\n'.join(self._pp(indent='')))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	356
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	357 def ppstr(self, indent=''):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	358 return '\n'.join(self._pp(indent=''))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	359
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	360 def _pp(self, indent=''):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	361 yield '{}{}/{}('.format(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	362 indent,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	363 self.__class__.__name__,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	364 self.token_type)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	365 for token in self:
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	366 if not hasattr(token, '_pp'):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	367 yield (indent + ' !! invalid element in token '
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	368 'list: {!r}'.format(token))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	369 else:
79378 fb90e2ff95b7 utilize yield from Philip Jenvey <pjenvey@underboss.org> parents: 77681 diff changeset	370 yield from token._pp(indent+' ')
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	371 if self.defects:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	372 extra = ' Defects: {}'.format(self.defects)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	373 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	374 extra = ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	375 yield '{}){}'.format(indent, extra)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	376
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	377
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	378 class WhiteSpaceTokenList(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	379
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	380 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	381 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	382 return ' '
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	383
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	384 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	385 def comments(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	386 return [x.content for x in self if x.token_type=='comment']
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	387
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	388
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	389 class UnstructuredTokenList(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	390
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	391 token_type = 'unstructured'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	392
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	393 def _fold(self, folded):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	394 if any(x.token_type=='encoded-word' for x in self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	395 return self._fold_encoded(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	396 # Here we can have either a pure ASCII string that may or may not
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	397 # have surrogateescape encoded bytes, or a unicode string.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	398 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	399 for part in self.parts:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	400 tstr = str(part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	401 is_ew = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	402 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	403 str(part).encode('us-ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	404 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	405 if any(isinstance(x, errors.UndecodableBytesDefect)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	406 for x in part.all_defects):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	407 charset = 'unknown-8bit'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	408 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	409 charset = 'utf-8'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	410 if last_ew is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	411 # We've already done an EW, combine this one with it
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	412 # if there's room.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	413 chunk = get_unstructured(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	414 ''.join(folded.current[last_ew:]+[tstr])).as_encoded_word(charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	415 oldlastlen = sum(len(x) for x in folded.current[:last_ew])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	416 schunk = str(chunk)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	417 lchunk = len(schunk)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	418 if oldlastlen + lchunk <= folded.maxlen:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	419 del folded.current[last_ew:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	420 folded.append(schunk)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	421 folded.lastlen = oldlastlen + lchunk
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	422 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	423 tstr = part.as_encoded_word(charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	424 is_ew = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	425 if folded.append_if_fits(part, tstr):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	426 if is_ew:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	427 last_ew = len(folded.current) - 1
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	428 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	429 if is_ew or last_ew:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	430 # It's too big to fit on the line, but since we've
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	431 # got encoded words we can use encoded word folding.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	432 part._fold_as_ew(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	433 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	434 # Peel off the leading whitespace if any and make it sticky, to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	435 # avoid infinite recursion.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	436 ws = part.pop_leading_fws()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	437 if ws is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	438 folded.stickyspace = str(ws)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	439 if folded.append_if_fits(part):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	440 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	441 if part.has_fws:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	442 part.fold(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	443 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	444 # It can't be split...we just have to put it on its own line.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	445 folded.append(tstr)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	446 folded.newline()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	447 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	448
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	449 def cte_encode(self, charset, policy):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	450 res = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	451 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	452 for part in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	453 spart = str(part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	454 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	455 spart.encode('us-ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	456 res.append(spart)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	457 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	458 if last_ew is None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	459 res.append(part.cte_encode(charset, policy))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	460 last_ew = len(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	461 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	462 tl = get_unstructured(''.join(res[last_ew:] + [spart]))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	463 res.append(tl.as_encoded_word())
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	464 return ''.join(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	465
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	466
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	467 class Phrase(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	468
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	469 token_type = 'phrase'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	470
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	471 def _fold(self, folded):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	472 # As with Unstructured, we can have pure ASCII with or without
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	473 # surrogateescape encoded bytes, or we could have unicode. But this
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	474 # case is more complicated, since we have to deal with the various
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	475 # sub-token types and how they can be composed in the face of
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	476 # unicode-that-needs-CTE-encoding, and the fact that if a token a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	477 # comment that becomes a barrier across which we can't compose encoded
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	478 # words.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	479 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	480 for part in self.parts:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	481 tstr = str(part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	482 tlen = len(tstr)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	483 has_ew = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	484 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	485 str(part).encode('us-ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	486 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	487 if any(isinstance(x, errors.UndecodableBytesDefect)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	488 for x in part.all_defects):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	489 charset = 'unknown-8bit'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	490 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	491 charset = 'utf-8'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	492 if last_ew is not None and not part.has_leading_comment():
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	493 # We've already done an EW, let's see if we can combine
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	494 # this one with it. The last_ew logic ensures that all we
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	495 # have at this point is atoms, no comments or quoted
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	496 # strings. So we can treat the text between the last
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	497 # encoded word and the content of this token as
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	498 # unstructured text, and things will work correctly. But
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	499 # we have to strip off any trailing comment on this token
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	500 # first, and if it is a quoted string we have to pull out
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	501 # the content (we're encoding it, so it no longer needs to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	502 # be quoted).
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	503 if part[-1].token_type == 'cfws' and part.comments:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	504 remainder = part.pop(-1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	505 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	506 remainder = ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	507 for i, token in enumerate(part):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	508 if token.token_type == 'bare-quoted-string':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	509 part[i] = UnstructuredTokenList(token[:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	510 chunk = get_unstructured(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	511 ''.join(folded.current[last_ew:]+[tstr])).as_encoded_word(charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	512 schunk = str(chunk)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	513 lchunk = len(schunk)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	514 if last_ew + lchunk <= folded.maxlen:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	515 del folded.current[last_ew:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	516 folded.append(schunk)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	517 folded.lastlen = sum(len(x) for x in folded.current)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	518 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	519 tstr = part.as_encoded_word(charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	520 tlen = len(tstr)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	521 has_ew = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	522 if folded.append_if_fits(part, tstr):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	523 if has_ew and not part.comments:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	524 last_ew = len(folded.current) - 1
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	525 elif part.comments or part.token_type == 'quoted-string':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	526 # If a comment is involved we can't combine EWs. And if a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	527 # quoted string is involved, it's not worth the effort to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	528 # try to combine them.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	529 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	530 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	531 part._fold(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	532
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	533 def cte_encode(self, charset, policy):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	534 res = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	535 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	536 is_ew = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	537 for part in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	538 spart = str(part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	539 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	540 spart.encode('us-ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	541 res.append(spart)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	542 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	543 is_ew = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	544 if last_ew is None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	545 if not part.comments:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	546 last_ew = len(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	547 res.append(part.cte_encode(charset, policy))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	548 elif not part.has_leading_comment():
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	549 if part[-1].token_type == 'cfws' and part.comments:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	550 remainder = part.pop(-1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	551 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	552 remainder = ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	553 for i, token in enumerate(part):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	554 if token.token_type == 'bare-quoted-string':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	555 part[i] = UnstructuredTokenList(token[:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	556 tl = get_unstructured(''.join(res[last_ew:] + [spart]))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	557 res[last_ew:] = [tl.as_encoded_word(charset)]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	558 if part.comments or (not is_ew and part.token_type == 'quoted-string'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	559 last_ew = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	560 return ''.join(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	561
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	562 class Word(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	563
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	564 token_type = 'word'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	565
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	566
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	567 class CFWSList(WhiteSpaceTokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	568
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	569 token_type = 'cfws'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	570
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	571 def has_leading_comment(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	572 return bool(self.comments)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	573
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	574
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	575 class Atom(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	576
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	577 token_type = 'atom'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	578
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	579
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	580 class Token(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	581
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	582 token_type = 'token'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	583
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	584
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	585 class EncodedWord(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	586
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	587 token_type = 'encoded-word'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	588 cte = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	589 charset = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	590 lang = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	591
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	592 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	593 def encoded(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	594 if self.cte is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	595 return self.cte
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	596 _ew.encode(str(self), self.charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	597
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	598
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	599
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	600 class QuotedString(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	601
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	602 token_type = 'quoted-string'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	603
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	604 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	605 def content(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	606 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	607 if x.token_type == 'bare-quoted-string':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	608 return x.value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	609
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	610 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	611 def quoted_value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	612 res = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	613 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	614 if x.token_type == 'bare-quoted-string':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	615 res.append(str(x))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	616 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	617 res.append(x.value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	618 return ''.join(res)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	619
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	620 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	621 def stripped_value(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	622 for token in self:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	623 if token.token_type == 'bare-quoted-string':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	624 return token.value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	625
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	626
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	627 class BareQuotedString(QuotedString):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	628
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	629 token_type = 'bare-quoted-string'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	630
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	631 def __str__(self):
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	632 return quote_string(''.join(str(x) for x in self))
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	633
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	634 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	635 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	636 return ''.join(str(x) for x in self)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	637
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	638
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	639 class Comment(WhiteSpaceTokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	640
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	641 token_type = 'comment'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	642
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	643 def __str__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	644 return ''.join(sum([
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	645 ["("],
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	646 [self.quote(x) for x in self],
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	647 [")"],
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	648 ], []))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	649
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	650 def quote(self, value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	651 if value.token_type == 'comment':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	652 return str(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	653 return str(value).replace('\\', '\\\\').replace(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	654 '(', '\(').replace(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	655 ')', '\)')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	656
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	657 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	658 def content(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	659 return ''.join(str(x) for x in self)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	660
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	661 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	662 def comments(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	663 return [self.content]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	664
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	665 class AddressList(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	666
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	667 token_type = 'address-list'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	668
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	669 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	670 def addresses(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	671 return [x for x in self if x.token_type=='address']
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	672
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	673 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	674 def mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	675 return sum((x.mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	676 for x in self if x.token_type=='address'), [])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	677
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	678 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	679 def all_mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	680 return sum((x.all_mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	681 for x in self if x.token_type=='address'), [])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	682
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	683
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	684 class Address(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	685
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	686 token_type = 'address'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	687
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	688 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	689 def display_name(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	690 if self[0].token_type == 'group':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	691 return self[0].display_name
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	692
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	693 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	694 def mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	695 if self[0].token_type == 'mailbox':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	696 return [self[0]]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	697 elif self[0].token_type == 'invalid-mailbox':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	698 return []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	699 return self[0].mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	700
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	701 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	702 def all_mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	703 if self[0].token_type == 'mailbox':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	704 return [self[0]]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	705 elif self[0].token_type == 'invalid-mailbox':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	706 return [self[0]]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	707 return self[0].all_mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	708
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	709 class MailboxList(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	710
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	711 token_type = 'mailbox-list'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	712
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	713 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	714 def mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	715 return [x for x in self if x.token_type=='mailbox']
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	716
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	717 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	718 def all_mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	719 return [x for x in self
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	720 if x.token_type in ('mailbox', 'invalid-mailbox')]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	721
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	722
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	723 class GroupList(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	724
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	725 token_type = 'group-list'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	726
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	727 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	728 def mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	729 if not self or self[0].token_type != 'mailbox-list':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	730 return []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	731 return self[0].mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	732
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	733 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	734 def all_mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	735 if not self or self[0].token_type != 'mailbox-list':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	736 return []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	737 return self[0].all_mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	738
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	739
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	740 class Group(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	741
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	742 token_type = "group"
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	743
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	744 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	745 def mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	746 if self[2].token_type != 'group-list':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	747 return []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	748 return self[2].mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	749
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	750 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	751 def all_mailboxes(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	752 if self[2].token_type != 'group-list':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	753 return []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	754 return self[2].all_mailboxes
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	755
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	756 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	757 def display_name(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	758 return self[0].display_name
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	759
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	760
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	761 class NameAddr(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	762
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	763 token_type = 'name-addr'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	764
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	765 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	766 def display_name(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	767 if len(self) == 1:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	768 return None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	769 return self[0].display_name
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	770
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	771 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	772 def local_part(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	773 return self[-1].local_part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	774
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	775 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	776 def domain(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	777 return self[-1].domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	778
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	779 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	780 def route(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	781 return self[-1].route
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	782
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	783 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	784 def addr_spec(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	785 return self[-1].addr_spec
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	786
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	787
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	788 class AngleAddr(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	789
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	790 token_type = 'angle-addr'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	791
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	792 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	793 def local_part(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	794 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	795 if x.token_type == 'addr-spec':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	796 return x.local_part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	797
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	798 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	799 def domain(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	800 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	801 if x.token_type == 'addr-spec':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	802 return x.domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	803
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	804 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	805 def route(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	806 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	807 if x.token_type == 'obs-route':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	808 return x.domains
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	809
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	810 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	811 def addr_spec(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	812 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	813 if x.token_type == 'addr-spec':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	814 return x.addr_spec
77165 a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	815 else:
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	816 return '<>'
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	817
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	818
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	819 class ObsRoute(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	820
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	821 token_type = 'obs-route'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	822
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	823 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	824 def domains(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	825 return [x.domain for x in self if x.token_type == 'domain']
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	826
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	827
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	828 class Mailbox(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	829
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	830 token_type = 'mailbox'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	831
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	832 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	833 def display_name(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	834 if self[0].token_type == 'name-addr':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	835 return self[0].display_name
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	836
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	837 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	838 def local_part(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	839 return self[0].local_part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	840
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	841 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	842 def domain(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	843 return self[0].domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	844
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	845 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	846 def route(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	847 if self[0].token_type == 'name-addr':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	848 return self[0].route
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	849
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	850 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	851 def addr_spec(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	852 return self[0].addr_spec
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	853
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	854
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	855 class InvalidMailbox(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	856
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	857 token_type = 'invalid-mailbox'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	858
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	859 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	860 def display_name(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	861 return None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	862
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	863 local_part = domain = route = addr_spec = display_name
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	864
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	865
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	866 class Domain(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	867
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	868 token_type = 'domain'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	869
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	870 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	871 def domain(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	872 return ''.join(super().value.split())
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	873
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	874
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	875 class DotAtom(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	876
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	877 token_type = 'dot-atom'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	878
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	879
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	880 class DotAtomText(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	881
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	882 token_type = 'dot-atom-text'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	883
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	884
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	885 class AddrSpec(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	886
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	887 token_type = 'addr-spec'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	888
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	889 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	890 def local_part(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	891 return self[0].local_part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	892
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	893 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	894 def domain(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	895 if len(self) < 3:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	896 return None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	897 return self[-1].domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	898
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	899 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	900 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	901 if len(self) < 3:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	902 return self[0].value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	903 return self[0].value.rstrip()+self[1].value+self[2].value.lstrip()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	904
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	905 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	906 def addr_spec(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	907 nameset = set(self.local_part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	908 if len(nameset) > len(nameset-DOT_ATOM_ENDS):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	909 lp = quote_string(self.local_part)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	910 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	911 lp = self.local_part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	912 if self.domain is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	913 return lp + '@' + self.domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	914 return lp
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	915
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	916
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	917 class ObsLocalPart(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	918
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	919 token_type = 'obs-local-part'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	920
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	921
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	922 class DisplayName(Phrase):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	923
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	924 token_type = 'display-name'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	925
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	926 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	927 def display_name(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	928 res = TokenList(self)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	929 if res[0].token_type == 'cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	930 res.pop(0)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	931 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	932 if res[0][0].token_type == 'cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	933 res[0] = TokenList(res[0][1:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	934 if res[-1].token_type == 'cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	935 res.pop()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	936 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	937 if res[-1][-1].token_type == 'cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	938 res[-1] = TokenList(res[-1][:-1])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	939 return res.value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	940
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	941 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	942 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	943 quote = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	944 if self.defects:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	945 quote = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	946 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	947 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	948 if x.token_type == 'quoted-string':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	949 quote = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	950 if quote:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	951 pre = post = ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	952 if self[0].token_type=='cfws' or self[0][0].token_type=='cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	953 pre = ' '
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	954 if self[-1].token_type=='cfws' or self[-1][-1].token_type=='cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	955 post = ' '
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	956 return pre+quote_string(self.display_name)+post
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	957 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	958 return super().value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	959
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	960
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	961 class LocalPart(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	962
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	963 token_type = 'local-part'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	964
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	965 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	966 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	967 if self[0].token_type == "quoted-string":
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	968 return self[0].quoted_value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	969 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	970 return self[0].value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	971
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	972 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	973 def local_part(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	974 # Strip whitespace from front, back, and around dots.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	975 res = [DOT]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	976 last = DOT
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	977 last_is_tl = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	978 for tok in self[0] + [DOT]:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	979 if tok.token_type == 'cfws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	980 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	981 if (last_is_tl and tok.token_type == 'dot' and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	982 last[-1].token_type == 'cfws'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	983 res[-1] = TokenList(last[:-1])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	984 is_tl = isinstance(tok, TokenList)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	985 if (is_tl and last.token_type == 'dot' and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	986 tok[0].token_type == 'cfws'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	987 res.append(TokenList(tok[1:]))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	988 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	989 res.append(tok)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	990 last = res[-1]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	991 last_is_tl = is_tl
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	992 res = TokenList(res[1:-1])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	993 return res.value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	994
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	995
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	996 class DomainLiteral(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	997
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	998 token_type = 'domain-literal'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	999
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1000 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1001 def domain(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1002 return ''.join(super().value.split())
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1003
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1004 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1005 def ip(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1006 for x in self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1007 if x.token_type == 'ptext':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1008 return x.value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1009
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1010
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1011 class MIMEVersion(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1012
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1013 token_type = 'mime-version'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1014 major = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1015 minor = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1016
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1017
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1018 class Parameter(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1019
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1020 token_type = 'parameter'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1021 sectioned = False
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1022 extended = False
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1023 charset = 'us-ascii'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1024
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1025 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1026 def section_number(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1027 # Because the first token, the attribute (name) eats CFWS, the second
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1028 # token is always the section if there is one.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1029 return self[1].number if self.sectioned else 0
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1030
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1031 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1032 def param_value(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1033 # This is part of the "handle quoted extended parameters" hack.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1034 for token in self:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1035 if token.token_type == 'value':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1036 return token.stripped_value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1037 if token.token_type == 'quoted-string':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1038 for token in token:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1039 if token.token_type == 'bare-quoted-string':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1040 for token in token:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1041 if token.token_type == 'value':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1042 return token.stripped_value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1043 return ''
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1044
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1045
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1046 class InvalidParameter(Parameter):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1047
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1048 token_type = 'invalid-parameter'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1049
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1050
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1051 class Attribute(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1052
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1053 token_type = 'attribute'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1054
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1055 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1056 def stripped_value(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1057 for token in self:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1058 if token.token_type.endswith('attrtext'):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1059 return token.value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1060
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1061 class Section(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1062
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1063 token_type = 'section'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1064 number = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1065
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1066
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1067 class Value(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1068
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1069 token_type = 'value'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1070
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1071 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1072 def stripped_value(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1073 token = self[0]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1074 if token.token_type == 'cfws':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1075 token = self[1]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1076 if token.token_type.endswith(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1077 ('quoted-string', 'attribute', 'extended-attribute')):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1078 return token.stripped_value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1079 return self.value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1080
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1081
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1082 class MimeParameters(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1083
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1084 token_type = 'mime-parameters'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1085
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1086 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1087 def params(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1088 # The RFC specifically states that the ordering of parameters is not
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1089 # guaranteed and may be reordered by the transport layer. So we have
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1090 # to assume the RFC 2231 pieces can come in any order. However, we
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1091 # output them in the order that we first see a given name, which gives
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1092 # us a stable __str__.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1093 params = OrderedDict()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1094 for token in self:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1095 if not token.token_type.endswith('parameter'):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1096 continue
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1097 if token[0].token_type != 'attribute':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1098 continue
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1099 name = token[0].value.strip()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1100 if name not in params:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1101 params[name] = []
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1102 params[name].append((token.section_number, token))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1103 for name, parts in params.items():
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1104 parts = sorted(parts)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1105 # XXX: there might be more recovery we could do here if, for
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1106 # example, this is really a case of a duplicate attribute name.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1107 value_parts = []
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1108 charset = parts[0][1].charset
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1109 for i, (section_number, param) in enumerate(parts):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1110 if section_number != i:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1111 param.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1112 "inconsistent multipart parameter numbering"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1113 value = param.param_value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1114 if param.extended:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1115 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1116 value = urllib.parse.unquote_to_bytes(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1117 except UnicodeEncodeError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1118 # source had surrogate escaped bytes. What we do now
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1119 # is a bit of an open question. I'm not sure this is
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1120 # the best choice, but it is what the old algorithm did
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1121 value = urllib.parse.unquote(value, encoding='latin-1')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1122 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1123 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1124 value = value.decode(charset, 'surrogateescape')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1125 except LookupError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1126 # XXX: there should really be a custom defect for
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1127 # unknown character set to make it easy to find,
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1128 # because otherwise unknown charset is a silent
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1129 # failure.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1130 value = value.decode('us-ascii', 'surrogateescape')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1131 if utils._has_surrogates(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1132 param.defects.append(errors.UndecodableBytesDefect())
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1133 value_parts.append(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1134 value = ''.join(value_parts)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1135 yield name, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1136
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1137 def __str__(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1138 params = []
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1139 for name, value in self.params:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1140 if value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1141 params.append('{}={}'.format(name, quote_string(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1142 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1143 params.append(name)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1144 params = '; '.join(params)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1145 return ' ' + params if params else ''
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1146
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1147
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1148 class ParameterizedHeaderValue(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1149
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1150 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1151 def params(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1152 for token in reversed(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1153 if token.token_type == 'mime-parameters':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1154 return token.params
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1155 return {}
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1156
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1157 @property
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1158 def parts(self):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1159 if self and self[-1].token_type == 'mime-parameters':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1160 # We don't want to start a new line if all of the params don't fit
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1161 # after the value, so unwrap the parameter list.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1162 return TokenList(self[:-1] + self[-1])
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1163 return TokenList(self).parts
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1164
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1165
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1166 class ContentType(ParameterizedHeaderValue):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1167
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1168 token_type = 'content-type'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1169 maintype = 'text'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1170 subtype = 'plain'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1171
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1172
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1173 class ContentDisposition(ParameterizedHeaderValue):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1174
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1175 token_type = 'content-disposition'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1176 content_disposition = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1177
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1178
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1179 class ContentTransferEncoding(TokenList):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1180
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1181 token_type = 'content-transfer-encoding'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1182 cte = '7bit'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1183
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1184
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1185 class HeaderLabel(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1186
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1187 token_type = 'header-label'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1188
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1189
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1190 class Header(TokenList):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1191
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1192 token_type = 'header'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1193
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1194 def _fold(self, folded):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1195 folded.append(str(self.pop(0)))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1196 folded.lastlen = len(folded.current[0])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1197 # The first line of the header is different from all others: we don't
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1198 # want to start a new object on a new line if it has any fold points in
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1199 # it that would allow part of it to be on the first header line.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1200 # Further, if the first fold point would fit on the new line, we want
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1201 # to do that, but if it doesn't we want to put it on the first line.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1202 # Folded supports this via the stickyspace attribute. If this
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1203 # attribute is not None, it does the special handling.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1204 folded.stickyspace = str(self.pop(0)) if self[0].token_type == 'cfws' else ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1205 rest = self.pop(0)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1206 if self:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1207 raise ValueError("Malformed Header token list")
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1208 rest._fold(folded)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1209
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1210
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1211 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1212 # Terminal classes and instances
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1213 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1214
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1215 class Terminal(str):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1216
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1217 def __new__(cls, value, token_type):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1218 self = super().__new__(cls, value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1219 self.token_type = token_type
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1220 self.defects = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1221 return self
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1222
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1223 def __repr__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1224 return "{}({})".format(self.__class__.__name__, super().__repr__())
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1225
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1226 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1227 def all_defects(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1228 return list(self.defects)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1229
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1230 def _pp(self, indent=''):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1231 return ["{}{}/{}({}){}".format(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1232 indent,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1233 self.__class__.__name__,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1234 self.token_type,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1235 super().__repr__(),
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1236 '' if not self.defects else ' {}'.format(self.defects),
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1237 )]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1238
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1239 def cte_encode(self, charset, policy):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1240 value = str(self)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1241 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1242 value.encode('us-ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1243 return value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1244 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1245 return _ew.encode(value, charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1246
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1247 def pop_trailing_ws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1248 # This terminates the recursion.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1249 return None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1250
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1251 def pop_leading_fws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1252 # This terminates the recursion.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1253 return None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1254
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1255 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1256 def comments(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1257 return []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1258
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1259 def has_leading_comment(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1260 return False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1261
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1262 def __getnewargs__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1263 return(str(self), self.token_type)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1264
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1265
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1266 class WhiteSpaceTerminal(Terminal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1267
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1268 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1269 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1270 return ' '
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1271
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1272 def startswith_fws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1273 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1274
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1275 has_fws = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1276
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1277
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1278 class ValueTerminal(Terminal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1279
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1280 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1281 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1282 return self
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1283
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1284 def startswith_fws(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1285 return False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1286
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1287 has_fws = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1288
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1289 def as_encoded_word(self, charset):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1290 return _ew.encode(str(self), charset)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1291
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1292
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1293 class EWWhiteSpaceTerminal(WhiteSpaceTerminal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1294
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1295 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1296 def value(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1297 return ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1298
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1299 @property
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1300 def encoded(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1301 return self[:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1302
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1303 def __str__(self):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1304 return ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1305
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1306 has_fws = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1307
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1308
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1309 # XXX these need to become classes and used as instances so
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1310 # that a program can't change them in a parse tree and screw
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1311 # up other parse trees. Maybe should have tests for that, too.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1312 DOT = ValueTerminal('.', 'dot')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1313 ListSeparator = ValueTerminal(',', 'list-separator')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1314 RouteComponentMarker = ValueTerminal('@', 'route-component-marker')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1315
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1316 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1317 # Parser
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1318 #
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1319
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1320 """Parse strings according to RFC822/2047/2822/5322 rules.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1321
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1322 This is a stateless parser. Each get_XXX function accepts a string and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1323 returns either a Terminal or a TokenList representing the RFC object named
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1324 by the method and a string containing the remaining unparsed characters
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1325 from the input. Thus a parser method consumes the next syntactic construct
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1326 of a given type and returns a token representing the construct plus the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1327 unparsed remainder of the input string.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1328
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1329 For example, if the first element of a structured header is a 'phrase',
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1330 then:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1331
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1332 phrase, value = get_phrase(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1333
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1334 returns the complete phrase from the start of the string value, plus any
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1335 characters left in the string after the phrase is removed.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1336
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1337 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1338
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1339 _wsp_splitter = re.compile(r'([{}]+)'.format(''.join(WSP))).split
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1340 _non_atom_end_matcher = re.compile(r"[^{}]+".format(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1341 ''.join(ATOM_ENDS).replace('\\','\\\\').replace(']','\]'))).match
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1342 _non_printable_finder = re.compile(r"[\x00-\x20\x7F]").findall
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1343 _non_token_end_matcher = re.compile(r"[^{}]+".format(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1344 ''.join(TOKEN_ENDS).replace('\\','\\\\').replace(']','\]'))).match
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1345 _non_attribute_end_matcher = re.compile(r"[^{}]+".format(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1346 ''.join(ATTRIBUTE_ENDS).replace('\\','\\\\').replace(']','\]'))).match
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1347 _non_extended_attribute_end_matcher = re.compile(r"[^{}]+".format(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1348 ''.join(EXTENDED_ATTRIBUTE_ENDS).replace(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	1349 '\\','\\\\').replace(']','\]'))).match
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1350
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1351 def _validate_xtext(xtext):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1352 """If input token contains ASCII non-printables, register a defect."""
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1353
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1354 non_printables = _non_printable_finder(xtext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1355 if non_printables:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1356 xtext.defects.append(errors.NonPrintableDefect(non_printables))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1357 if utils._has_surrogates(xtext):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1358 xtext.defects.append(errors.UndecodableBytesDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1359 "Non-ASCII characters found in header token"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1360
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1361 def _get_ptext_to_endchars(value, endchars):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1362 """Scan printables/quoted-pairs until endchars and return unquoted ptext.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1363
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1364 This function turns a run of qcontent, ccontent-without-comments, or
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1365 dtext-with-quoted-printables into a single string by unquoting any
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1366 quoted printables. It returns the string, the remaining value, and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1367 a flag that is True iff there were any quoted printables decoded.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1368
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1369 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1370 fragment, *remainder = _wsp_splitter(value, 1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1371 vchars = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1372 escape = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1373 had_qp = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1374 for pos in range(len(fragment)):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1375 if fragment[pos] == '\\':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1376 if escape:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1377 escape = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1378 had_qp = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1379 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1380 escape = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1381 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1382 if escape:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1383 escape = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1384 elif fragment[pos] in endchars:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1385 break
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1386 vchars.append(fragment[pos])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1387 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1388 pos = pos + 1
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1389 return ''.join(vchars), ''.join([fragment[pos:]] + remainder), had_qp
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1390
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1391 def _decode_ew_run(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1392 """ Decode a run of RFC2047 encoded words.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1393
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1394 _decode_ew_run(value) -> (text, value, defects)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1395
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1396 Scans the supplied value for a run of tokens that look like they are RFC
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1397 2047 encoded words, decodes those words into text according to RFC 2047
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1398 rules (whitespace between encoded words is discarded), and returns the text
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1399 and the remaining value (including any leading whitespace on the remaining
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1400 value), as well as a list of any defects encountered while decoding. The
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1401 input value may not have any leading whitespace.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1402
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1403 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1404 res = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1405 defects = []
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1406 last_ws = ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1407 while value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1408 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1409 tok, ws, value = _wsp_splitter(value, 1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1410 except ValueError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1411 tok, ws, value = value, '', ''
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1412 if not (tok.startswith('=?') and tok.endswith('?=')):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1413 return ''.join(res), last_ws + tok + ws + value, defects
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1414 text, charset, lang, new_defects = _ew.decode(tok)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1415 res.append(text)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1416 defects.extend(new_defects)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1417 last_ws = ws
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1418 return ''.join(res), last_ws, defects
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1419
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1420 def get_fws(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1421 """FWS = 1*WSP
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1422
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1423 This isn't the RFC definition. We're using fws to represent tokens where
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1424 folding can be done, but when we are parsing the unfolding has already
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1425 been done so we don't need to watch out for CRLF.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1426
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1427 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1428 newvalue = value.lstrip()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1429 fws = WhiteSpaceTerminal(value[:len(value)-len(newvalue)], 'fws')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1430 return fws, newvalue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1431
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1432 def get_encoded_word(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1433 """ encoded-word = "=?" charset "?" encoding "?" encoded-text "?="
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1434
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1435 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1436 ew = EncodedWord()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1437 if not value.startswith('=?'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1438 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1439 "expected encoded word but found {}".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1440 tok, *remainder = value[2:].split('?=', 1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1441 if tok == value[2:]:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1442 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1443 "expected encoded word but found {}".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1444 remstr = ''.join(remainder)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1445 if remstr[:2].isdigit():
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1446 rest, *remainder = remstr.split('?=', 1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1447 tok = tok + '?=' + rest
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1448 if len(tok.split()) > 1:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1449 ew.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1450 "whitespace inside encoded word"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1451 ew.cte = value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1452 value = ''.join(remainder)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1453 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1454 text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1455 except ValueError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1456 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1457 "encoded word format invalid: '{}'".format(ew.cte))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1458 ew.charset = charset
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1459 ew.lang = lang
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1460 ew.defects.extend(defects)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1461 while text:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1462 if text[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1463 token, text = get_fws(text)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1464 ew.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1465 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1466 chars, *remainder = _wsp_splitter(text, 1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1467 vtext = ValueTerminal(chars, 'vtext')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1468 _validate_xtext(vtext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1469 ew.append(vtext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1470 text = ''.join(remainder)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1471 return ew, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1472
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1473 def get_unstructured(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1474 """unstructured = (([FWS] vchar) WSP) / obs-unstruct
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1475 obs-unstruct = ((LF CR (obs-utext) LF CR)) / FWS)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1476 obs-utext = %d0 / obs-NO-WS-CTL / LF / CR
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1477
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1478 obs-NO-WS-CTL is control characters except WSP/CR/LF.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1479
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1480 So, basically, we have printable runs, plus control characters or nulls in
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1481 the obsolete syntax, separated by whitespace. Since RFC 2047 uses the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1482 obsolete syntax in its specification, but requires whitespace on either
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1483 side of the encoded words, I can see no reason to need to separate the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1484 non-printable-non-whitespace from the printable runs if they occur, so we
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1485 parse this into xtext tokens separated by WSP tokens.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1486
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1487 Because an 'unstructured' value must by definition constitute the entire
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1488 value, this 'get' routine does not return a remaining value, only the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1489 parsed TokenList.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1490
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1491 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1492 # XXX: but what about bare CR and LF? They might signal the start or
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1493 # end of an encoded word. YAGNI for now, since out current parsers
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1494 # will never send us strings with bard CR or LF.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1495
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1496 unstructured = UnstructuredTokenList()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1497 while value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1498 if value[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1499 token, value = get_fws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1500 unstructured.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1501 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1502 if value.startswith('=?'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1503 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1504 token, value = get_encoded_word(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1505 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1506 pass
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1507 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1508 have_ws = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1509 if len(unstructured) > 0:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1510 if unstructured[-1].token_type != 'fws':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1511 unstructured.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1512 "missing whitespace before encoded word"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1513 have_ws = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1514 if have_ws and len(unstructured) > 1:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1515 if unstructured[-2].token_type == 'encoded-word':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1516 unstructured[-1] = EWWhiteSpaceTerminal(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1517 unstructured[-1], 'fws')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1518 unstructured.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1519 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1520 tok, *remainder = _wsp_splitter(value, 1)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1521 vtext = ValueTerminal(tok, 'vtext')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1522 _validate_xtext(vtext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1523 unstructured.append(vtext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1524 value = ''.join(remainder)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1525 return unstructured
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1526
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1527 def get_qp_ctext(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1528 """ctext = <printable ascii except \ ( )>
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1529
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1530 This is not the RFC ctext, since we are handling nested comments in comment
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1531 and unquoting quoted-pairs here. We allow anything except the '()'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1532 characters, but if we find any ASCII other than the RFC defined printable
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1533 ASCII an NonPrintableDefect is added to the token's defects list. Since
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1534 quoted pairs are converted to their unquoted values, what is returned is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1535 a 'ptext' token. In this case it is a WhiteSpaceTerminal, so it's value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1536 is ' '.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1537
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1538 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1539 ptext, value, _ = _get_ptext_to_endchars(value, '()')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1540 ptext = WhiteSpaceTerminal(ptext, 'ptext')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1541 _validate_xtext(ptext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1542 return ptext, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1543
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1544 def get_qcontent(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1545 """qcontent = qtext / quoted-pair
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1546
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1547 We allow anything except the DQUOTE character, but if we find any ASCII
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1548 other than the RFC defined printable ASCII an NonPrintableDefect is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1549 added to the token's defects list. Any quoted pairs are converted to their
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1550 unquoted values, so what is returned is a 'ptext' token. In this case it
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1551 is a ValueTerminal.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1552
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1553 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1554 ptext, value, _ = _get_ptext_to_endchars(value, '"')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1555 ptext = ValueTerminal(ptext, 'ptext')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1556 _validate_xtext(ptext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1557 return ptext, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1558
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1559 def get_atext(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1560 """atext = <matches _atext_matcher>
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1561
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1562 We allow any non-ATOM_ENDS in atext, but add an InvalidATextDefect to
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1563 the token's defects list if we find non-atext characters.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1564 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1565 m = _non_atom_end_matcher(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1566 if not m:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1567 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1568 "expected atext but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1569 atext = m.group()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1570 value = value[len(atext):]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1571 atext = ValueTerminal(atext, 'atext')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1572 _validate_xtext(atext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1573 return atext, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1574
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1575 def get_bare_quoted_string(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1576 """bare-quoted-string = DQUOTE *([FWS] qcontent) [FWS] DQUOTE
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1577
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1578 A quoted-string without the leading or trailing white space. Its
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1579 value is the text between the quote marks, with whitespace
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1580 preserved and quoted pairs decoded.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1581 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1582 if value[0] != '"':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1583 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1584 "expected '\"' but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1585 bare_quoted_string = BareQuotedString()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1586 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1587 while value and value[0] != '"':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1588 if value[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1589 token, value = get_fws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1590 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1591 token, value = get_qcontent(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1592 bare_quoted_string.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1593 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1594 bare_quoted_string.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1595 "end of header inside quoted string"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1596 return bare_quoted_string, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1597 return bare_quoted_string, value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1598
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1599 def get_comment(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1600 """comment = "(" *([FWS] ccontent) [FWS] ")"
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1601 ccontent = ctext / quoted-pair / comment
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1602
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1603 We handle nested comments here, and quoted-pair in our qp-ctext routine.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1604 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1605 if value and value[0] != '(':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1606 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1607 "expected '(' but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1608 comment = Comment()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1609 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1610 while value and value[0] != ")":
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1611 if value[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1612 token, value = get_fws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1613 elif value[0] == '(':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1614 token, value = get_comment(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1615 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1616 token, value = get_qp_ctext(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1617 comment.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1618 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1619 comment.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1620 "end of header inside comment"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1621 return comment, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1622 return comment, value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1623
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1624 def get_cfws(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1625 """CFWS = (1*([FWS] comment) [FWS]) / FWS
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1626
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1627 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1628 cfws = CFWSList()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1629 while value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1630 if value[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1631 token, value = get_fws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1632 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1633 token, value = get_comment(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1634 cfws.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1635 return cfws, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1636
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1637 def get_quoted_string(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1638 """quoted-string = [CFWS] <bare-quoted-string> [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1639
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1640 'bare-quoted-string' is an intermediate class defined by this
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1641 parser and not by the RFC grammar. It is the quoted string
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1642 without any attached CFWS.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1643 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1644 quoted_string = QuotedString()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1645 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1646 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1647 quoted_string.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1648 token, value = get_bare_quoted_string(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1649 quoted_string.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1650 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1651 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1652 quoted_string.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1653 return quoted_string, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1654
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1655 def get_atom(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1656 """atom = [CFWS] 1*atext [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1657
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1658 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1659 atom = Atom()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1660 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1661 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1662 atom.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1663 if value and value[0] in ATOM_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1664 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1665 "expected atom but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1666 token, value = get_atext(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1667 atom.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1668 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1669 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1670 atom.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1671 return atom, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1672
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1673 def get_dot_atom_text(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1674 """ dot-text = 1atext ("." 1*atext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1675
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1676 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1677 dot_atom_text = DotAtomText()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1678 if not value or value[0] in ATOM_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1679 raise errors.HeaderParseError("expected atom at a start of "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1680 "dot-atom-text but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1681 while value and value[0] not in ATOM_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1682 token, value = get_atext(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1683 dot_atom_text.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1684 if value and value[0] == '.':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1685 dot_atom_text.append(DOT)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1686 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1687 if dot_atom_text[-1] is DOT:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1688 raise errors.HeaderParseError("expected atom at end of dot-atom-text "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1689 "but found '{}'".format('.'+value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1690 return dot_atom_text, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1691
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1692 def get_dot_atom(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1693 """ dot-atom = [CFWS] dot-atom-text [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1694
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1695 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1696 dot_atom = DotAtom()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1697 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1698 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1699 dot_atom.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1700 token, value = get_dot_atom_text(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1701 dot_atom.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1702 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1703 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1704 dot_atom.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1705 return dot_atom, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1706
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1707 def get_word(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1708 """word = atom / quoted-string
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1709
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1710 Either atom or quoted-string may start with CFWS. We have to peel off this
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1711 CFWS first to determine which type of word to parse. Afterward we splice
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1712 the leading CFWS, if any, into the parsed sub-token.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1713
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1714 If neither an atom or a quoted-string is found before the next special, a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1715 HeaderParseError is raised.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1716
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1717 The token returned is either an Atom or a QuotedString, as appropriate.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1718 This means the 'word' level of the formal grammar is not represented in the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1719 parse tree; this is because having that extra layer when manipulating the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1720 parse tree is more confusing than it is helpful.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1721
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1722 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1723 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1724 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1725 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1726 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1727 if value[0]=='"':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1728 token, value = get_quoted_string(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1729 elif value[0] in SPECIALS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1730 raise errors.HeaderParseError("Expected 'atom' or 'quoted-string' "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1731 "but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1732 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1733 token, value = get_atom(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1734 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1735 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1736 return token, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1737
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1738 def get_phrase(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1739 """ phrase = 1*word / obs-phrase
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1740 obs-phrase = word *(word / "." / CFWS)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1741
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1742 This means a phrase can be a sequence of words, periods, and CFWS in any
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1743 order as long as it starts with at least one word. If anything other than
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1744 words is detected, an ObsoleteHeaderDefect is added to the token's defect
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1745 list. We also accept a phrase that starts with CFWS followed by a dot;
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1746 this is registered as an InvalidHeaderDefect, since it is not supported by
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1747 even the obsolete grammar.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1748
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1749 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1750 phrase = Phrase()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1751 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1752 token, value = get_word(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1753 phrase.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1754 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1755 phrase.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1756 "phrase does not start with word"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1757 while value and value[0] not in PHRASE_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1758 if value[0]=='.':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1759 phrase.append(DOT)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1760 phrase.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1761 "period in 'phrase'"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1762 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1763 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1764 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1765 token, value = get_word(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1766 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1767 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1768 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1769 phrase.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1770 "comment found without atom"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1771 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1772 raise
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1773 phrase.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1774 return phrase, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1775
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1776 def get_local_part(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1777 """ local-part = dot-atom / quoted-string / obs-local-part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1778
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1779 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1780 local_part = LocalPart()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1781 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1782 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1783 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1784 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1785 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1786 "expected local-part but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1787 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1788 token, value = get_dot_atom(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1789 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1790 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1791 token, value = get_word(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1792 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1793 if value[0] != '\\' and value[0] in PHRASE_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1794 raise
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1795 token = TokenList()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1796 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1797 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1798 local_part.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1799 if value and (value[0]=='\\' or value[0] not in PHRASE_ENDS):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1800 obs_local_part, value = get_obs_local_part(str(local_part) + value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1801 if obs_local_part.token_type == 'invalid-obs-local-part':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1802 local_part.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1803 "local-part is not dot-atom, quoted-string, or obs-local-part"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1804 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1805 local_part.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1806 "local-part is not a dot-atom (contains CFWS)"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1807 local_part[0] = obs_local_part
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1808 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1809 local_part.value.encode('ascii')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1810 except UnicodeEncodeError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1811 local_part.defects.append(errors.NonASCIILocalPartDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1812 "local-part contains non-ASCII characters)"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1813 return local_part, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1814
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1815 def get_obs_local_part(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1816 """ obs-local-part = word *("." word)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1817 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1818 obs_local_part = ObsLocalPart()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1819 last_non_ws_was_dot = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1820 while value and (value[0]=='\\' or value[0] not in PHRASE_ENDS):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1821 if value[0] == '.':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1822 if last_non_ws_was_dot:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1823 obs_local_part.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1824 "invalid repeated '.'"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1825 obs_local_part.append(DOT)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1826 last_non_ws_was_dot = True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1827 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1828 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1829 elif value[0]=='\\':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1830 obs_local_part.append(ValueTerminal(value[0],
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1831 'misplaced-special'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1832 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1833 obs_local_part.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1834 "'\\' character outside of quoted-string/ccontent"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1835 last_non_ws_was_dot = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1836 continue
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1837 if obs_local_part and obs_local_part[-1].token_type != 'dot':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1838 obs_local_part.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1839 "missing '.' between words"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1840 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1841 token, value = get_word(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1842 last_non_ws_was_dot = False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1843 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1844 if value[0] not in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1845 raise
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1846 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1847 obs_local_part.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1848 if (obs_local_part[0].token_type == 'dot' or
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1849 obs_local_part[0].token_type=='cfws' and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1850 obs_local_part[1].token_type=='dot'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1851 obs_local_part.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1852 "Invalid leading '.' in local part"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1853 if (obs_local_part[-1].token_type == 'dot' or
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1854 obs_local_part[-1].token_type=='cfws' and
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1855 obs_local_part[-2].token_type=='dot'):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1856 obs_local_part.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1857 "Invalid trailing '.' in local part"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1858 if obs_local_part.defects:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1859 obs_local_part.token_type = 'invalid-obs-local-part'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1860 return obs_local_part, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1861
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1862 def get_dtext(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1863 """ dtext = <printable ascii except \ [ ]> / obs-dtext
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1864 obs-dtext = obs-NO-WS-CTL / quoted-pair
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1865
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1866 We allow anything except the excluded characters, but but if we find any
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1867 ASCII other than the RFC defined printable ASCII an NonPrintableDefect is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1868 added to the token's defects list. Quoted pairs are converted to their
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1869 unquoted values, so what is returned is a ptext token, in this case a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1870 ValueTerminal. If there were quoted-printables, an ObsoleteHeaderDefect is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1871 added to the returned token's defect list.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1872
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1873 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1874 ptext, value, had_qp = _get_ptext_to_endchars(value, '[]')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1875 ptext = ValueTerminal(ptext, 'ptext')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1876 if had_qp:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1877 ptext.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1878 "quoted printable found in domain-literal"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1879 _validate_xtext(ptext)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1880 return ptext, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1881
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1882 def _check_for_early_dl_end(value, domain_literal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1883 if value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1884 return False
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1885 domain_literal.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1886 "end of input inside domain-literal"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1887 domain_literal.append(ValueTerminal(']', 'domain-literal-end'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1888 return True
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1889
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1890 def get_domain_literal(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1891 """ domain-literal = [CFWS] "[" *([FWS] dtext) [FWS] "]" [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1892
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1893 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1894 domain_literal = DomainLiteral()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1895 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1896 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1897 domain_literal.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1898 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1899 raise errors.HeaderParseError("expected domain-literal")
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1900 if value[0] != '[':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1901 raise errors.HeaderParseError("expected '[' at start of domain-literal "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1902 "but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1903 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1904 if _check_for_early_dl_end(value, domain_literal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1905 return domain_literal, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1906 domain_literal.append(ValueTerminal('[', 'domain-literal-start'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1907 if value[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1908 token, value = get_fws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1909 domain_literal.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1910 token, value = get_dtext(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1911 domain_literal.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1912 if _check_for_early_dl_end(value, domain_literal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1913 return domain_literal, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1914 if value[0] in WSP:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1915 token, value = get_fws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1916 domain_literal.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1917 if _check_for_early_dl_end(value, domain_literal):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1918 return domain_literal, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1919 if value[0] != ']':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1920 raise errors.HeaderParseError("expected ']' at end of domain-literal "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1921 "but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1922 domain_literal.append(ValueTerminal(']', 'domain-literal-end'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1923 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1924 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1925 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1926 domain_literal.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1927 return domain_literal, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1928
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1929 def get_domain(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1930 """ domain = dot-atom / domain-literal / obs-domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1931 obs-domain = atom *("." atom))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1932
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1933 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1934 domain = Domain()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1935 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1936 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1937 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1938 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1939 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1940 "expected domain but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1941 if value[0] == '[':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1942 token, value = get_domain_literal(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1943 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1944 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1945 domain.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1946 return domain, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1947 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1948 token, value = get_dot_atom(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1949 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1950 token, value = get_atom(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1951 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1952 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1953 domain.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1954 if value and value[0] == '.':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1955 domain.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1956 "domain is not a dot-atom (contains CFWS)"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1957 if domain[0].token_type == 'dot-atom':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1958 domain[:] = domain[0]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1959 while value and value[0] == '.':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1960 domain.append(DOT)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1961 token, value = get_atom(value[1:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1962 domain.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1963 return domain, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1964
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1965 def get_addr_spec(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1966 """ addr-spec = local-part "@" domain
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1967
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1968 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1969 addr_spec = AddrSpec()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1970 token, value = get_local_part(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1971 addr_spec.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1972 if not value or value[0] != '@':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1973 addr_spec.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1974 "add-spec local part with no domain"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1975 return addr_spec, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1976 addr_spec.append(ValueTerminal('@', 'address-at-symbol'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1977 token, value = get_domain(value[1:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1978 addr_spec.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1979 return addr_spec, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1980
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1981 def get_obs_route(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1982 """ obs-route = obs-domain-list ":"
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1983 obs-domain-list = (CFWS / ",") "@" domain ("," [CFWS] ["@" domain])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1984
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1985 Returns an obs-route token with the appropriate sub-tokens (that is,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1986 there is no obs-domain-list in the parse tree).
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1987 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1988 obs_route = ObsRoute()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1989 while value and (value[0]==',' or value[0] in CFWS_LEADER):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1990 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1991 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1992 obs_route.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1993 elif value[0] == ',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1994 obs_route.append(ListSeparator)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1995 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1996 if not value or value[0] != '@':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1997 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1998 "expected obs-route domain but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	1999 obs_route.append(RouteComponentMarker)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2000 token, value = get_domain(value[1:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2001 obs_route.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2002 while value and value[0]==',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2003 obs_route.append(ListSeparator)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2004 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2005 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2006 break
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2007 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2008 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2009 obs_route.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2010 if value[0] == '@':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2011 obs_route.append(RouteComponentMarker)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2012 token, value = get_domain(value[1:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2013 obs_route.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2014 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2015 raise errors.HeaderParseError("end of header while parsing obs-route")
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2016 if value[0] != ':':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2017 raise errors.HeaderParseError( "expected ':' marking end of "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2018 "obs-route but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2019 obs_route.append(ValueTerminal(':', 'end-of-obs-route-marker'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2020 return obs_route, value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2021
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2022 def get_angle_addr(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2023 """ angle-addr = [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2024 obs-angle-addr = [CFWS] "<" obs-route addr-spec ">" [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2025
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2026 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2027 angle_addr = AngleAddr()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2028 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2029 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2030 angle_addr.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2031 if not value or value[0] != '<':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2032 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2033 "expected angle-addr but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2034 angle_addr.append(ValueTerminal('<', 'angle-addr-start'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2035 value = value[1:]
77165 a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2036 # Although it is not legal per RFC5322, SMTP uses '<>' in certain
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2037 # circumstances.
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2038 if value[0] == '>':
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2039 angle_addr.append(ValueTerminal('>', 'angle-addr-end'))
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2040 angle_addr.defects.append(errors.InvalidHeaderDefect(
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2041 "null addr-spec in angle-addr"))
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2042 value = value[1:]
a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2043 return angle_addr, value
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2044 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2045 token, value = get_addr_spec(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2046 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2047 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2048 token, value = get_obs_route(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2049 angle_addr.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2050 "obsolete route specification in angle-addr"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2051 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2052 raise errors.HeaderParseError(
77165 a66bf07fca4d Recognize '<>' as a special case of an angle-addr in header_value_parser. R David Murray <rdmurray@bitdance.com> parents: 77148 diff changeset	2053 "expected addr-spec or obs-route but found '{}'".format(value))
77148 0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2054 angle_addr.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2055 token, value = get_addr_spec(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2056 angle_addr.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2057 if value and value[0] == '>':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2058 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2059 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2060 angle_addr.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2061 "missing trailing '>' on angle-addr"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2062 angle_addr.append(ValueTerminal('>', 'angle-addr-end'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2063 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2064 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2065 angle_addr.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2066 return angle_addr, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2067
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2068 def get_display_name(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2069 """ display-name = phrase
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2070
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2071 Because this is simply a name-rule, we don't return a display-name
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2072 token containing a phrase, but rather a display-name token with
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2073 the content of the phrase.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2074
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2075 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2076 display_name = DisplayName()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2077 token, value = get_phrase(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2078 display_name.extend(token[:])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2079 display_name.defects = token.defects[:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2080 return display_name, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2081
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2082
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2083 def get_name_addr(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2084 """ name-addr = [display-name] angle-addr
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2085
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2086 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2087 name_addr = NameAddr()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2088 # Both the optional display name and the angle-addr can start with cfws.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2089 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2090 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2091 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2092 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2093 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2094 "expected name-addr but found '{}'".format(leader))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2095 if value[0] != '<':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2096 if value[0] in PHRASE_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2097 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2098 "expected name-addr but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2099 token, value = get_display_name(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2100 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2101 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2102 "expected name-addr but found '{}'".format(token))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2103 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2104 token[0][:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2105 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2106 name_addr.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2107 token, value = get_angle_addr(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2108 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2109 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2110 name_addr.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2111 return name_addr, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2112
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2113 def get_mailbox(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2114 """ mailbox = name-addr / addr-spec
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2115
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2116 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2117 # The only way to figure out if we are dealing with a name-addr or an
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2118 # addr-spec is to try parsing each one.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2119 mailbox = Mailbox()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2120 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2121 token, value = get_name_addr(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2122 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2123 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2124 token, value = get_addr_spec(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2125 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2126 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2127 "expected mailbox but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2128 if any(isinstance(x, errors.InvalidHeaderDefect)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2129 for x in token.all_defects):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2130 mailbox.token_type = 'invalid-mailbox'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2131 mailbox.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2132 return mailbox, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2133
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2134 def get_invalid_mailbox(value, endchars):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2135 """ Read everything up to one of the chars in endchars.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2136
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2137 This is outside the formal grammar. The InvalidMailbox TokenList that is
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2138 returned acts like a Mailbox, but the data attributes are None.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2139
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2140 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2141 invalid_mailbox = InvalidMailbox()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2142 while value and value[0] not in endchars:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2143 if value[0] in PHRASE_ENDS:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2144 invalid_mailbox.append(ValueTerminal(value[0],
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2145 'misplaced-special'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2146 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2147 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2148 token, value = get_phrase(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2149 invalid_mailbox.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2150 return invalid_mailbox, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2151
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2152 def get_mailbox_list(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2153 """ mailbox-list = (mailbox *("," mailbox)) / obs-mbox-list
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2154 obs-mbox-list = ([CFWS] ",") mailbox ("," [mailbox / CFWS])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2155
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2156 For this routine we go outside the formal grammar in order to improve error
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2157 handling. We recognize the end of the mailbox list only at the end of the
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2158 value or at a ';' (the group terminator). This is so that we can turn
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2159 invalid mailboxes into InvalidMailbox tokens and continue parsing any
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2160 remaining valid mailboxes. We also allow all mailbox entries to be null,
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2161 and this condition is handled appropriately at a higher level.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2162
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2163 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2164 mailbox_list = MailboxList()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2165 while value and value[0] != ';':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2166 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2167 token, value = get_mailbox(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2168 mailbox_list.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2169 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2170 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2171 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2172 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2173 if not value or value[0] in ',;':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2174 mailbox_list.append(leader)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2175 mailbox_list.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2176 "empty element in mailbox-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2177 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2178 token, value = get_invalid_mailbox(value, ',;')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2179 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2180 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2181 mailbox_list.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2182 mailbox_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2183 "invalid mailbox in mailbox-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2184 elif value[0] == ',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2185 mailbox_list.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2186 "empty element in mailbox-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2187 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2188 token, value = get_invalid_mailbox(value, ',;')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2189 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2190 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2191 mailbox_list.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2192 mailbox_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2193 "invalid mailbox in mailbox-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2194 if value and value[0] not in ',;':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2195 # Crap after mailbox; treat it as an invalid mailbox.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2196 # The mailbox info will still be available.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2197 mailbox = mailbox_list[-1]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2198 mailbox.token_type = 'invalid-mailbox'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2199 token, value = get_invalid_mailbox(value, ',;')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2200 mailbox.extend(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2201 mailbox_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2202 "invalid mailbox in mailbox-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2203 if value and value[0] == ',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2204 mailbox_list.append(ListSeparator)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2205 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2206 return mailbox_list, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2207
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2208
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2209 def get_group_list(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2210 """ group-list = mailbox-list / CFWS / obs-group-list
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2211 obs-group-list = 1*([CFWS] ",") [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2212
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2213 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2214 group_list = GroupList()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2215 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2216 group_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2217 "end of header before group-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2218 return group_list, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2219 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2220 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2221 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2222 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2223 # This should never happen in email parsing, since CFWS-only is a
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2224 # legal alternative to group-list in a group, which is the only
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2225 # place group-list appears.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2226 group_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2227 "end of header in group-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2228 group_list.append(leader)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2229 return group_list, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2230 if value[0] == ';':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2231 group_list.append(leader)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2232 return group_list, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2233 token, value = get_mailbox_list(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2234 if len(token.all_mailboxes)==0:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2235 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2236 group_list.append(leader)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2237 group_list.extend(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2238 group_list.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2239 "group-list with empty entries"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2240 return group_list, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2241 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2242 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2243 group_list.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2244 return group_list, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2245
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2246 def get_group(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2247 """ group = display-name ":" [group-list] ";" [CFWS]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2248
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2249 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2250 group = Group()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2251 token, value = get_display_name(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2252 if not value or value[0] != ':':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2253 raise errors.HeaderParseError("expected ':' at end of group "
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2254 "display name but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2255 group.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2256 group.append(ValueTerminal(':', 'group-display-name-terminator'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2257 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2258 if value and value[0] == ';':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2259 group.append(ValueTerminal(';', 'group-terminator'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2260 return group, value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2261 token, value = get_group_list(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2262 group.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2263 if not value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2264 group.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2265 "end of header in group"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2266 if value[0] != ';':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2267 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2268 "expected ';' at end of group but found {}".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2269 group.append(ValueTerminal(';', 'group-terminator'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2270 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2271 if value and value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2272 token, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2273 group.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2274 return group, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2275
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2276 def get_address(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2277 """ address = mailbox / group
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2278
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2279 Note that counter-intuitively, an address can be either a single address or
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2280 a list of addresses (a group). This is why the returned Address object has
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2281 a 'mailboxes' attribute which treats a single address as a list of length
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2282 one. When you need to differentiate between to two cases, extract the single
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2283 element, which is either a mailbox or a group token.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2284
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2285 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2286 # The formal grammar isn't very helpful when parsing an address. mailbox
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2287 # and group, especially when allowing for obsolete forms, start off very
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2288 # similarly. It is only when you reach one of @, <, or : that you know
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2289 # what you've got. So, we try each one in turn, starting with the more
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2290 # likely of the two. We could perhaps make this more efficient by looking
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2291 # for a phrase and then branching based on the next character, but that
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2292 # would be a premature optimization.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2293 address = Address()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2294 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2295 token, value = get_group(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2296 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2297 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2298 token, value = get_mailbox(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2299 except errors.HeaderParseError:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2300 raise errors.HeaderParseError(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2301 "expected address but found '{}'".format(value))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2302 address.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2303 return address, value
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2304
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2305 def get_address_list(value):
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2306 """ address_list = (address *("," address)) / obs-addr-list
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2307 obs-addr-list = ([CFWS] ",") address ("," [address / CFWS])
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2308
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2309 We depart from the formal grammar here by continuing to parse until the end
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2310 of the input, assuming the input to be entirely composed of an
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2311 address-list. This is always true in email parsing, and allows us
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2312 to skip invalid addresses to parse additional valid ones.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2313
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2314 """
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2315 address_list = AddressList()
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2316 while value:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2317 try:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2318 token, value = get_address(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2319 address_list.append(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2320 except errors.HeaderParseError as err:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2321 leader = None
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2322 if value[0] in CFWS_LEADER:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2323 leader, value = get_cfws(value)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2324 if not value or value[0] == ',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2325 address_list.append(leader)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2326 address_list.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2327 "address-list entry with no content"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2328 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2329 token, value = get_invalid_mailbox(value, ',')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2330 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2331 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2332 address_list.append(Address([token]))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2333 address_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2334 "invalid address in address-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2335 elif value[0] == ',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2336 address_list.defects.append(errors.ObsoleteHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2337 "empty element in address-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2338 else:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2339 token, value = get_invalid_mailbox(value, ',')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2340 if leader is not None:
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2341 token[:0] = [leader]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2342 address_list.append(Address([token]))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2343 address_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2344 "invalid address in address-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2345 if value and value[0] != ',':
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2346 # Crap after address; treat it as an invalid mailbox.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2347 # The mailbox info will still be available.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2348 mailbox = address_list[-1][0]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2349 mailbox.token_type = 'invalid-mailbox'
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2350 token, value = get_invalid_mailbox(value, ',')
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2351 mailbox.extend(token)
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2352 address_list.defects.append(errors.InvalidHeaderDefect(
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2353 "invalid address in address-list"))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2354 if value: # Must be a , at this point.
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2355 address_list.append(ValueTerminal(',', 'list-separator'))
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2356 value = value[1:]
0189b9d2d6bc #12586: add provisional email policy with new header parsing and folding. R David Murray <rdmurray@bitdance.com> parents: diff changeset	2357 return address_list, value
77681 0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2358
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2359 #
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2360 # XXX: As I begin to add additional header parsers, I'm realizing we probably
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2361 # have two level of parser routines: the get_XXX methods that get a token in
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2362 # the grammar, and parse_XXX methods that parse an entire field value. So
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2363 # get_address_list above should really be a parse_ method, as probably should
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2364 # be get_unstructured.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2365 #
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2366
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2367 def parse_mime_version(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2368 """ mime-version = [CFWS] 1digit [CFWS] "." [CFWS] 1digit [CFWS]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2369
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2370 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2371 # The [CFWS] is implicit in the RFC 2045 BNF.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2372 # XXX: This routine is a bit verbose, should factor out a get_int method.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2373 mime_version = MIMEVersion()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2374 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2375 mime_version.defects.append(errors.HeaderMissingRequiredValue(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2376 "Missing MIME version number (eg: 1.0)"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2377 return mime_version
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2378 if value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2379 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2380 mime_version.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2381 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2382 mime_version.defects.append(errors.HeaderMissingRequiredValue(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2383 "Expected MIME version number but found only CFWS"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2384 digits = ''
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2385 while value and value[0] != '.' and value[0] not in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2386 digits += value[0]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2387 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2388 if not digits.isdigit():
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2389 mime_version.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2390 "Expected MIME major version number but found {!r}".format(digits)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2391 mime_version.append(ValueTerminal(digits, 'xtext'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2392 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2393 mime_version.major = int(digits)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2394 mime_version.append(ValueTerminal(digits, 'digits'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2395 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2396 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2397 mime_version.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2398 if not value or value[0] != '.':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2399 if mime_version.major is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2400 mime_version.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2401 "Incomplete MIME version; found only major number"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2402 if value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2403 mime_version.append(ValueTerminal(value, 'xtext'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2404 return mime_version
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2405 mime_version.append(ValueTerminal('.', 'version-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2406 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2407 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2408 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2409 mime_version.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2410 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2411 if mime_version.major is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2412 mime_version.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2413 "Incomplete MIME version; found only major number"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2414 return mime_version
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2415 digits = ''
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2416 while value and value[0] not in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2417 digits += value[0]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2418 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2419 if not digits.isdigit():
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2420 mime_version.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2421 "Expected MIME minor version number but found {!r}".format(digits)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2422 mime_version.append(ValueTerminal(digits, 'xtext'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2423 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2424 mime_version.minor = int(digits)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2425 mime_version.append(ValueTerminal(digits, 'digits'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2426 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2427 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2428 mime_version.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2429 if value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2430 mime_version.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2431 "Excess non-CFWS text after MIME version"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2432 mime_version.append(ValueTerminal(value, 'xtext'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2433 return mime_version
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2434
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2435 def get_invalid_parameter(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2436 """ Read everything up to the next ';'.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2437
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2438 This is outside the formal grammar. The InvalidParameter TokenList that is
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2439 returned acts like a Parameter, but the data attributes are None.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2440
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2441 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2442 invalid_parameter = InvalidParameter()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2443 while value and value[0] != ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2444 if value[0] in PHRASE_ENDS:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2445 invalid_parameter.append(ValueTerminal(value[0],
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2446 'misplaced-special'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2447 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2448 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2449 token, value = get_phrase(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2450 invalid_parameter.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2451 return invalid_parameter, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2452
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2453 def get_ttext(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2454 """ttext = <matches _ttext_matcher>
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2455
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2456 We allow any non-TOKEN_ENDS in ttext, but add defects to the token's
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2457 defects list if we find non-ttext characters. We also register defects for
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2458 any non-printables even though the RFC doesn't exclude all of them,
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2459 because we follow the spirit of RFC 5322.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2460
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2461 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2462 m = _non_token_end_matcher(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2463 if not m:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2464 raise errors.HeaderParseError(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2465 "expected ttext but found '{}'".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2466 ttext = m.group()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2467 value = value[len(ttext):]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2468 ttext = ValueTerminal(ttext, 'ttext')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2469 _validate_xtext(ttext)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2470 return ttext, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2471
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2472 def get_token(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2473 """token = [CFWS] 1*ttext [CFWS]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2474
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2475 The RFC equivalent of ttext is any US-ASCII chars except space, ctls, or
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2476 tspecials. We also exclude tabs even though the RFC doesn't.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2477
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2478 The RFC implies the CFWS but is not explicit about it in the BNF.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2479
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2480 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2481 mtoken = Token()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2482 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2483 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2484 mtoken.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2485 if value and value[0] in TOKEN_ENDS:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2486 raise errors.HeaderParseError(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2487 "expected token but found '{}'".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2488 token, value = get_ttext(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2489 mtoken.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2490 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2491 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2492 mtoken.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2493 return mtoken, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2494
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2495 def get_attrtext(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2496 """attrtext = 1*(any non-ATTRIBUTE_ENDS character)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2497
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2498 We allow any non-ATTRIBUTE_ENDS in attrtext, but add defects to the
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2499 token's defects list if we find non-attrtext characters. We also register
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2500 defects for any non-printables even though the RFC doesn't exclude all of
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2501 them, because we follow the spirit of RFC 5322.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2502
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2503 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2504 m = _non_attribute_end_matcher(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2505 if not m:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2506 raise errors.HeaderParseError(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2507 "expected attrtext but found {!r}".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2508 attrtext = m.group()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2509 value = value[len(attrtext):]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2510 attrtext = ValueTerminal(attrtext, 'attrtext')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2511 _validate_xtext(attrtext)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2512 return attrtext, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2513
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2514 def get_attribute(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2515 """ [CFWS] 1*attrtext [CFWS]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2516
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2517 This version of the BNF makes the CFWS explicit, and as usual we use a
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2518 value terminal for the actual run of characters. The RFC equivalent of
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2519 attrtext is the token characters, with the subtraction of '*', "'", and '%'.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2520 We include tab in the excluded set just as we do for token.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2521
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2522 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2523 attribute = Attribute()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2524 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2525 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2526 attribute.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2527 if value and value[0] in ATTRIBUTE_ENDS:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2528 raise errors.HeaderParseError(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2529 "expected token but found '{}'".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2530 token, value = get_attrtext(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2531 attribute.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2532 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2533 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2534 attribute.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2535 return attribute, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2536
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2537 def get_extended_attrtext(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2538 """attrtext = 1*(any non-ATTRIBUTE_ENDS character plus '%')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2539
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2540 This is a special parsing routine so that we get a value that
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2541 includes % escapes as a single string (which we decode as a single
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2542 string later).
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2543
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2544 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2545 m = _non_extended_attribute_end_matcher(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2546 if not m:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2547 raise errors.HeaderParseError(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2548 "expected extended attrtext but found {!r}".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2549 attrtext = m.group()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2550 value = value[len(attrtext):]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2551 attrtext = ValueTerminal(attrtext, 'extended-attrtext')
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2552 _validate_xtext(attrtext)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2553 return attrtext, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2554
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2555 def get_extended_attribute(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2556 """ [CFWS] 1*extended_attrtext [CFWS]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2557
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2558 This is like the non-extended version except we allow % characters, so that
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2559 we can pick up an encoded value as a single string.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2560
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2561 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2562 # XXX: should we have an ExtendedAttribute TokenList?
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2563 attribute = Attribute()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2564 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2565 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2566 attribute.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2567 if value and value[0] in EXTENDED_ATTRIBUTE_ENDS:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2568 raise errors.HeaderParseError(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2569 "expected token but found '{}'".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2570 token, value = get_extended_attrtext(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2571 attribute.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2572 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2573 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2574 attribute.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2575 return attribute, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2576
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2577 def get_section(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2578 """ '*' digits
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2579
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2580 The formal BNF is more complicated because leading 0s are not allowed. We
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2581 check for that and add a defect. We also assume no CFWS is allowed between
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2582 the '*' and the digits, though the RFC is not crystal clear on that.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2583 The caller should already have dealt with leading CFWS.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2584
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2585 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2586 section = Section()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2587 if not value or value[0] != '*':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2588 raise errors.HeaderParseError("Expected section but found {}".format(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2589 value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2590 section.append(ValueTerminal('*', 'section-marker'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2591 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2592 if not value or not value[0].isdigit():
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2593 raise errors.HeaderParseError("Expected section number but "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2594 "found {}".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2595 digits = ''
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2596 while value and value[0].isdigit():
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2597 digits += value[0]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2598 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2599 if digits[0] == '0' and digits != '0':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2600 section.defects.append(errors.InvalidHeaderError("section number"
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2601 "has an invalid leading 0"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2602 section.number = int(digits)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2603 section.append(ValueTerminal(digits, 'digits'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2604 return section, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2605
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2606
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2607 def get_value(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2608 """ quoted-string / attribute
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2609
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2610 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2611 v = Value()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2612 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2613 raise errors.HeaderParseError("Expected value but found end of string")
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2614 leader = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2615 if value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2616 leader, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2617 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2618 raise errors.HeaderParseError("Expected value but found "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2619 "only {}".format(leader))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2620 if value[0] == '"':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2621 token, value = get_quoted_string(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2622 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2623 token, value = get_extended_attribute(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2624 if leader is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2625 token[:0] = [leader]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2626 v.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2627 return v, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2628
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2629 def get_parameter(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2630 """ attribute [section] ["*"] [CFWS] "=" value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2631
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2632 The CFWS is implied by the RFC but not made explicit in the BNF. This
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2633 simplified form of the BNF from the RFC is made to conform with the RFC BNF
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2634 through some extra checks. We do it this way because it makes both error
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2635 recovery and working with the resulting parse tree easier.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2636 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2637 # It is possible CFWS would also be implicitly allowed between the section
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2638 # and the 'extended-attribute' marker (the '*') , but we've never seen that
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2639 # in the wild and we will therefore ignore the possibility.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2640 param = Parameter()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2641 token, value = get_attribute(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2642 param.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2643 if not value or value[0] == ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2644 param.defects.append(errors.InvalidHeaderDefect("Parameter contains "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2645 "name ({}) but no value".format(token)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2646 return param, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2647 if value[0] == '*':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2648 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2649 token, value = get_section(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2650 param.sectioned = True
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2651 param.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2652 except errors.HeaderParseError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2653 pass
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2654 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2655 raise errors.HeaderParseError("Incomplete parameter")
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2656 if value[0] == '*':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2657 param.append(ValueTerminal('*', 'extended-parameter-marker'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2658 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2659 param.extended = True
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2660 if value[0] != '=':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2661 raise errors.HeaderParseError("Parameter not followed by '='")
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2662 param.append(ValueTerminal('=', 'parameter-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2663 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2664 leader = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2665 if value and value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2666 token, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2667 param.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2668 remainder = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2669 appendto = param
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2670 if param.extended and value and value[0] == '"':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2671 # Now for some serious hackery to handle the common invalid case of
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2672 # double quotes around an extended value. We also accept (with defect)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2673 # a value marked as encoded that isn't really.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2674 qstring, remainder = get_quoted_string(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2675 inner_value = qstring.stripped_value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2676 semi_valid = False
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2677 if param.section_number == 0:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2678 if inner_value and inner_value[0] == "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2679 semi_valid = True
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2680 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2681 token, rest = get_attrtext(inner_value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2682 if rest and rest[0] == "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2683 semi_valid = True
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2684 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2685 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2686 token, rest = get_extended_attrtext(inner_value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2687 except:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2688 pass
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2689 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2690 if not rest:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2691 semi_valid = True
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2692 if semi_valid:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2693 param.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2694 "Quoted string value for extended parameter is invalid"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2695 param.append(qstring)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2696 for t in qstring:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2697 if t.token_type == 'bare-quoted-string':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2698 t[:] = []
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2699 appendto = t
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2700 break
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2701 value = inner_value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2702 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2703 remainder = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2704 param.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2705 "Parameter marked as extended but appears to have a "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2706 "quoted string value that is non-encoded"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2707 if value and value[0] == "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2708 token = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2709 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2710 token, value = get_value(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2711 if not param.extended or param.section_number > 0:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2712 if not value or value[0] != "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2713 appendto.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2714 if remainder is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2715 assert not value, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2716 value = remainder
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2717 return param, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2718 param.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2719 "Apparent initial-extended-value but attribute "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2720 "was not marked as extended or was not initial section"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2721 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2722 # Assume the charset/lang is missing and the token is the value.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2723 param.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2724 "Missing required charset/lang delimiters"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2725 appendto.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2726 if remainder is None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2727 return param, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2728 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2729 if token is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2730 for t in token:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2731 if t.token_type == 'extended-attrtext':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2732 break
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2733 t.token_type == 'attrtext'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2734 appendto.append(t)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2735 param.charset = t.value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2736 if value[0] != "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2737 raise errors.HeaderParseError("Expected RFC2231 char/lang encoding "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2738 "delimiter, but found {!r}".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2739 appendto.append(ValueTerminal("'", 'RFC2231 delimiter'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2740 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2741 if value and value[0] != "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2742 token, value = get_attrtext(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2743 appendto.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2744 param.lang = token.value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2745 if not value or value[0] != "'":
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2746 raise errors.HeaderParseError("Expected RFC2231 char/lang encoding "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2747 "delimiter, but found {}".format(value))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2748 appendto.append(ValueTerminal("'", 'RFC2231 delimiter'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2749 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2750 if remainder is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2751 # Treat the rest of value as bare quoted string content.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2752 v = Value()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2753 while value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2754 if value[0] in WSP:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2755 token, value = get_fws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2756 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2757 token, value = get_qcontent(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2758 v.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2759 token = v
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2760 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2761 token, value = get_value(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2762 appendto.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2763 if remainder is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2764 assert not value, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2765 value = remainder
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2766 return param, value
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2767
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2768 def parse_mime_parameters(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2769 """ parameter *( ";" parameter )
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2770
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2771 That BNF is meant to indicate this routine should only be called after
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2772 finding and handling the leading ';'. There is no corresponding rule in
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2773 the formal RFC grammar, but it is more convenient for us for the set of
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2774 parameters to be treated as its own TokenList.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2775
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2776 This is 'parse' routine because it consumes the reminaing value, but it
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2777 would never be called to parse a full header. Instead it is called to
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2778 parse everything after the non-parameter value of a specific MIME header.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2779
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2780 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2781 mime_parameters = MimeParameters()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2782 while value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2783 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2784 token, value = get_parameter(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2785 mime_parameters.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2786 except errors.HeaderParseError as err:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2787 leader = None
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2788 if value[0] in CFWS_LEADER:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2789 leader, value = get_cfws(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2790 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2791 mime_parameters.append(leader)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2792 return mime_parameters
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2793 if value[0] == ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2794 if leader is not None:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2795 mime_parameters.append(leader)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2796 mime_parameters.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2797 "parameter entry with no content"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2798 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2799 token, value = get_invalid_parameter(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2800 if leader:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2801 token[:0] = [leader]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2802 mime_parameters.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2803 mime_parameters.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2804 "invalid parameter {!r}".format(token)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2805 if value and value[0] != ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2806 # Junk after the otherwise valid parameter. Mark it as
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2807 # invalid, but it will have a value.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2808 param = mime_parameters[-1]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2809 param.token_type = 'invalid-parameter'
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2810 token, value = get_invalid_parameter(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2811 param.extend(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2812 mime_parameters.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2813 "parameter with invalid trailing text {!r}".format(token)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2814 if value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2815 # Must be a ';' at this point.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2816 mime_parameters.append(ValueTerminal(';', 'parameter-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2817 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2818 return mime_parameters
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2819
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2820 def _find_mime_parameters(tokenlist, value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2821 """Do our best to find the parameters in an invalid MIME header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2822
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2823 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2824 while value and value[0] != ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2825 if value[0] in PHRASE_ENDS:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2826 tokenlist.append(ValueTerminal(value[0], 'misplaced-special'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2827 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2828 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2829 token, value = get_phrase(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2830 tokenlist.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2831 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2832 return
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2833 tokenlist.append(ValueTerminal(';', 'parameter-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2834 tokenlist.append(parse_mime_parameters(value[1:]))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2835
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2836 def parse_content_type_header(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2837 """ maintype "/" subtype *( ";" parameter )
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2838
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2839 The maintype and substype are tokens. Theoretically they could
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2840 be checked against the official IANA list + x-token, but we
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2841 don't do that.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2842 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2843 ctype = ContentType()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2844 recover = False
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2845 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2846 ctype.defects.append(errors.HeaderMissingRequiredValue(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2847 "Missing content type specification"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2848 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2849 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2850 token, value = get_token(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2851 except errors.HeaderParseError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2852 ctype.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2853 "Expected content maintype but found {!r}".format(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2854 _find_mime_parameters(ctype, value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2855 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2856 ctype.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2857 # XXX: If we really want to follow the formal grammer we should make
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2858 # mantype and subtype specialized TokenLists here. Probably not worth it.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2859 if not value or value[0] != '/':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2860 ctype.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2861 "Invalid content type"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2862 if value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2863 _find_mime_parameters(ctype, value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2864 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2865 ctype.maintype = token.value.strip().lower()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2866 ctype.append(ValueTerminal('/', 'content-type-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2867 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2868 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2869 token, value = get_token(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2870 except errors.HeaderParseError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2871 ctype.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2872 "Expected content subtype but found {!r}".format(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2873 _find_mime_parameters(ctype, value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2874 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2875 ctype.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2876 ctype.subtype = token.value.strip().lower()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2877 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2878 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2879 if value[0] != ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2880 ctype.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2881 "Only parameters are valid after content type, but "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2882 "found {!r}".format(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2883 # The RFC requires that a syntactically invalid content-type be treated
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2884 # as text/plain. Perhaps we should postel this, but we should probably
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2885 # only do that if we were checking the subtype value against IANA.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2886 del ctype.maintype, ctype.subtype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2887 _find_mime_parameters(ctype, value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2888 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2889 ctype.append(ValueTerminal(';', 'parameter-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2890 ctype.append(parse_mime_parameters(value[1:]))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2891 return ctype
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2892
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2893 def parse_content_disposition_header(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2894 """ disposition-type *( ";" parameter )
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2895
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2896 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2897 disp_header = ContentDisposition()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2898 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2899 disp_header.defects.append(errors.HeaderMissingRequiredValue(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2900 "Missing content disposition"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2901 return disp_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2902 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2903 token, value = get_token(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2904 except errors.HeaderParseError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2905 ctype.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2906 "Expected content disposition but found {!r}".format(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2907 _find_mime_parameters(disp_header, value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2908 return disp_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2909 disp_header.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2910 disp_header.content_disposition = token.value.strip().lower()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2911 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2912 return disp_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2913 if value[0] != ';':
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2914 disp_header.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2915 "Only parameters are valid after content disposition, but "
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2916 "found {!r}".format(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2917 _find_mime_parameters(disp_header, value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2918 return disp_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2919 disp_header.append(ValueTerminal(';', 'parameter-separator'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2920 disp_header.append(parse_mime_parameters(value[1:]))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2921 return disp_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2922
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2923 def parse_content_transfer_encoding_header(value):
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2924 """ mechanism
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2925
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2926 """
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2927 # We should probably validate the values, since the list is fixed.
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2928 cte_header = ContentTransferEncoding()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2929 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2930 cte_header.defects.append(errors.HeaderMissingRequiredValue(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2931 "Missing content transfer encoding"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2932 return cte_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2933 try:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2934 token, value = get_token(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2935 except errors.HeaderParseError:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2936 ctype.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2937 "Expected content trnasfer encoding but found {!r}".format(value)))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2938 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2939 cte_header.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2940 cte_header.cte = token.value.strip().lower()
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2941 if not value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2942 return cte_header
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2943 while value:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2944 cte_header.defects.append(errors.InvalidHeaderDefect(
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2945 "Extra text after content transfer encoding"))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2946 if value[0] in PHRASE_ENDS:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2947 cte_header.append(ValueTerminal(value[0], 'misplaced-special'))
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2948 value = value[1:]
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2949 else:
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2950 token, value = get_phrase(value)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2951 cte_header.append(token)
0adccb941200 #15160: Extend the new email parser to handle MIME headers. R David Murray <rdmurray@bitdance.com> parents: 77165 diff changeset	2952 return cte_header

Mercurial > cpython-lru_cache-weakref

annotate Lib/email/_header_value_parser.py @ 79691:05c8928362ee