cpython-lru_cache-weakref: Lib/csv.py annotate

annotate Lib/csv.py @ 58765:2ba7677bfa95 legacy-trunk

testCopysign was defined twice in test_math; combine the definitions

author	Mark Dickinson <dickinsm@gmail.com>
date	Sat, 06 Feb 2010 23:11:25 +0000
parents	c76674c2d74d
children	dda5165a2374

rev	line source
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	2 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	3 csv.py - read/write/investigate CSV files
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	4 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	5
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	6 import re
48206 30135b6e38fd Silence warnings in csv about using reduce() when run under -3 by using Brett Cannon <bcannon@gmail.com> parents: 48198 diff changeset	7 from functools import reduce
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	8 from _csv import Error, __version__, writer, reader, register_dialect, \
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	9 unregister_dialect, get_dialect, list_dialects, \
34406 b93845ce4454 Rename csv.set_field_limit to csv.field_size_limit (since it both sets and Andrew McNamara <andrewm@object-craft.com.au> parents: 34396 diff changeset	10 field_size_limit, \
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	11 QUOTE_MINIMAL, QUOTE_ALL, QUOTE_NONNUMERIC, QUOTE_NONE, \
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	12 __doc__
34393 b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	13 from _csv import Dialect as _Dialect
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	14
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	15 try:
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	16 from cStringIO import StringIO
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	17 except ImportError:
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	18 from StringIO import StringIO
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	19
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	20 __all__ = [ "QUOTE_MINIMAL", "QUOTE_ALL", "QUOTE_NONNUMERIC", "QUOTE_NONE",
48278 62c69d88e985 Update __all__ for cookielib, csv, os, and urllib2 for objects imported into Brett Cannon <bcannon@gmail.com> parents: 48206 diff changeset	21 "Error", "Dialect", "__doc__", "excel", "excel_tab",
62c69d88e985 Update __all__ for cookielib, csv, os, and urllib2 for objects imported into Brett Cannon <bcannon@gmail.com> parents: 48206 diff changeset	22 "field_size_limit", "reader", "writer",
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	23 "register_dialect", "get_dialect", "list_dialects", "Sniffer",
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	24 "unregister_dialect", "__version__", "DictReader", "DictWriter" ]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	25
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	26 class Dialect:
34325 ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	27 """Describe an Excel dialect.
ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	28
ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	29 This must be subclassed (see csv.excel). Valid attributes are:
ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	30 delimiter, quotechar, escapechar, doublequote, skipinitialspace,
ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	31 lineterminator, quoting.
ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	32
ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	33 """
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	34 _name = ""
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	35 _valid = False
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	36 # placeholders
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	37 delimiter = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	38 quotechar = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	39 escapechar = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	40 doublequote = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	41 skipinitialspace = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	42 lineterminator = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	43 quoting = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	44
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	45 def __init__(self):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	46 if self.__class__ != Dialect:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	47 self._valid = True
34393 b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	48 self._validate()
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	49
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	50 def _validate(self):
34393 b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	51 try:
b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	52 _Dialect(self)
b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	53 except TypeError, e:
b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	54 # We do this for compatibility with py2.3
b6ec1b3b9fce Replace python-coded validation of csv dialect with a call to the C Andrew McNamara <andrewm@object-craft.com.au> parents: 34340 diff changeset	55 raise Error(str(e))
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	56
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	57 class excel(Dialect):
34325 ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	58 """Describe the usual properties of Excel-generated CSV files."""
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	59 delimiter = ','
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	60 quotechar = '"'
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	61 doublequote = True
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	62 skipinitialspace = False
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	63 lineterminator = '\r\n'
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	64 quoting = QUOTE_MINIMAL
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	65 register_dialect("excel", excel)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	66
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	67 class excel_tab(excel):
34325 ee808e08dd6f add a couple missing docstrings Skip Montanaro <skip@pobox.com> parents: 30632 diff changeset	68 """Describe the usual properties of Excel-generated TAB-delimited files."""
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	69 delimiter = '\t'
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	70 register_dialect("excel-tab", excel_tab)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	71
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	72
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	73 class DictReader:
30632 f9dfdcb897bb Make the fieldnames argument optional in the DictReader. If self.fieldnames Skip Montanaro <skip@pobox.com> parents: 30356 diff changeset	74 def __init__(self, f, fieldnames=None, restkey=None, restval=None,
30356 7504812fdb7e *kwds arg was missing from __init__ for Dict{Reader,Writer} classes. Skip Montanaro <skip@pobox.com>* parents: 30292 diff changeset	75 dialect="excel", args, *kwds):
48198 47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	76 self._fieldnames = fieldnames # list of keys for the dict
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	77 self.restkey = restkey # key to catch long rows
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	78 self.restval = restval # default value for short rows
30356 7504812fdb7e *kwds arg was missing from __init__ for Dict{Reader,Writer} classes. Skip Montanaro <skip@pobox.com>* parents: 30292 diff changeset	79 self.reader = reader(f, dialect, args, *kwds)
45981 85fbd1d265a2 Issue #2432: give DictReader the dialect and line_num attributes Georg Brandl <georg@python.org> parents: 41145 diff changeset	80 self.dialect = dialect
85fbd1d265a2 Issue #2432: give DictReader the dialect and line_num attributes Georg Brandl <georg@python.org> parents: 41145 diff changeset	81 self.line_num = 0
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	82
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	83 def __iter__(self):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	84 return self
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	85
48198 47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	86 @property
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	87 def fieldnames(self):
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	88 if self._fieldnames is None:
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	89 try:
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	90 self._fieldnames = self.reader.next()
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	91 except StopIteration:
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	92 pass
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	93 self.line_num = self.reader.line_num
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	94 return self._fieldnames
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	95
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	96 @fieldnames.setter
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	97 def fieldnames(self, value):
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	98 self._fieldnames = value
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	99
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	100 def next(self):
48198 47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	101 if self.line_num == 0:
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	102 # Used only for its side effect.
47967690b585 accept issue 3436 Skip Montanaro <skip@pobox.com> parents: 45981 diff changeset	103 self.fieldnames
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	104 row = self.reader.next()
45981 85fbd1d265a2 Issue #2432: give DictReader the dialect and line_num attributes Georg Brandl <georg@python.org> parents: 41145 diff changeset	105 self.line_num = self.reader.line_num
30632 f9dfdcb897bb Make the fieldnames argument optional in the DictReader. If self.fieldnames Skip Montanaro <skip@pobox.com> parents: 30356 diff changeset	106
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	107 # unlike the basic reader, we prefer not to return blanks,
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	108 # because we will typically wind up with a dict full of None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	109 # values
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	110 while row == []:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	111 row = self.reader.next()
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	112 d = dict(zip(self.fieldnames, row))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	113 lf = len(self.fieldnames)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	114 lr = len(row)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	115 if lf < lr:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	116 d[self.restkey] = row[lf:]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	117 elif lf > lr:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	118 for key in self.fieldnames[lr:]:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	119 d[key] = self.restval
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	120 return d
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	121
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	122
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	123 class DictWriter:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	124 def __init__(self, f, fieldnames, restval="", extrasaction="raise",
30356 7504812fdb7e *kwds arg was missing from __init__ for Dict{Reader,Writer} classes. Skip Montanaro <skip@pobox.com>* parents: 30292 diff changeset	125 dialect="excel", args, *kwds):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	126 self.fieldnames = fieldnames # list of keys for the dict
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	127 self.restval = restval # for writing short dicts
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	128 if extrasaction.lower() not in ("raise", "ignore"):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	129 raise ValueError, \
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	130 ("extrasaction (%s) must be 'raise' or 'ignore'" %
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	131 extrasaction)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	132 self.extrasaction = extrasaction
30356 7504812fdb7e *kwds arg was missing from __init__ for Dict{Reader,Writer} classes. Skip Montanaro <skip@pobox.com>* parents: 30292 diff changeset	133 self.writer = writer(f, dialect, args, *kwds)
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	134
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	135 def _dict_to_list(self, rowdict):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	136 if self.extrasaction == "raise":
41145 bbaf6c935663 Patch #1635454: the csv.DictWriter class now includes the offending Georg Brandl <georg@python.org> parents: 36157 diff changeset	137 wrong_fields = [k for k in rowdict if k not in self.fieldnames]
bbaf6c935663 Patch #1635454: the csv.DictWriter class now includes the offending Georg Brandl <georg@python.org> parents: 36157 diff changeset	138 if wrong_fields:
bbaf6c935663 Patch #1635454: the csv.DictWriter class now includes the offending Georg Brandl <georg@python.org> parents: 36157 diff changeset	139 raise ValueError("dict contains fields not in fieldnames: " +
bbaf6c935663 Patch #1635454: the csv.DictWriter class now includes the offending Georg Brandl <georg@python.org> parents: 36157 diff changeset	140 ", ".join(wrong_fields))
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	141 return [rowdict.get(key, self.restval) for key in self.fieldnames]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	142
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	143 def writerow(self, rowdict):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	144 return self.writer.writerow(self._dict_to_list(rowdict))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	145
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	146 def writerows(self, rowdicts):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	147 rows = []
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	148 for rowdict in rowdicts:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	149 rows.append(self._dict_to_list(rowdict))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	150 return self.writer.writerows(rows)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	151
29411 bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	152 # Guard Sniffer's type checking against builds that exclude complex()
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	153 try:
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	154 complex
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	155 except NameError:
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	156 complex = float
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	157
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	158 class Sniffer:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	159 '''
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	160 "Sniffs" the format of a CSV file (i.e. delimiter, quotechar)
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	161 Returns a Dialect object.
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	162 '''
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	163 def __init__(self):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	164 # in case there is more than one possible delimiter
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	165 self.preferred = [',', '\t', ';', ' ', ':']
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	166
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	167
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	168 def sniff(self, sample, delimiters=None):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	169 """
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	170 Returns a dialect (or None) corresponding to the sample
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	171 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	172
56253 c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	173 quotechar, doublequote, delimiter, skipinitialspace = \
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	174 self._guess_quote_and_delimiter(sample, delimiters)
36157 96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	175 if not delimiter:
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	176 delimiter, skipinitialspace = self._guess_delimiter(sample,
434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	177 delimiters)
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	178
36157 96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	179 if not delimiter:
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	180 raise Error, "Could not determine delimiter"
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	181
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	182 class dialect(Dialect):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	183 _name = "sniffed"
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	184 lineterminator = '\r\n'
28825 2fad548391de Attempt to deal with some obvious errors in the code. These were all Fred Drake <fdrake@acm.org> parents: 28804 diff changeset	185 quoting = QUOTE_MINIMAL
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	186 # escapechar = ''
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	187
56253 c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	188 dialect.doublequote = doublequote
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	189 dialect.delimiter = delimiter
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	190 # _csv.reader won't accept a quotechar of ''
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	191 dialect.quotechar = quotechar or '"'
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	192 dialect.skipinitialspace = skipinitialspace
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	193
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	194 return dialect
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	195
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	196
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	197 def _guess_quote_and_delimiter(self, data, delimiters):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	198 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	199 Looks for text enclosed between two identical quotes
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	200 (the probable quotechar) which are preceded and followed
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	201 by the same character (the probable delimiter).
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	202 For example:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	203 ,'some text',
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	204 The quote with the most wins, same with the delimiter.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	205 If there is no quotechar the delimiter can't be determined
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	206 this way.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	207 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	208
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	209 matches = []
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	210 for restr in ('(?P<delim>[^\w\n"\'])(?P<space> ?)(?P<quote>["\']).?(?P=quote)(?P=delim)', # ,".?",
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	211 '(?:^\|\n)(?P<quote>["\']).?(?P=quote)(?P<delim>[^\w\n"\'])(?P<space> ?)', # ".?",
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	212 '(?P<delim>>[^\w\n"\'])(?P<space> ?)(?P<quote>["\']).?(?P=quote)(?:$\|\n)', # ,".?"
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	213 '(?:^\|\n)(?P<quote>["\']).?(?P=quote)(?:$\|\n)'): # ".?" (no delim, no space)
30292 79b1a9ec5223 use the long names for re compilation options; this makes it easier to Fred Drake <fdrake@acm.org> parents: 29422 diff changeset	214 regexp = re.compile(restr, re.DOTALL \| re.MULTILINE)
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	215 matches = regexp.findall(data)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	216 if matches:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	217 break
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	218
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	219 if not matches:
56253 c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	220 # (quotechar, doublequote, delimiter, skipinitialspace)
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	221 return ('', False, None, 0)
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	222 quotes = {}
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	223 delims = {}
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	224 spaces = 0
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	225 for m in matches:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	226 n = regexp.groupindex['quote'] - 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	227 key = m[n]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	228 if key:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	229 quotes[key] = quotes.get(key, 0) + 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	230 try:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	231 n = regexp.groupindex['delim'] - 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	232 key = m[n]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	233 except KeyError:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	234 continue
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	235 if key and (delimiters is None or key in delimiters):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	236 delims[key] = delims.get(key, 0) + 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	237 try:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	238 n = regexp.groupindex['space'] - 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	239 except KeyError:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	240 continue
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	241 if m[n]:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	242 spaces += 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	243
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	244 quotechar = reduce(lambda a, b, quotes = quotes:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	245 (quotes[a] > quotes[b]) and a or b, quotes.keys())
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	246
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	247 if delims:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	248 delim = reduce(lambda a, b, delims = delims:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	249 (delims[a] > delims[b]) and a or b, delims.keys())
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	250 skipinitialspace = delims[delim] == spaces
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	251 if delim == '\n': # most likely a file with a single column
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	252 delim = ''
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	253 else:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	254 # there is no delimiter, it's a single column of quoted data
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	255 delim = ''
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	256 skipinitialspace = 0
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	257
56253 c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	258 # if we see an extra quote between delimiters, we've got a
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	259 # double quoted format
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	260 dq_regexp = re.compile(r"((%(delim)s)\|^)\W%(quote)s[^%(delim)s\n]%(quote)s[^%(delim)s\n]%(quote)s\W((%(delim)s)\|$)" % \
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	261 {'delim':delim, 'quote':quotechar}, re.MULTILINE)
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	262
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	263
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	264
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	265 if dq_regexp.search(data):
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	266 doublequote = True
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	267 else:
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	268 doublequote = False
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	269
c76674c2d74d Patch from Thomas Barr so that csv.Sniffer will set doublequote property. Skip Montanaro <skip@pobox.com> parents: 48278 diff changeset	270 return (quotechar, doublequote, delim, skipinitialspace)
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	271
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	272
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	273 def _guess_delimiter(self, data, delimiters):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	274 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	275 The delimiter /should/ occur the same number of times on
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	276 each row. However, due to malformed data, it may not. We don't want
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	277 an all or nothing approach, so we allow for small variations in this
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	278 number.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	279 1) build a table of the frequency of each character on every line.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	280 2) build a table of freqencies of this frequency (meta-frequency?),
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	281 e.g. 'x occurred 5 times in 10 rows, 6 times in 1000 rows,
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	282 7 times in 2 rows'
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	283 3) use the mode of the meta-frequency to determine the /expected/
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	284 frequency for that character
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	285 4) find out how often the character actually meets that goal
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	286 5) the character that best meets its goal is the delimiter
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	287 For performance reasons, the data is evaluated in chunks, so it can
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	288 try and evaluate the smallest portion of the data possible, evaluating
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	289 additional chunks as necessary.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	290 """
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	291
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	292 data = filter(None, data.split('\n'))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	293
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	294 ascii = [chr(c) for c in range(127)] # 7-bit ASCII
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	295
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	296 # build frequency tables
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	297 chunkLength = min(10, len(data))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	298 iteration = 0
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	299 charFrequency = {}
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	300 modes = {}
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	301 delims = {}
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	302 start, end = 0, min(chunkLength, len(data))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	303 while start < len(data):
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	304 iteration += 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	305 for line in data[start:end]:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	306 for char in ascii:
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	307 metaFrequency = charFrequency.get(char, {})
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	308 # must count even if frequency is 0
36146 22dbf5818b0b Fix for problem with Sniffer class. If your delimiter is whitespace and the Skip Montanaro <skip@pobox.com> parents: 34406 diff changeset	309 freq = line.count(char)
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	310 # value is the mode
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	311 metaFrequency[freq] = metaFrequency.get(freq, 0) + 1
b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	312 charFrequency[char] = metaFrequency
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	313
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	314 for char in charFrequency.keys():
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	315 items = charFrequency[char].items()
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	316 if len(items) == 1 and items[0][0] == 0:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	317 continue
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	318 # get the mode of the frequencies
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	319 if len(items) > 1:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	320 modes[char] = reduce(lambda a, b: a[1] > b[1] and a or b,
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	321 items)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	322 # adjust the mode - subtract the sum of all
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	323 # other frequencies
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	324 items.remove(modes[char])
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	325 modes[char] = (modes[char][0], modes[char][1]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	326 - reduce(lambda a, b: (0, a[1] + b[1]),
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	327 items)[1])
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	328 else:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	329 modes[char] = items[0]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	330
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	331 # build a list of possible delimiters
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	332 modeList = modes.items()
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	333 total = float(chunkLength * iteration)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	334 # (rows of consistent data) / (number of rows) = 100%
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	335 consistency = 1.0
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	336 # minimum consistency threshold
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	337 threshold = 0.9
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	338 while len(delims) == 0 and consistency >= threshold:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	339 for k, v in modeList:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	340 if v[0] > 0 and v[1] > 0:
29169 434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	341 if ((v[1]/total) >= consistency and
434ef174f478 * Correct Sniffer doc to correspond to the implementation. Skip Montanaro <skip@pobox.com> parents: 28828 diff changeset	342 (delimiters is None or k in delimiters)):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	343 delims[k] = v
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	344 consistency -= 0.01
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	345
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	346 if len(delims) == 1:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	347 delim = delims.keys()[0]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	348 skipinitialspace = (data[0].count(delim) ==
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	349 data[0].count("%c " % delim))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	350 return (delim, skipinitialspace)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	351
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	352 # analyze another chunkLength lines
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	353 start = end
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	354 end += chunkLength
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	355
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	356 if not delims:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	357 return ('', 0)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	358
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	359 # if there's more than one, fall back to a 'preferred' list
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	360 if len(delims) > 1:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	361 for d in self.preferred:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	362 if d in delims.keys():
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	363 skipinitialspace = (data[0].count(d) ==
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	364 data[0].count("%c " % d))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	365 return (d, skipinitialspace)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	366
36157 96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	367 # nothing else indicates a preference, pick the character that
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	368 # dominates(?)
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	369 items = [(v,k) for (k,v) in delims.items()]
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	370 items.sort()
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	371 delim = items[-1][1]
96eb2a5cfbef Fix a delimiter detection problem in sniffer. Sniffing "a\|b\|c\r\n" was Skip Montanaro <skip@pobox.com> parents: 36146 diff changeset	372
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	373 skipinitialspace = (data[0].count(delim) ==
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	374 data[0].count("%c " % delim))
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	375 return (delim, skipinitialspace)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	376
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	377
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	378 def has_header(self, sample):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	379 # Creates a dictionary of types of data in each column. If any
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	380 # column is of a single type (say, integers), except for the first
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	381 # row, then the first row is presumed to be labels. If the type
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	382 # can't be determined, it is assumed to be a string in which case
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	383 # the length of the string is the determining factor: if all of the
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	384 # rows except for the first are the same length, it's a header.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	385 # Finally, a 'vote' is taken at the end for each column, adding or
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	386 # subtracting from the likelihood of the first row being a header.
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	387
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	388 rdr = reader(StringIO(sample), self.sniff(sample))
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	389
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	390 header = rdr.next() # assume first row is header
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	391
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	392 columns = len(header)
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	393 columnTypes = {}
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	394 for i in range(columns): columnTypes[i] = None
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	395
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	396 checked = 0
28828 b37af21b77be rework Sniffer api significantly Skip Montanaro <skip@pobox.com> parents: 28825 diff changeset	397 for row in rdr:
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	398 # arbitrary number of rows to check, to keep it sane
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	399 if checked > 20:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	400 break
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	401 checked += 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	402
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	403 if len(row) != columns:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	404 continue # skip rows that have irregular number of columns
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	405
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	406 for col in columnTypes.keys():
29411 bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	407
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	408 for thisType in [int, long, float, complex]:
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	409 try:
29411 bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	410 thisType(row[col])
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	411 break
29420 f87fcd0870bf The multiple exception catch should be in a tuple. Raymond Hettinger <python@rcn.com> parents: 29411 diff changeset	412 except (ValueError, OverflowError):
29411 bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	413 pass
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	414 else:
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	415 # fallback to length of string
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	416 thisType = len(row[col])
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	417
29411 bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	418 # treat longs as ints
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	419 if thisType == long:
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	420 thisType = int
bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	421
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	422 if thisType != columnTypes[col]:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	423 if columnTypes[col] is None: # add new column type
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	424 columnTypes[col] = thisType
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	425 else:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	426 # type is inconsistent, remove column from
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	427 # consideration
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	428 del columnTypes[col]
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	429
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	430 # finally, compare results against first row and "vote"
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	431 # on whether it's a header
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	432 hasHeader = 0
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	433 for col, colType in columnTypes.items():
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	434 if type(colType) == type(0): # it's a length
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	435 if len(header[col]) != colType:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	436 hasHeader += 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	437 else:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	438 hasHeader -= 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	439 else: # attempt typecast
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	440 try:
29411 bac2c0e7c20a SF Patch #744104: Remove eval() from csv Raymond Hettinger <python@rcn.com> parents: 29169 diff changeset	441 colType(header[col])
29422 cf8ae3b0dcdc One more multiple exception catch should be in a tuple. Raymond Hettinger <python@rcn.com> parents: 29420 diff changeset	442 except (ValueError, TypeError):
28804 0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	443 hasHeader += 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	444 else:
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	445 hasHeader -= 1
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	446
0c950e3323ee cvs is going to be a module again Skip Montanaro <skip@pobox.com> parents: diff changeset	447 return hasHeader > 0

Mercurial > cpython-lru_cache-weakref

annotate Lib/csv.py @ 58765:2ba7677bfa95 legacy-trunk