new CSV file processing module - see PEP 305

Doc/lib/libcsv.tex

@@ -0,0 +1,281 @@
+\section{\module{csv} --- CSV File Reading and Writing}
+
+\declaremodule{standard}{csv}
+\modulesynopsis{Write and read tabular data to and from delimited files.}
+
+\versionadded{2.3}
+\index{csv}
+\indexii{data}{tabular}
+
+The so-called CSV (Comma Separated Values) format is the most common import
+and export format for spreadsheets and databases.  There is no ``CSV
+standard'', so the format is operationally defined by the many applications
+which read and write it.  The lack of a standard means that subtle
+differences often exist in the data produced and consumed by different
+applications.  These differences can make it annoying to process CSV files
+from multiple sources.  Still, while the delimiters and quoting characters
+vary, the overall format is similar enough that it is possible to write a
+single module which can efficiently manipulate such data, hiding the details
+of reading and writing the data from the programmer.
+
+The \module{csv} module implements classes to read and write tabular data in
+CSV format.  It allows programmers to say, ``write this data in the format
+preferred by Excel,'' or ``read data from this file which was generated by
+Excel,'' without knowing the precise details of the CSV format used by
+Excel.  Programmers can also describe the CSV formats understood by other
+applications or define their own special-purpose CSV formats.
+
+The \module{csv} module's \class{reader} and \class{writer} objects read and
+write sequences.  Programmers can also read and write data in dictionary
+form using the \class{DictReader} and \class{DictWriter} classes.
+
+\note{The first version of the \module{csv} module doesn't support Unicode
+input.  Also, there are currently some issues regarding \ASCII{} NUL
+characters.  Accordingly, all input should generally be plain \ASCII{} to be
+safe.  These restrictions will be removed in the future.}
+
+\begin{seealso}
+%  \seemodule{array}{Arrays of uniformly types numeric values.}
+  \seepep{305}{CSV File API}
+         {The Python Enhancement Proposal which proposed this addition
+          to Python.}
+\end{seealso}
+
+
+\subsection{Module Contents}
+
+
+The \module{csv} module defines the following functions:
+
+\begin{funcdesc}{reader}{csvfile\optional{,
+                         dialect=\code{'excel'}\optional{, fmtparam}}}
+Return a reader object which will iterate over lines in the given
+{}\var{csvfile}.  \var{csvfile} can be any object which supports the
+iterator protocol and returns a string each time its \method{next}
+method is called.  An optional \var{dialect} parameter can be given
+which is used to define a set of parameters specific to a particular CSV
+dialect.  It may be an instance of a subclass of the \class{Dialect}
+class or one of the strings returned by the \function{list_dialects}
+function.  The other optional {}\var{fmtparam} keyword arguments can be
+given to override individual formatting parameters in the current
+dialect.  For more information about the dialect and formatting
+parameters, see section~\ref{fmt-params}, ``Dialects and Formatting
+Parameters'' for details of these parameters.
+
+All data read are returned as strings.  No automatic data type
+conversion is performed.
+\end{funcdesc}
+
+\begin{funcdesc}{writer}{csvfile\optional{,
+                         dialect=\code{'excel'}\optional{, fmtparam}}}
+Return a writer object responsible for converting the user's data into
+delimited strings on the given file-like object.  An optional
+{}\var{dialect} parameter can be given which is used to define a set of
+parameters specific to a particular CSV dialect.  It may be an instance
+of a subclass of the \class{Dialect} class or one of the strings
+returned by the \function{list_dialects} function.  The other optional
+{}\var{fmtparam} keyword arguments can be given to override individual
+formatting parameters in the current dialect.  For more information
+about the dialect and formatting parameters, see
+section~\ref{fmt-params}, ``Dialects and Formatting Parameters'' for
+details of these parameters.  To make it as easy as possible to
+interface with modules which implement the DB API, the value
+\constant{None} is written as the empty string.  While this isn't a
+reversible transformation, it makes it easier to dump SQL NULL data values
+to CSV files without preprocessing the data returned from a
+\code{cursor.fetch*()} call.  All other non-string data are stringified
+with \function{str()} before being written.
+\end{funcdesc}
+
+\begin{funcdesc}{register_dialect}{name, dialect}
+Associate \var{dialect} with \var{name}.  \var{dialect} must be a subclass
+of \class{csv.Dialect}.  \var{name} must be a string or Unicode object.
+\end{funcdesc}
+
+\begin{funcdesc}{unregister_dialect}{name}
+Delete the dialect associated with \var{name} from the dialect registry.  An
+\exception{Error} is raised if \var{name} is not a registered dialect
+name.
+\end{funcdesc}
+
+\begin{funcdesc}{get_dialect}{name}
+Return the dialect associated with \var{name}.  An \exception{Error} is
+raised if \var{name} is not a registered dialect name.
+\end{funcdesc}
+
+\begin{funcdesc}{list_dialects}{}
+Return the names of all registered dialects.
+\end{funcdesc}
+
+
+The \module{csv} module defines the following classes:
+
+\begin{classdesc}{DictReader}{csvfile, fieldnames\optional{,
+                              restkey=\code{None}\optional{,
+			      restval=\code{None}\optional{,
+                              dialect=\code{'excel'}\optional{,
+			      fmtparam}}}}}
+Create an object which operates like a regular reader but maps the
+information read into a dict whose keys are given by the \var{fieldnames}
+parameter.  If the row read has fewer fields than the fieldnames sequence,
+the value of \var{restval} will be used as the default value.  If the row
+read has more fields than the fieldnames sequence, the remaining data is
+added as a sequence keyed by the value of \var{restkey}.  If the row read
+has fewer fields than the fieldnames sequence, the remaining keys take the
+value of the optiona \var{restval} parameter.  All other parameters are
+interpreted as for regular readers.
+\end{classdesc}
+
+
+\begin{classdesc}{DictWriter}{csvfile, fieldnames\optional{,
+                              restval=""\optional{,
+                              extrasaction=\code{'raise'}\optional{,
+                              dialect=\code{'excel'}\optional{, fmtparam}}}}}
+Create an object which operates like a regular writer but maps dictionaries
+onto output rows.  The \var{fieldnames} parameter identifies the order in
+which values in the dictionary passed to the \method{writerow()} method are
+written to the \var{csvfile}.  The optional \var{restval} parameter
+specifies the value to be written if the dictionary is missing a key in
+\var{fieldnames}.  If the dictionary passed to the \method{writerow()}
+method contains a key not found in \var{fieldnames}, the optional
+\var{extrasaction} parameter indicates what action to take.  If it is set
+to \code{'raise'} a \exception{ValueError} is raised.  If it is set to
+\code{'ignore'}, extra values in the dictionary are ignored.  All other
+parameters are interpreted as for regular writers.
+\end{classdesc}
+
+
+\begin{classdesc*}{Dialect}{}
+The \class{Dialect} class is a container class relied on primarily for its
+attributes, which are used to define the parameters for a specific
+\class{reader} or \class{writer} instance.  Dialect objects support the
+following data attributes:
+
+\begin{memberdesc}[string]{delimiter}
+A one-character string used to separate fields.  It defaults to \code{","}.
+\end{memberdesc}
+
+\begin{memberdesc}[boolean]{doublequote}
+Controls how instances of \var{quotechar} appearing inside a field should be
+themselves be quoted.  When \constant{True}, the character is doubledd.
+When \constant{False}, the \var{escapechar} must be a one-character string
+which is used as a prefix to the \var{quotechar}.  It defaults to
+\constant{True}.
+\end{memberdesc}
+
+\begin{memberdesc}{escapechar}
+A one-character string used to escape the \var{delimiter} if \var{quoting}
+is set to \constant{QUOTE_NONE}.  It defaults to \constant{None}.
+\end{memberdesc}
+
+\begin{memberdesc}[string]{lineterminator}
+The string used to terminate lines in the CSV file.  It defaults to
+\code{"\e r\e n"}.
+\end{memberdesc}
+
+\begin{memberdesc}[string]{quotechar}
+A one-character string used to quote elements containing the \var{delimiter}
+or which start with the \var{quotechar}.  It defaults to \code{'"'}.
+\end{memberdesc}
+
+\begin{memberdesc}[integer]{quoting}
+Controls when quotes should be generated by the writer.  It can take on any
+of the \code{QUOTE_*} constants defined below and defaults to
+\constant{QUOTE_MINIMAL}. 
+\end{memberdesc}
+
+\begin{memberdesc}[boolean]{skipinitialspace}
+When \constant{True}, whitespace immediately following the \var{delimiter}
+is ignored.  The default is \constant{False}.
+\end{memberdesc}
+
+\end{classdesc*}
+
+The \module{csv} module defines the following constants:
+
+\begin{datadesc}{QUOTE_ALWAYS}
+Instructs \class{writer} objects to quote all fields.
+\end{datadesc}
+
+\begin{datadesc}{QUOTE_MINIMAL}
+Instructs \class{writer} objects to only quote those fields which contain
+the current \var{delimiter} or begin with the current \var{quotechar}.
+\end{datadesc}
+
+\begin{datadesc}{QUOTE_NONNUMERIC}
+Instructs \class{writer} objects to quote all non-numeric fields.
+\end{datadesc}
+
+\begin{datadesc}{QUOTE_NONE}
+Instructs \class{writer} objects to never quote fields.  When the current
+\var{delimiter} occurs in output data it is preceded by the current
+\var{escapechar} character.  When \constant{QUOTE_NONE} is in effect, it
+is an error not to have a single-character \var{escapechar} defined, even if
+no data to be written contains the \var{delimiter} character.
+\end{datadesc}
+
+
+The \module{csv} module defines the following exception:
+
+\begin{excdesc}{Error}
+Raised by any of the functions when an error is detected.
+\end{excdesc}
+
+
+\subsection{Dialects and Formatting Parameters\label{fmt-params}}
+
+To make it easier to specify the format of input and output records,
+specific formatting parameters are grouped together into dialects.  A
+dialect is a subclass of the \class{Dialect} class having a set of specific
+methods and a single \method{validate()} method.  When creating \class{reader}
+or \class{writer} objects, the programmer can specify a string or a subclass
+of the \class{Dialect} class as the dialect parameter.  In addition to, or
+instead of, the \var{dialect} parameter, the programmer can also specify
+individual formatting parameters, which have the same names as the
+attributes defined above for the \class{Dialect} class.
+
+
+\subsection{Reader Objects}
+
+\class{DictReader} and \var{reader} objects have the following public
+methods:
+
+\begin{methoddesc}{next}{}
+Return the next row of the reader's iterable object as a list, parsed
+according to the current dialect.
+\end{methoddesc}
+
+
+\subsection{Writer Objects}
+
+\class{DictWriter} and \var{writer} objects have the following public
+methods:
+
+\begin{methoddesc}{writerow}{row}
+Write the \var{row} parameter to the writer's file object, formatted
+according to the current dialect.
+\end{methoddesc}
+
+\begin{methoddesc}{writerows}{rows}
+Write all the \var{rows} parameters to the writer's file object, formatted
+according to the current dialect.
+\end{methoddesc}
+
+
+\subsection{Examples}
+
+The ``Hello, world'' of csv reading is
+
+\begin{verbatim}
+    reader = csv.reader(file("some.csv"))
+    for row in reader:
+        print row
+\end{verbatim}
+
+The corresponding simplest possible writing example is
+
+\begin{verbatim}
+    writer = csv.writer(file("some.csv", "w"))
+    for row in someiterable:
+        writer.writerow(row)
+\end{verbatim}

Lib/csv/__init__.py

@@ -0,0 +1 @@
+

Lib/csv/csv.py

@@ -0,0 +1,138 @@
+from _csv import Error, __version__, writer, reader, register_dialect, \
+                 unregister_dialect, get_dialect, list_dialects, \
+                 QUOTE_MINIMAL, QUOTE_ALL, QUOTE_NONNUMERIC, QUOTE_NONE, \
+                 __doc__
+
+__all__ = [ "QUOTE_MINIMAL", "QUOTE_ALL", "QUOTE_NONNUMERIC", "QUOTE_NONE",
+            "Error", "Dialect", "excel", "excel_tab", "reader", "writer",
+            "register_dialect", "get_dialect", "list_dialects",
+            "unregister_dialect", "__version__", "DictReader", "DictWriter" ]
+
+class Dialect:
+    _name = ""
+    _valid = False
+    # placeholders
+    delimiter = None
+    quotechar = None
+    escapechar = None
+    doublequote = None
+    skipinitialspace = None
+    lineterminator = None
+    quoting = None
+
+    def __init__(self):
+        if self.__class__ != Dialect:
+            self._valid = True
+        errors = self._validate()
+        if errors != []:
+            raise Error, "Dialect did not validate: %s" % ", ".join(errors)
+
+    def _validate(self):
+        errors = []
+        if not self._valid:
+            errors.append("can't directly instantiate Dialect class")
+
+        if self.delimiter is None:
+            errors.append("delimiter character not set")
+        elif (not isinstance(self.delimiter, str) or
+              len(self.delimiter) > 1):
+            errors.append("delimiter must be one-character string")
+
+        if self.quotechar is None:
+            if self.quoting != QUOTE_NONE:
+                errors.append("quotechar not set")
+        elif (not isinstance(self.quotechar, str) or
+              len(self.quotechar) > 1):
+            errors.append("quotechar must be one-character string")
+
+        if self.lineterminator is None:
+            errors.append("lineterminator not set")
+        elif not isinstance(self.lineterminator, str):
+            errors.append("lineterminator must be a string")
+
+        if self.doublequote not in (True, False):
+            errors.append("doublequote parameter must be True or False")
+
+        if self.skipinitialspace not in (True, False):
+            errors.append("skipinitialspace parameter must be True or False")
+
+        if self.quoting is None:
+            errors.append("quoting parameter not set")
+
+        if self.quoting is QUOTE_NONE:
+            if (not isinstance(self.escapechar, (unicode, str)) or
+                len(self.escapechar) > 1):
+                errors.append("escapechar must be a one-character string or unicode object")
+
+        return errors
+
+class excel(Dialect):
+    delimiter = ','
+    quotechar = '"'
+    doublequote = True
+    skipinitialspace = False
+    lineterminator = '\r\n'
+    quoting = QUOTE_MINIMAL
+register_dialect("excel", excel)
+
+class excel_tab(excel):
+    delimiter = '\t'
+register_dialect("excel-tab", excel_tab)
+
+
+class DictReader:
+    def __init__(self, f, fieldnames, restkey=None, restval=None,
+                 dialect="excel", *args):
+        self.fieldnames = fieldnames    # list of keys for the dict
+        self.restkey = restkey          # key to catch long rows
+        self.restval = restval          # default value for short rows
+        self.reader = reader(f, dialect, *args)
+
+    def __iter__(self):
+        return self
+
+    def next(self):
+        row = self.reader.next()
+        # unlike the basic reader, we prefer not to return blanks,
+        # because we will typically wind up with a dict full of None
+        # values
+        while row == []:
+            row = self.reader.next()
+        d = dict(zip(self.fieldnames, row))
+        lf = len(self.fieldnames)
+        lr = len(row)
+        if lf < lr:
+            d[self.restkey] = row[lf:]
+        elif lf > lr:
+            for key in self.fieldnames[lr:]:
+                d[key] = self.restval
+        return d
+
+
+class DictWriter:
+    def __init__(self, f, fieldnames, restval="", extrasaction="raise",
+                 dialect="excel", *args):
+        self.fieldnames = fieldnames    # list of keys for the dict
+        self.restval = restval          # for writing short dicts
+        if extrasaction.lower() not in ("raise", "ignore"):
+            raise ValueError, \
+                  ("extrasaction (%s) must be 'raise' or 'ignore'" % 
+                   extrasaction)
+        self.extrasaction = extrasaction
+        self.writer = writer(f, dialect, *args)
+
+    def _dict_to_list(self, rowdict):
+        if self.extrasaction == "raise":
+            for k in rowdict.keys():
+                if k not in self.fieldnames:
+                    raise ValueError, "dict contains fields not in fieldnames"
+        return [rowdict.get(key, self.restval) for key in self.fieldnames] 
+
+    def writerow(self, rowdict):
+        return self.writer.writerow(self._dict_to_list(rowdict))
+
+    def writerows(self, rowdicts):
+        rows = []
+        for rowdict in rowdicts:
+            rows.append(self._dict_to_list(rowdict))
+        return self.writer.writerows(rows)

Lib/test/test_csv.py

@@ -0,0 +1,619 @@
+# Copyright (C) 2001,2002 Python Software Foundation
+# csv package unit tests
+
+import sys
+import unittest
+from StringIO import StringIO
+from csv import csv
+import gc
+
+class Test_Csv(unittest.TestCase):
+    """
+    Test the underlying C csv parser in ways that are not appropriate 
+    from the high level interface. Further tests of this nature are done
+    in TestDialectRegistry.
+    """
+    def test_reader_arg_valid(self):
+        self.assertRaises(TypeError, csv.reader)
+        self.assertRaises(TypeError, csv.reader, None)
+        self.assertRaises(AttributeError, csv.reader, [], bad_attr = 0)
+        self.assertRaises(csv.Error, csv.reader, [], 'foo')
+        class BadClass:
+            def __init__(self):
+                raise IOError
+        self.assertRaises(IOError, csv.reader, [], BadClass)
+        self.assertRaises(TypeError, csv.reader, [], None)
+        class BadDialect:
+            bad_attr = 0
+        self.assertRaises(AttributeError, csv.reader, [], BadDialect)
+
+    def test_writer_arg_valid(self):
+        self.assertRaises(TypeError, csv.writer)
+        self.assertRaises(TypeError, csv.writer, None)
+        self.assertRaises(AttributeError, csv.writer, StringIO(), bad_attr = 0)
+
+    def _test_attrs(self, obj):
+        self.assertEqual(obj.dialect.delimiter, ',')
+        obj.dialect.delimiter = '\t'
+        self.assertEqual(obj.dialect.delimiter, '\t')
+        self.assertRaises(TypeError, delattr, obj.dialect, 'delimiter')
+        self.assertRaises(TypeError, setattr, obj.dialect, 
+                          'lineterminator', None)
+        obj.dialect.escapechar = None
+        self.assertEqual(obj.dialect.escapechar, None)
+        self.assertRaises(TypeError, delattr, obj.dialect, 'quoting')
+        self.assertRaises(TypeError, setattr, obj.dialect, 'quoting', None)
+        obj.dialect.quoting = csv.QUOTE_MINIMAL
+        self.assertEqual(obj.dialect.quoting, csv.QUOTE_MINIMAL)
+
+    def test_reader_attrs(self):
+        self._test_attrs(csv.reader([]))
+
+    def test_writer_attrs(self):
+        self._test_attrs(csv.writer(StringIO()))
+
+    def _write_test(self, fields, expect, **kwargs):
+        fileobj = StringIO()
+        writer = csv.writer(fileobj, **kwargs)
+        writer.writerow(fields)
+        self.assertEqual(fileobj.getvalue(), 
+                         expect + writer.dialect.lineterminator)
+
+    def test_write_arg_valid(self):
+        self.assertRaises(csv.Error, self._write_test, None, '')
+        self._write_test((), '')
+        self._write_test([None], '""')
+        self.assertRaises(csv.Error, self._write_test, 
+                          [None], None, quoting = csv.QUOTE_NONE)
+        # Check that exceptions are passed up the chain
+        class BadList:
+            def __len__(self):
+                return 10;
+            def __getitem__(self, i):
+                if i > 2:
+                    raise IOError
+        self.assertRaises(IOError, self._write_test, BadList(), '')
+        class BadItem:
+            def __str__(self):
+                raise IOError
+        self.assertRaises(IOError, self._write_test, [BadItem()], '')
+
+    def test_write_bigfield(self):
+        # This exercises the buffer realloc functionality
+        bigstring = 'X' * 50000
+        self._write_test([bigstring,bigstring], '%s,%s' % \
+                         (bigstring, bigstring))
+
+    def test_write_quoting(self):
+        self._write_test(['a','1','p,q'], 'a,1,"p,q"')
+        self.assertRaises(csv.Error, 
+                          self._write_test,
+                          ['a','1','p,q'], 'a,1,"p,q"',
+                          quoting = csv.QUOTE_NONE)
+        self._write_test(['a','1','p,q'], 'a,1,"p,q"',
+                         quoting = csv.QUOTE_MINIMAL)
+        self._write_test(['a','1','p,q'], '"a",1,"p,q"',
+                         quoting = csv.QUOTE_NONNUMERIC)
+        self._write_test(['a','1','p,q'], '"a","1","p,q"',
+                         quoting = csv.QUOTE_ALL)
+
+    def test_write_escape(self):
+        self._write_test(['a','1','p,q'], 'a,1,"p,q"',
+                         escapechar='\\')
+# FAILED - needs to be fixed [am]:
+#        self._write_test(['a','1','p,"q"'], 'a,1,"p,\\"q\\"',
+#                         escapechar='\\', doublequote = 0)
+        self._write_test(['a','1','p,q'], 'a,1,p\\,q',
+                         escapechar='\\', quoting = csv.QUOTE_NONE)
+
+    def test_writerows(self):
+        class BrokenFile:
+            def write(self, buf):
+                raise IOError
+        writer = csv.writer(BrokenFile())
+        self.assertRaises(IOError, writer.writerows, [['a']])
+        fileobj = StringIO()
+        writer = csv.writer(fileobj)
+        self.assertRaises(TypeError, writer.writerows, None)
+        writer.writerows([['a','b'],['c','d']])
+        self.assertEqual(fileobj.getvalue(), "a,b\r\nc,d\r\n")
+
+    def _read_test(self, input, expect, **kwargs):
+        reader = csv.reader(input, **kwargs)
+        result = list(reader)
+        self.assertEqual(result, expect)
+
+    def test_read_oddinputs(self):
+        self._read_test([], [])
+        self._read_test([''], [[]])
+        self.assertRaises(csv.Error, self._read_test,
+                          ['"ab"c'], None, strict = 1)
+        # cannot handle null bytes for the moment
+        self.assertRaises(csv.Error, self._read_test,
+                          ['ab\0c'], None, strict = 1)
+        self._read_test(['"ab"c'], [['abc']], doublequote = 0)
+
+    def test_read_eol(self):
+        self._read_test(['a,b'], [['a','b']])
+        self._read_test(['a,b\n'], [['a','b']])
+        self._read_test(['a,b\r\n'], [['a','b']])
+        self._read_test(['a,b\r'], [['a','b']])
+        self.assertRaises(csv.Error, self._read_test, ['a,b\rc,d'], [])
+        self.assertRaises(csv.Error, self._read_test, ['a,b\nc,d'], [])
+        self.assertRaises(csv.Error, self._read_test, ['a,b\r\nc,d'], [])
+
+    def test_read_escape(self):
+        self._read_test(['a,\\b,c'], [['a', '\\b', 'c']], escapechar='\\')
+        self._read_test(['a,b\\,c'], [['a', 'b,c']], escapechar='\\')
+        self._read_test(['a,"b\\,c"'], [['a', 'b,c']], escapechar='\\')
+        self._read_test(['a,"b,\\c"'], [['a', 'b,\\c']], escapechar='\\')
+        self._read_test(['a,"b,c\\""'], [['a', 'b,c"']], escapechar='\\')
+        self._read_test(['a,"b,c"\\'], [['a', 'b,c\\']], escapechar='\\')
+
+    def test_read_bigfield(self):
+        # This exercises the buffer realloc functionality
+        bigstring = 'X' * 50000
+        bigline = '%s,%s' % (bigstring, bigstring)
+        self._read_test([bigline], [[bigstring, bigstring]])
+
+class TestDialectRegistry(unittest.TestCase):
+    def test_registry_badargs(self):
+        self.assertRaises(TypeError, csv.list_dialects, None)
+        self.assertRaises(TypeError, csv.get_dialect)
+        self.assertRaises(csv.Error, csv.get_dialect, None)
+        self.assertRaises(csv.Error, csv.get_dialect, "nonesuch")
+        self.assertRaises(TypeError, csv.unregister_dialect)
+        self.assertRaises(csv.Error, csv.unregister_dialect, None)
+        self.assertRaises(csv.Error, csv.unregister_dialect, "nonesuch")
+        self.assertRaises(TypeError, csv.register_dialect, None)
+        self.assertRaises(TypeError, csv.register_dialect, None, None)
+        self.assertRaises(TypeError, csv.register_dialect, "nonesuch", None)
+        class bogus:
+            def __init__(self):
+                raise KeyError
+        self.assertRaises(KeyError, csv.register_dialect, "nonesuch", bogus)
+
+    def test_registry(self):
+        class myexceltsv(csv.excel):
+            delimiter = "\t"
+        name = "myexceltsv"
+        expected_dialects = csv.list_dialects() + [name]
+        expected_dialects.sort()
+        csv.register_dialect(name, myexceltsv)
+        try:
+            self.failUnless(isinstance(csv.get_dialect(name), myexceltsv))
+            got_dialects = csv.list_dialects()
+            got_dialects.sort()
+            self.assertEqual(expected_dialects, got_dialects)
+        finally:
+            csv.unregister_dialect(name)
+
+    def test_incomplete_dialect(self):
+        class myexceltsv(csv.Dialect):
+            delimiter = "\t"
+        self.assertRaises(csv.Error, myexceltsv)
+
+    def test_space_dialect(self):
+        class space(csv.excel):
+            delimiter = " "
+            quoting = csv.QUOTE_NONE
+            escapechar = "\\"
+
+        s = StringIO("abc def\nc1ccccc1 benzene\n")
+        rdr = csv.reader(s, dialect=space())
+        self.assertEqual(rdr.next(), ["abc", "def"])
+        self.assertEqual(rdr.next(), ["c1ccccc1", "benzene"])
+
+    def test_dialect_apply(self):
+        class testA(csv.excel):
+            delimiter = "\t"
+        class testB(csv.excel):
+            delimiter = ":"
+        class testC(csv.excel):
+            delimiter = "|"
+
+        csv.register_dialect('testC', testC)
+        try:
+            fileobj = StringIO()
+            writer = csv.writer(fileobj)
+            writer.writerow([1,2,3])
+            self.assertEqual(fileobj.getvalue(), "1,2,3\r\n")
+
+            fileobj = StringIO()
+            writer = csv.writer(fileobj, testA)
+            writer.writerow([1,2,3])
+            self.assertEqual(fileobj.getvalue(), "1\t2\t3\r\n")
+
+            fileobj = StringIO()
+            writer = csv.writer(fileobj, dialect=testB())
+            writer.writerow([1,2,3])
+            self.assertEqual(fileobj.getvalue(), "1:2:3\r\n")
+
+            fileobj = StringIO()
+            writer = csv.writer(fileobj, dialect='testC')
+            writer.writerow([1,2,3])
+            self.assertEqual(fileobj.getvalue(), "1|2|3\r\n")
+
+            fileobj = StringIO()
+            writer = csv.writer(fileobj, dialect=testA, delimiter=';')
+            writer.writerow([1,2,3])
+            self.assertEqual(fileobj.getvalue(), "1;2;3\r\n")
+        finally:
+            csv.unregister_dialect('testC')
+
+    def test_bad_dialect(self):
+        # Unknown parameter
+        self.assertRaises(AttributeError, csv.reader, [], bad_attr = 0)
+        # Bad values
+        self.assertRaises(TypeError, csv.reader, [], delimiter = None)
+        self.assertRaises(TypeError, csv.reader, [], quoting = -1)
+        self.assertRaises(TypeError, csv.reader, [], quoting = 100)
+
+class TestCsvBase(unittest.TestCase):
+    def readerAssertEqual(self, input, expected_result):
+        reader = csv.reader(StringIO(input), dialect = self.dialect)
+        fields = list(reader)
+        self.assertEqual(fields, expected_result)
+
+    def writerAssertEqual(self, input, expected_result):
+        fileobj = StringIO()
+        writer = csv.writer(fileobj, dialect = self.dialect)
+        writer.writerows(input)
+        self.assertEqual(fileobj.getvalue(), expected_result)
+
+class TestDialectExcel(TestCsvBase):
+    dialect = 'excel'
+
+    def test_single(self):
+        self.readerAssertEqual('abc', [['abc']])
+
+    def test_simple(self):
+        self.readerAssertEqual('1,2,3,4,5', [['1','2','3','4','5']])
+
+    def test_blankline(self):
+        self.readerAssertEqual('', [])
+
+    def test_empty_fields(self):
+        self.readerAssertEqual(',', [['', '']])
+
+    def test_singlequoted(self):
+        self.readerAssertEqual('""', [['']])
+
+    def test_singlequoted_left_empty(self):
+        self.readerAssertEqual('"",', [['','']])
+
+    def test_singlequoted_right_empty(self):
+        self.readerAssertEqual(',""', [['','']])
+
+    def test_single_quoted_quote(self):
+        self.readerAssertEqual('""""', [['"']])
+
+    def test_quoted_quotes(self):
+        self.readerAssertEqual('""""""', [['""']])
+
+    def test_inline_quote(self):
+        self.readerAssertEqual('a""b', [['a""b']])
+
+    def test_inline_quotes(self):
+        self.readerAssertEqual('a"b"c', [['a"b"c']])
+
+    def test_quotes_and_more(self):
+        self.readerAssertEqual('"a"b', [['ab']])
+
+    def test_lone_quote(self):
+        self.readerAssertEqual('a"b', [['a"b']])
+
+    def test_quote_and_quote(self):
+        self.readerAssertEqual('"a" "b"', [['a "b"']])
+
+    def test_space_and_quote(self):
+        self.readerAssertEqual(' "a"', [[' "a"']])
+
+    def test_quoted(self):
+        self.readerAssertEqual('1,2,3,"I think, therefore I am",5,6', 
+                               [['1', '2', '3', 
+                                 'I think, therefore I am', 
+                                 '5', '6']])
+
+    def test_quoted_quote(self):
+        self.readerAssertEqual('1,2,3,"""I see,"" said the blind man","as he picked up his hammer and saw"',
+                               [['1', '2', '3', 
+                                 '"I see," said the blind man', 
+                                 'as he picked up his hammer and saw']])
+
+    def test_quoted_nl(self):
+        input = '''\
+1,2,3,"""I see,""
+said the blind man","as he picked up his
+hammer and saw"
+9,8,7,6'''
+        self.readerAssertEqual(input,
+                               [['1', '2', '3', 
+                                   '"I see,"\nsaid the blind man', 
+                                   'as he picked up his\nhammer and saw'],
+                                ['9','8','7','6']])
+
+    def test_dubious_quote(self):
+        self.readerAssertEqual('12,12,1",', [['12', '12', '1"', '']])
+
+    def test_null(self):
+        self.writerAssertEqual([], '')
+
+    def test_single(self):
+        self.writerAssertEqual([['abc']], 'abc\r\n')
+
+    def test_simple(self):
+        self.writerAssertEqual([[1, 2, 'abc', 3, 4]], '1,2,abc,3,4\r\n')
+
+    def test_quotes(self):
+        self.writerAssertEqual([[1, 2, 'a"bc"', 3, 4]], '1,2,"a""bc""",3,4\r\n')
+
+    def test_quote_fieldsep(self):
+        self.writerAssertEqual([['abc,def']], '"abc,def"\r\n')
+
+    def test_newlines(self):
+        self.writerAssertEqual([[1, 2, 'a\nbc', 3, 4]], '1,2,"a\nbc",3,4\r\n')
+
+class EscapedExcel(csv.excel):
+    quoting = csv.QUOTE_NONE
+    escapechar = '\\'
+
+class TestEscapedExcel(TestCsvBase):
+    dialect = EscapedExcel()
+
+    def test_escape_fieldsep(self):
+        self.writerAssertEqual([['abc,def']], 'abc\\,def\r\n')
+
+    def test_read_escape_fieldsep(self):
+        self.readerAssertEqual('abc\\,def\r\n', [['abc,def']])
+
+class QuotedEscapedExcel(csv.excel):
+    quoting = csv.QUOTE_NONNUMERIC
+    escapechar = '\\'
+
+class TestQuotedEscapedExcel(TestCsvBase):
+    dialect = QuotedEscapedExcel()
+
+    def test_write_escape_fieldsep(self):
+        self.writerAssertEqual([['abc,def']], '"abc,def"\r\n')
+
+    def test_read_escape_fieldsep(self):
+        self.readerAssertEqual('"abc\\,def"\r\n', [['abc,def']])
+
+# Disabled, pending support in csv.utils module
+class TestDictFields(unittest.TestCase):
+    ### "long" means the row is longer than the number of fieldnames
+    ### "short" means there are fewer elements in the row than fieldnames
+    def test_write_simple_dict(self):
+        fileobj = StringIO()
+        writer = csv.DictWriter(fileobj, fieldnames = ["f1", "f2", "f3"])
+        writer.writerow({"f1": 10, "f3": "abc"})
+        self.assertEqual(fileobj.getvalue(), "10,,abc\r\n")
+
+    def test_write_no_fields(self):
+        fileobj = StringIO()
+        self.assertRaises(TypeError, csv.DictWriter, fileobj)
+
+    def test_read_dict_fields(self):
+        reader = csv.DictReader(StringIO("1,2,abc\r\n"),
+                                fieldnames=["f1", "f2", "f3"])
+        self.assertEqual(reader.next(), {"f1": '1', "f2": '2', "f3": 'abc'})
+
+    def test_read_long(self):
+        reader = csv.DictReader(StringIO("1,2,abc,4,5,6\r\n"),
+                                fieldnames=["f1", "f2"])
+        self.assertEqual(reader.next(), {"f1": '1', "f2": '2',
+                                         None: ["abc", "4", "5", "6"]})
+
+    def test_read_long_with_rest(self):
+        reader = csv.DictReader(StringIO("1,2,abc,4,5,6\r\n"),
+                                fieldnames=["f1", "f2"], restkey="_rest")
+        self.assertEqual(reader.next(), {"f1": '1', "f2": '2',
+                                         "_rest": ["abc", "4", "5", "6"]})
+
+    def test_read_short(self):
+        reader = csv.DictReader(["1,2,abc,4,5,6\r\n","1,2,abc\r\n"],
+                                fieldnames="1 2 3 4 5 6".split(),
+                                restval="DEFAULT")
+        self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
+                                         "4": '4', "5": '5', "6": '6'})
+        self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
+                                         "4": 'DEFAULT', "5": 'DEFAULT',
+                                         "6": 'DEFAULT'})
+
+    def test_read_with_blanks(self):
+        reader = csv.DictReader(["1,2,abc,4,5,6\r\n","\r\n",
+                                 "1,2,abc,4,5,6\r\n"],
+                                fieldnames="1 2 3 4 5 6".split())
+        self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
+                                         "4": '4', "5": '5', "6": '6'})
+        self.assertEqual(reader.next(), {"1": '1', "2": '2', "3": 'abc',
+                                         "4": '4', "5": '5', "6": '6'})
+
+class TestArrayWrites(unittest.TestCase):
+    def test_int_write(self):
+        import array
+        contents = [(20-i) for i in range(20)]
+        a = array.array('i', contents)
+        fileobj = StringIO()
+        writer = csv.writer(fileobj, dialect="excel")
+        writer.writerow(a)
+        expected = ",".join([str(i) for i in a])+"\r\n"
+        self.assertEqual(fileobj.getvalue(), expected)
+
+    def test_double_write(self):
+        import array
+        contents = [(20-i)*0.1 for i in range(20)]
+        a = array.array('d', contents)
+        fileobj = StringIO()
+        writer = csv.writer(fileobj, dialect="excel")
+        writer.writerow(a)
+        expected = ",".join([str(i) for i in a])+"\r\n"
+        self.assertEqual(fileobj.getvalue(), expected)
+
+    def test_float_write(self):
+        import array
+        contents = [(20-i)*0.1 for i in range(20)]
+        a = array.array('f', contents)
+        fileobj = StringIO()
+        writer = csv.writer(fileobj, dialect="excel")
+        writer.writerow(a)
+        expected = ",".join([str(i) for i in a])+"\r\n"
+        self.assertEqual(fileobj.getvalue(), expected)
+
+    def test_char_write(self):
+        import array, string
+        a = array.array('c', string.letters)
+        fileobj = StringIO()
+        writer = csv.writer(fileobj, dialect="excel")
+        writer.writerow(a)
+        expected = ",".join(a)+"\r\n"
+        self.assertEqual(fileobj.getvalue(), expected)
+
+class TestDialectValidity(unittest.TestCase):
+    def test_quoting(self):
+        class mydialect(csv.Dialect):
+            delimiter = ";"
+            escapechar = '\\'
+            doublequote = False
+            skipinitialspace = True
+            lineterminator = '\r\n'
+            quoting = csv.QUOTE_NONE
+        d = mydialect()
+
+        mydialect.quoting = None
+        self.assertRaises(csv.Error, mydialect)
+
+        mydialect.quoting = csv.QUOTE_NONE
+        mydialect.escapechar = None
+        self.assertRaises(csv.Error, mydialect)
+
+        mydialect.doublequote = True
+        mydialect.quoting = csv.QUOTE_ALL
+        mydialect.quotechar = '"'
+        d = mydialect()
+
+        mydialect.quotechar = "''"
+        self.assertRaises(csv.Error, mydialect)
+
+        mydialect.quotechar = 4
+        self.assertRaises(csv.Error, mydialect)
+
+    def test_delimiter(self):
+        class mydialect(csv.Dialect):
+            delimiter = ";"
+            escapechar = '\\'
+            doublequote = False
+            skipinitialspace = True
+            lineterminator = '\r\n'
+            quoting = csv.QUOTE_NONE
+        d = mydialect()
+
+        mydialect.delimiter = ":::"
+        self.assertRaises(csv.Error, mydialect)
+
+        mydialect.delimiter = 4
+        self.assertRaises(csv.Error, mydialect)
+
+    def test_lineterminator(self):
+        class mydialect(csv.Dialect):
+            delimiter = ";"
+            escapechar = '\\'
+            doublequote = False
+            skipinitialspace = True
+            lineterminator = '\r\n'
+            quoting = csv.QUOTE_NONE
+        d = mydialect()
+
+        mydialect.lineterminator = ":::"
+        d = mydialect()
+
+        mydialect.lineterminator = 4
+        self.assertRaises(csv.Error, mydialect)
+
+
+if not hasattr(sys, "gettotalrefcount"):
+    print "*** skipping leakage tests ***"
+else:
+    class NUL:
+        def write(s, *args):
+            pass
+        writelines = write
+
+    class TestLeaks(unittest.TestCase):
+        def test_create_read(self):
+            delta = 0
+            lastrc = sys.gettotalrefcount()
+            for i in xrange(20):
+                gc.collect()
+                self.assertEqual(gc.garbage, [])
+                rc = sys.gettotalrefcount()
+                csv.reader(["a,b,c\r\n"])
+                csv.reader(["a,b,c\r\n"])
+                csv.reader(["a,b,c\r\n"])
+                delta = rc-lastrc
+                lastrc = rc
+            # if csv.reader() leaks, last delta should be 3 or more
+            self.assertEqual(delta < 3, True)
+
+        def test_create_write(self):
+            delta = 0
+            lastrc = sys.gettotalrefcount()
+            s = NUL()
+            for i in xrange(20):
+                gc.collect()
+                self.assertEqual(gc.garbage, [])
+                rc = sys.gettotalrefcount()
+                csv.writer(s)
+                csv.writer(s)
+                csv.writer(s)
+                delta = rc-lastrc
+                lastrc = rc
+            # if csv.writer() leaks, last delta should be 3 or more
+            self.assertEqual(delta < 3, True)
+
+        def test_read(self):
+            delta = 0
+            rows = ["a,b,c\r\n"]*5
+            lastrc = sys.gettotalrefcount()
+            for i in xrange(20):
+                gc.collect()
+                self.assertEqual(gc.garbage, [])
+                rc = sys.gettotalrefcount()
+                rdr = csv.reader(rows)
+                for row in rdr:
+                    pass
+                delta = rc-lastrc
+                lastrc = rc
+            # if reader leaks during read, delta should be 5 or more
+            self.assertEqual(delta < 5, True)
+
+        def test_write(self):
+            delta = 0
+            rows = [[1,2,3]]*5
+            s = NUL()
+            lastrc = sys.gettotalrefcount()
+            for i in xrange(20):
+                gc.collect()
+                self.assertEqual(gc.garbage, [])
+                rc = sys.gettotalrefcount()
+                writer = csv.writer(s)
+                for row in rows:
+                    writer.writerow(row)
+                delta = rc-lastrc
+                lastrc = rc
+            # if writer leaks during write, last delta should be 5 or more
+            self.assertEqual(delta < 5, True)
+
+def _testclasses():
+    mod = sys.modules[__name__]
+    return [getattr(mod, name) for name in dir(mod) if name.startswith('Test')]
+
+def suite():
+    suite = unittest.TestSuite()
+    for testclass in _testclasses():
+        suite.addTest(unittest.makeSuite(testclass))
+    return suite
+
+if __name__ == '__main__':
+    unittest.main(defaultTest='suite')