StreamReader and StreamWriter issues

• StreamReader is unable to translate newlines.
• StreamWriter doesn’t support “line buffering” (flush if the input text contains a newline).
• StreamReader classes of the CJK encodings (e.g. GB18030) only supports UNIX newlines (’\n’).
• StreamReader and StreamWriter are stateful codecs but don’t expose functions to control their state (getstate() or setstate()). Each codec has to handle corner cases, see Appendix A.
• StreamReader and StreamWriter are very similar to IncrementalReader and IncrementalEncoder, some code is duplicated for stateful codecs (e.g. UTF-16).
• Each codec has to reimplement its own StreamReader and StreamWriter class, even if it’s trivial (just call the encoder/decoder).
• codecs.open(filename, “r”) creates an io.TextIOWrapper object.
• No codec implements an optimized method in StreamReader or StreamWriter based on the specificities of the codec.

Issues in the bug tracker:

• Issue #5445 (2009-03-08): codecs.StreamWriter.writelines problem when passed generator
• Issue #7262: (2009-11-04): codecs.open() + eol (windows)
• Issue #8260 (2010-03-29): When I use codecs.open(…) and f.readline() follow up by f.read() return bad result
• Issue #8630 (2010-05-05): Keepends param in codec readline(s)
• Issue #10344 (2010-11-06): codecs.readline doesn’t care buffering
• Issue #11461 (2011-03-10): Reading UTF-16 with codecs.readline() breaks on surrogate pairs
• Issue #12446 (2011-06-30): StreamReader Readlines behavior odd
• Issue #12508 (2011-07-06): Codecs Anomaly
• Issue #12512 (2011-07-07): codecs: StreamWriter issues with stateful codecs after a seek or with append mode
• Issue #12513 (2011-07-07): codec.StreamReaderWriter: issues with interlaced read-write

TextIOWrapper features

• TextIOWrapper supports any kind of newline, including translating newlines (to UNIX newlines), to read and write.
• TextIOWrapper reuses codecs incremental encoders and decoders (no duplication of code).
• The io module (TextIOWrapper) is faster than the codecs module (StreamReader). It is implemented in C, whereas codecs is implemented in Python.
• TextIOWrapper has a readahead algorithm which speeds up small reads: read character by character or line by line (io is 10x through 25x faster than codecs on these operations).
• TextIOWrapper has a write buffer.
• TextIOWrapper.tell() is optimized.
• TextIOWrapper supports random access (read+write) using a single class which permit to optimize interlaced read-write (but no such optimization is implemented).

TextIOWrapper issues

• Issue #12215 (2011-05-30): TextIOWrapper: issues with interlaced read-write

Possible improvements of StreamReader and StreamWriter

By adding codec state read/write functions to the StreamReader and StreamWriter classes, it will become possible to fix issues with stateful codecs in a base class instead of in each stateful StreamReader and StreamWriter classes.

It would be possible to change StreamReader and StreamWriter to make them use IncrementalDecoder and IncrementalEncoder.

A codec can implement variants which are optimized for the specific encoding or intercept certain stream methods to add functionality or improve the encoding/decoding performance. TextIOWrapper cannot implement such optimization, but TextIOWrapper uses incremental encoders and decoders and uses read and write buffers, so the overhead of incomplete inputs is low or nul.

A lot more could be done for other variable length encoding codecs, e.g. UTF-8, since these often have problems near the end of a read due to missing bytes. The UTF-32-BE/LE codecs could simply multiply the character position by 4 to get the byte position.

Usage of StreamReader and StreamWriter

These classes are rarely used directly, but indirectly using codecs.open(). They are not used in Python 3 standard library (except in the codecs module).

Some projects implement their own codec with StreamReader and StreamWriter, but don’t use these classes.

Keep the public API, codecs.open

codecs.open() can be replaced by the builtin open() function. open() has a similar API but has also more options. Both functions return file-like objects (same API).

codecs.open() was the only way to open a text file in Unicode mode until Python 2.6. Many Python 2 programs uses this function. Removing codecs.open() implies more work to port programs from Python 2 to Python 3, especially projects using the same code base for the two Python versions (without using 2to3 program).

codecs.open() is kept for backward compatibility with Python 2.

Deprecate StreamReader and StreamWriter

Instantiating StreamReader or StreamWriter must emit a DeprecationWarning in Python 3.3. Defining a subclass doesn’t emit a DeprecationWarning.

codecs.open() will be changed to reuse the builtin open() function (TextIOWrapper) to read-write text files.

Stateful codecs

Python supports the following stateful codecs:

• cp932
• cp949
• cp950
• euc_jis_2004
• euc_jisx2003
• euc_jp
• euc_kr
• gb18030
• gbk
• hz
• iso2022_jp
• iso2022_jp_1
• iso2022_jp_2
• iso2022_jp_2004
• iso2022_jp_3
• iso2022_jp_ext
• iso2022_kr
• shift_jis
• shift_jis_2004
• shift_jisx0213
• utf_8_sig
• utf_16
• utf_32

Read and seek(0)

with open(filename, 'w', encoding='utf-16') as f:
    f.write('abc')
    f.write('def')
    f.seek(0)
    assert f.read() == 'abcdef'
    f.seek(0)
    assert f.read() == 'abcdef'

The io and codecs modules support this usecase correctly.

seek(n)

with open(filename, 'w', encoding='utf-16') as f:
    f.write('abc')
    pos = f.tell()
with open(filename, 'w', encoding='utf-16') as f:
    f.seek(pos)
    f.write('def')
    f.seek(0)
    f.write('###')
with open(filename, 'r', encoding='utf-16') as f:
    assert f.read() == '###def'

The io module supports this usecase, whereas codecs fails because it writes a new BOM on the second write (issue #12512).

Append mode

with open(filename, 'w', encoding='utf-16') as f:
    f.write('abc')
with open(filename, 'a', encoding='utf-16') as f:
    f.write('def')
with open(filename, 'r', encoding='utf-16') as f:
    assert f.read() == 'abcdef'

The io module supports this usecase, whereas codecs fails because it writes a new BOM on the second write (issue #12512).