UnicodeDecodeError: 'utf-8' codec can't decode
Python tried to read bytes as UTF-8 and hit a byte sequence that is not valid UTF-8. The file (or socket) is almost certainly another encoding — or it is binary.
What the traceback is saying
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 42: invalid continuation byte
# or: invalid start byte / unexpected end of dataByte 0xe9 alone is fine in Latin-1/Windows-1252 (é) but incomplete as UTF-8 (UTF-8 would need a continuation byte). Python 3 defaults to UTF-8 for many text paths — including open() on modern Linux — so older files explode.
Usual causes
- CSV / log exported from Excel or a Windows app — often
cp1252(Windows-1252) or legacycp850. Open with that encoding, or re-save as UTF-8 from the source app. - Binary file opened as text — images, zip, protobuf,
.pyc. Useopen(path, 'rb')and do not decode. - Mixed encodings in one stream — concatenated logs from different machines. Decode line-by-line with a fallback, or normalise upstream.
- HTTP / subprocess output — declared charset lies, or none given. Prefer bytes first:
resp.content, then decode using the header or chardet/charset-normalizer. - Git showed mojibake earlier — if the file already looks like
éinstead ofé, see mojibake; decoding as UTF-8 may “succeed” into nonsense, which is a different failure mode.
Fixes that work
# 1) Know the encoding (Excel Western Europe → cp1252)
open('export.csv', encoding='cp1252')
# latin-1 never raises (maps every byte); good for forensic reads, not always correct semantics
open('export.csv', encoding='latin-1')
# 2) Binary
open('image.png', 'rb')
# 3) Sniff carefully (third-party)
# pip install charset-normalizer
from charset_normalizer import from_path
print(from_path('export.csv').best())
# 4) Explicit UTF-8 when you control the writer
open('out.txt', 'w', encoding='utf-8', newline='')- Prefer fixing at the source — save CSV as UTF-8 (Excel: “CSV UTF-8”), set
PYTHONIOENCODING=utf-8only when you understand the console. errors='replace'/ignore— hides corruption; OK for best-effort logs, bad for money/IDs.- Postgres / MySQL — connection charset mismatches create the sibling problem on insert; see MySQL utf8mb4.
UTF-8 vs Windows-1252 cheat sheet
| Symptom | Likely story |
|---|---|
| UnicodeDecodeError on read | Bytes are not UTF-8; pick the real encoding |
| File opens but shows é / ’ | UTF-8 bytes misread as Latin-1/cp1252 (mojibake) |
| Emoji / Asian text breaks in MySQL | Column/connection not utf8mb4 |
Related
mojibake UTF-8 vs Windows-1252 · MySQL Incorrect string value utf8mb4 · LF will be replaced by CRLF · invisible zero-width characters · emoji shows as box / tofu
Questions
What does UnicodeDecodeError utf-8 codec can't decode mean?
Python tried to interpret bytes as UTF-8 and hit an invalid sequence. Open with the correct encoding or as binary.
Which encoding should I pick?
For old Windows/Excel CSVs try cp1252. For Western European text latin-1/ISO-8859-1 maps every byte. When unsure, sniff with charset-normalizer and verify a few rows.
Why did it work on my Mac but fail on Linux?
macOS locale or a library may have guessed differently; Linux Python 3 often defaults open() to UTF-8. Always pass encoding= explicitly in shared code.
Is errors='ignore' safe?
It drops bytes silently. Fine for rough log scraping; dangerous for data you will write back or bill against.
How is this different from mojibake?
UnicodeDecodeError refuses to decode. Mojibake means a wrong decoding already produced plausible-looking wrong characters. Related but opposite failure modes.
JSON / API responses throw this
Decode response bytes with the charset from Content-Type, or use the HTTP client’s .text helper. Do not assume UTF-8 if the server lied.
Published .