Repository navigation
Conversation
`DateTime.__reduce_ex__()` returned the class plus the 8-tuple state, so the
fold flag never made it into the pickle stream. A `DateTime` on the second
occurrence of an ambiguous wall time came back as the first one, with a
different offset and a timestamp shifted by an hour:
```python
chicago = pendulum.timezone("America/Chicago")
after = pendulum.datetime(2024, 11, 3, 7, tz="utc").astimezone(chicago)
# 2024-11-03 01:00:00-06:00, fold=1, timestamp 1730617200.0
pickle.loads(pickle.dumps(after)).timestamp()
# Before: 1730613600.0 (2024-11-03 01:00:00-05:00, fold=0)
# After: 1730617200.0
```
The standard library rebuilds a datetime from its state, but `fold` is
keyword-only for `datetime.__new__()`, so it cannot be appended to the
positional tuple that gets expanded into `cls(*args)`. Restore through a
module-level function instead, which rebuilds the instance from its raw
state without going through `create()` (no timezone conversion, no
normalization of non-existent wall times, `tzinfo` kept as-is) and passes
`fold` as a keyword argument.
`copy.copy()` goes through the same `__reduce_ex__()` and is fixed as well.
Pickles written by older versions still load: they store the class plus the
8-tuple, which still builds a valid instance; they simply cannot carry a
fold they never contained.
Closes python-pendulum#908
|
Status note so this does not look stuck. The two workflow runs on Local results on that commit while waiting (Python 3.12.14, versions pinned in
Two things I would like your call on:
I will report the CI results once the workflows have run. |
Pull Request Check List
docs/, nothing to update.Fixes #908.
What was wrong
DateTime.__reduce_ex__()handed the class plus the 8-tuple from_getstate()to pickle, so thefoldflag was never serialized. Unpickling aDateTimethat sits on the second occurrence of an ambiguous wall time gave back the first occurrence instead: different offset, timestamp off by an hour.Why not restore through
create()#909 (
self.createplus a 9th element in_getstate()) does fix the timestamp above, but routing the state throughcreate()re-normalizes the instance. Measured on master with that approach: azoneinfo.ZoneInfotzinfo comes back asFixedTimezone/Timezone, and a wall time that does not exist (2024-03-10 02:30 in Chicago, which can only be built with the constructor) moves to 01:30 and changes its timestamp. This branch restores from the raw state instead: a module-level_restore_datetime()buildscls(*state, fold=fold)with no timezone conversion and the originaltzinfo.foldhas to be passed as a keyword because it is keyword-only fordatetime.__new__()— which is also why the class itself cannot stay the callable.copy.copy()goes through the same__reduce_ex__()and is fixed by this too (copy.deepcopy()was already correct).Compatibility
Pickles written by older versions still load: they store the class plus the 8-tuple, which still builds a valid instance, they only lack the fold they never contained. Checked by dumping with master and loading with this branch for 6 cases × 6 protocols (36 files): 36/36 load with identical type/value/timestamp, except the ambiguous
fold=1case where an old stream has nothing to restore from.Tests
Added to
tests/datetime/test_behavior.py: fold 0 and 1 across protocols 0-5, shallow copy, andtzinfopreservation. On master 8 of the 14 new cases fail (sixfold=1pickle cases, shallow copy, and thetzinfocase through its offset/timestamp assertions); all pass here.Local run, Python 3.12.14 with the versions pinned in
poetry.lock/prek.toml(python-dateutil 2.9.0.post0, pytest 9.0.3, ruff 0.14.11):PENDULUM_EXTENSIONS=0 python -m pytest -q tests→ 1882 passed, 5 skipped (master: 1868 passed, 5 skipped)python -m mypy→ no issues in 228 filesruff check/ruff format --check→ cleanNot covered locally: the
maturin developextension build and the free-threaded 3.14t / 3.15t jobs. The change is pure Python and does not touch parsing.Side note, not changed here
pendulum.Time._getstate()has the same shape and also losesfoldwhen pickled (Time(1, 30, tzinfo=timezone, fold=1)comes back withfold=0). Happy to extend this PR or send a separate one if you want it fixed too.AI assistance disclosure: developed with AI assistance, as disclosed in #1039. Every number above was produced by running the commands listed.