Skip to content

Preserve the fold when pickling DateTime - #1073

Open
Duo-ee wants to merge 1 commit into
python-pendulum:masterfrom
Duo-ee:fix/datetime-pickle-preserves-fold
Open

Duo-ee wants to merge 1 commit into
python-pendulum:masterfrom
Duo-ee:fix/datetime-pickle-preserves-fold

Conversation

@Duo-ee

@Duo-ee Duo-ee commented Oct 28, 2026

Copy link
Copy Markdown

Pull Request Check List

  • Added tests for changed code.
  • Updated documentation for changed code — pickling is not described in docs/, nothing to update.

Fixes #908.

What was wrong

DateTime.__reduce_ex__() handed the class plus the 8-tuple from _getstate() to pickle, so the fold flag was never serialized. Unpickling a DateTime that sits on the second occurrence of an ambiguous wall time gave back the first occurrence instead: different offset, timestamp off by an hour.

import pickle
import pendulum

chicago = pendulum.timezone("America/Chicago")
after = pendulum.datetime(2024, 11, 3, 7, tz="utc").astimezone(chicago)
after.timestamp()  # 1730617200.0, 01:00 -06:00, fold=1
pickle.loads(pickle.dumps(after)).timestamp()
# master: 1730613600.0 (01:00 -05:00, fold=0); this branch: 1730617200.0

Why not restore through create()

#909 (self.create plus a 9th element in _getstate()) does fix the timestamp above, but routing the state through create() re-normalizes the instance. Measured on master with that approach: a zoneinfo.ZoneInfo tzinfo comes back as FixedTimezone/Timezone, and a wall time that does not exist (2024-03-10 02:30 in Chicago, which can only be built with the constructor) moves to 01:30 and changes its timestamp. This branch restores from the raw state instead: a module-level _restore_datetime() builds cls(*state, fold=fold) with no timezone conversion and the original tzinfo. fold has to be passed as a keyword because it is keyword-only for datetime.__new__() — which is also why the class itself cannot stay the callable.

copy.copy() goes through the same __reduce_ex__() and is fixed by this too (copy.deepcopy() was already correct).

Compatibility

Pickles written by older versions still load: they store the class plus the 8-tuple, which still builds a valid instance, they only lack the fold they never contained. Checked by dumping with master and loading with this branch for 6 cases × 6 protocols (36 files): 36/36 load with identical type/value/timestamp, except the ambiguous fold=1 case where an old stream has nothing to restore from.

Tests

Added to tests/datetime/test_behavior.py: fold 0 and 1 across protocols 0-5, shallow copy, and tzinfo preservation. On master 8 of the 14 new cases fail (six fold=1 pickle cases, shallow copy, and the tzinfo case through its offset/timestamp assertions); all pass here.

Local run, Python 3.12.14 with the versions pinned in poetry.lock / prek.toml (python-dateutil 2.9.0.post0, pytest 9.0.3, ruff 0.14.11):

  • PENDULUM_EXTENSIONS=0 python -m pytest -q tests → 1882 passed, 5 skipped (master: 1868 passed, 5 skipped)
  • python -m mypy → no issues in 228 files
  • ruff check / ruff format --check → clean

Not covered locally: the maturin develop extension build and the free-threaded 3.14t / 3.15t jobs. The change is pure Python and does not touch parsing.

Side note, not changed here

pendulum.Time._getstate() has the same shape and also loses fold when pickled (Time(1, 30, tzinfo=timezone, fold=1) comes back with fold=0). Happy to extend this PR or send a separate one if you want it fixed too.

AI assistance disclosure: developed with AI assistance, as disclosed in #1039. Every number above was produced by running the commands listed.

`DateTime.__reduce_ex__()` returned the class plus the 8-tuple state, so the
fold flag never made it into the pickle stream. A `DateTime` on the second
occurrence of an ambiguous wall time came back as the first one, with a
different offset and a timestamp shifted by an hour:

```python
chicago = pendulum.timezone("America/Chicago")
after = pendulum.datetime(2024, 11, 3, 7, tz="utc").astimezone(chicago)
# 2024-11-03 01:00:00-06:00, fold=1, timestamp 1730617200.0
pickle.loads(pickle.dumps(after)).timestamp()
# Before: 1730613600.0 (2024-11-03 01:00:00-05:00, fold=0)
# After:  1730617200.0
```

The standard library rebuilds a datetime from its state, but `fold` is
keyword-only for `datetime.__new__()`, so it cannot be appended to the
positional tuple that gets expanded into `cls(*args)`. Restore through a
module-level function instead, which rebuilds the instance from its raw
state without going through `create()` (no timezone conversion, no
normalization of non-existent wall times, `tzinfo` kept as-is) and passes
`fold` as a keyword argument.

`copy.copy()` goes through the same `__reduce_ex__()` and is fixed as well.
Pickles written by older versions still load: they store the class plus the
8-tuple, which still builds a valid instance; they simply cannot carry a
fold they never contained.

Closes python-pendulum#908
@Duo-ee Duo-ee mentioned this pull request Oct 28, 2026
2 tasks
@Duo-ee

Duo-ee commented Oct 28, 2026

Copy link
Copy Markdown
Author

Status note so this does not look stuck.

The two workflow runs on f97c213 (Tests #1004, codspeed #646) both ended at action_required, which is the outside-contributor approval gate, so the matrix has not actually run yet. Approving the runs on this PR is enough; there is nothing to fix on my side.

Local results on that commit while waiting (Python 3.12.14, versions pinned in poetry.lock / prek.toml):

  • PENDULUM_EXTENSIONS=0 python -m pytest -q tests: 1882 passed, 5 skipped (master: 1868 passed, 5 skipped)
  • python -m mypy: no issues in 228 files
  • ruff check and ruff format --check: clean

Two things I would like your call on:

  1. fix: pickling DateTime #909 fixes pickling doesn't preserve DST offset #908 by restoring through self.create, which re-resolves the wall time rather than rebuilding the instance. Measured on master with that patch applied: a zoneinfo.ZoneInfo tzinfo comes back as FixedTimezone/Timezone, and the wall time that does not exist (2024-03-10 02:30 in America/Chicago, only reachable through the constructor) moves to 01:30 and its timestamp shifts by an hour. This PR rebuilds from the raw state instead. If you would rather land fix: pickling DateTime #909, the regression tests here apply as they are, and I do not mind either way who lands the fix.

  2. pendulum.Time._getstate() has the same shape and also loses fold: Time(1, 30, tzinfo=..., fold=1) pickles back with fold=0. Would you like that folded into this PR or sent separately?

I will report the CI results once the workflows have run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pickling doesn't preserve DST offset

1 participant