[johanbeytell1]

Summary

Release 0.1.8: security hardening, bug fixes, an offline test suite, and CI/publishing changes. Full changelog is in README.md under Changes in 0.1.8.

Breaking changes

• Python 3.9+ is now required (was 3.6+ in metadata).
• Failed pages and images are skipped and logged (logger pywebscrapr) instead of aborting the run.
• max_links_to_follow counts child links only; 0 means no child links.
• rate_limit is now global across threads.
• Child links must stay on the same domain after redirects.
• Result URL values are final URLs after redirects.
• Default User-Agent is PyWebScrapr/0.1.8 (...) (pass user_agent=None for the requests default).
• Duplicate similarity is measured against the larger of the two texts.

Security

• Timeouts on every request (default (10, 30)), overridable.
• Response size caps (max_page_bytes, max_image_bytes), enforced from Content-Length and while streaming.
• Redirects validated hop by hop (max_redirects); only http/https fetched.
• New block_private_addresses flag (default False, so private hosts remain allowed by default).
• CSV formula injection neutralised.
• Safe, unique image filenames; corrupt and decompression-bomb images skipped.
• Dependency floors: requests>=2.32.4 (CVE-2024-47081), pillow>=10.3.0 (CVE-2024-28219).
• CI: actions pinned to commit SHAs, permissions: contents: read, no persisted checkout credentials, PyPI token only in the publish job.

Bug fixes

• Inline markup no longer glues words; nested containers no longer repeat text; <head>/<title>/<script>/<style>/<template>/comments excluded; <br> gives line breaks.
• links mode returns img/video/audio/source URLs (absolute, http(s) only); unseen mode includes script data.
• 4xx/5xx pages skipped instead of scraped.
• Each page and each image fetched once.
• Images without src skipped; empty image names no longer crash.
• Fragment and host/port variants de-duplicated.
• Output is deterministic (discovery order); de-duplication runs after the crawl.
• Links files: UTF-8 (BOM tolerated), blank lines ignored.
• Charset from HTTP header or <meta charset>; <base href> honoured.
• Invalid options raise ValueError early.

Packaging, CI, tests

• Version defined once in pywebscrapr/__init__.py (__version__); setup.py reads it.
• bs4 replaced by beautifulsoup4; build with python -m build.
• Workflow runs the test suite on Python 3.9, 3.12 and 3.13 for pushes and PRs. Publishing runs only for tags that pass the tests and match __version__, using the existing PYPI_API_TOKEN secret.
• tests.py rewritten: 39 offline tests against local fixture servers, plus an opt-in live test (PYWEBSCRAPR_LIVE_TESTS=1).

Verification

• python -m unittest -v tests: 39 offline tests pass, 1 live test skipped by default. Run three times to check the timing-sensitive tests (timeout, rate limit).
• Original failing cases from the code review were re-run against the new code. Results match the expected fixes.
• python -m build produces pywebscrapr-0.1.8 sdist and wheel; the wheel contains only the package.
• pyflakes clean on the package and tests.

Notes for reviewers

• The GitHub Actions workflow has not run yet. This PR is the first run. Please check the Test job on all three Python versions before merging.
• The code was tested locally on Python 3.11 only.
• Known limitations are listed in the README (robots.txt not checked, no retries, no lazy-loaded images, no JS rendering, block_private_addresses does not stop DNS rebinding).
• No release is created by merging this PR. Publishing happens only when a matching tag (0.1.8) is pushed.