Release 0.1.8: security hardening, bug fixes, offline tests and CI

.github/workflows/build_and_publish.yml

@@ -1,34 +1,64 @@
-name: Publish Python 🐍 distributions 📦 to PyPI
+name: Test and publish to PyPI
 
 on:
   push:
-    tags:
-       - '*'
+    branches: [main]
+    tags: ['*']
+  pull_request:
+
+# Least privilege by default. Jobs that need more say so explicitly.
+permissions:
+  contents: read
 
 jobs:
-  build-n-publish:
-    name: Build and publish Python 🐍 distributions 📦 to PyPI
+  test:
+    name: Test (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ['3.9', '3.12', '3.13']
+    steps:
+      - name: Check out code
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          persist-credentials: false
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0  # v7.1.0
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: python -m pip install -r requirements.txt
+      - name: Run tests (offline)
+        run: python -m unittest -v tests
+
+  build-and-publish:
+    name: Build and publish to PyPI
+    needs: test
+    # Only tag pushes publish. Pull requests and branch pushes stop after the tests.
+    if: startsWith(github.ref, 'refs/tags/')
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
-    - name: Set up Python 3.12
-      uses: actions/setup-python@v3
-      with:
-        python-version: '3.12'
-    - name: Install pypa/setuptools
-      run: >-
-        python -m
-        pip install setuptools wheel
-    - name: Extract tag name
-      id: tag
-      run: echo ::set-output name=TAG_NAME::$(echo $GITHUB_REF | cut -d / -f 3)
-    - name: Update version in setup.py
-      run: >-
-        sed -i "s/{{VERSION_PLACEHOLDER}}/${{ steps.tag.outputs.TAG_NAME }}/g" setup.py
-    - name: Build a binary wheel
-      run: >-
-        python setup.py sdist bdist_wheel
-    - name: Publish distribution 📦 to PyPI
-      uses: pypa/gh-action-pypi-publish@master
-      with:
-        password: ${{ secrets.PYPI_API_TOKEN }}
+      - name: Check out code
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          persist-credentials: false
+      - name: Set up Python
+        uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0  # v7.1.0
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: python -m pip install -r requirements.txt
+      - name: Check that the release tag matches the package version
+        env:
+          TAG_NAME: ${{ github.ref_name }}
+        run: |
+          python -c "import os, pywebscrapr; tag = os.environ['TAG_NAME']; version = pywebscrapr.__version__; assert tag == version, 'tag %s does not match __version__ %s' % (tag, version); print('Releasing', version)"
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install build
+          python -m build
+      - name: Publish distribution to PyPI
+        uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086  # v1.14.3
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}

.gitignore

@@ -4,4 +4,6 @@ dist/
 build/
 venv/
 json_output.json
-output.txt
+output.txt
+output/
+*.egg-info/

README.md

@@ -7,6 +7,71 @@
 
 An open-source Python library for web scraping tasks. Includes support for both text and image scraping.
 
+## Changes in 0.1.8
+
+0.1.8 is a security, reliability and correctness release. It changes some defaults and behaviour, so read **Breaking changes** before upgrading.
+
+### Breaking changes
+- **Python 3.9 or later is required** (the package previously claimed 3.6+). Pillow 10.3+ and requests 2.32.4+ are the new minimums, and the CI matrix tests 3.9, 3.12 and 3.13.
+- **Failed pages and images are skipped and logged instead of raising.** A bad URL, timeout, DNS failure, HTTP error, oversized response or corrupt image no longer aborts the run or discards the output. Failures are logged as warnings and downloads as info messages on the `pywebscrapr` logger. Previously "Downloaded" and "Ignored" messages were printed; to see them again, use `logging.basicConfig(level=logging.INFO)`.
+- **Text output changes.** Words are no longer glued together, nested containers no longer repeat their text, and `<head>`, `<title>`, `<style>`, `<template>` and comments are excluded. Expect different (cleaner) output from earlier versions.
+- **Duplicate detection measures similarity against the larger of the two paragraphs.** The old measure was against the earlier paragraph, which let a short item such as "Home" delete unrelated content. Results for paragraphs of similar length are essentially unchanged.
+- **`max_links_to_follow` counts child links only.** Seed links are always scraped. Previously seeds counted toward the limit and `0` meant "unlimited"; now `0` means "follow no child links".
+- **Child links must stay on the same domain after redirects.** Seed links may still redirect anywhere.
+- **Result URLs are final URLs.** The `URL` field in CSV and JSON output is the address after redirects.
+- **Default User-Agent.** Requests now send `PyWebScrapr/0.1.8 (+https://github.com/infinitode/pywebscrapr)`. Pass `user_agent=None` to keep the requests default.
+- **Invalid options raise `ValueError` straight away**, for example an unknown `elements_to_scrape`, `max_workers < 1`, or a non-positive timeout.
+- **Image file extensions match the detected format** (for example `photo.jpeg` is saved as `photo.jpg` when the image really is a JPEG).
+- **Links files are read as UTF-8** (a byte-order mark is tolerated). Blank lines and surrounding whitespace are ignored.
+- **`output_file` can be `None`** to skip the plain-text file. The default is still `output.txt`.
+- Private helpers `_download_image` and `_check_image_dimensions` were replaced. They were never documented, but they are no longer importable under those names.
+
+### Security
+- **Timeouts on every request.** The default is `timeout=(10, 30)` (connect, read seconds). Previously a server that never answered could hang the run forever. Pass `timeout=None` only if you accept that risk.
+- **Response size limits.** `max_page_bytes` (default 10 MB) and `max_image_bytes` (default 20 MB) are enforced from `Content-Length` and again while the body streams. Previously the whole response was loaded into memory with no cap.
+- **Redirects are checked hop by hop.** Each redirect target is validated for scheme and, when enabled, address. `max_redirects` (default 10) bounds the chain.
+- **Only `http` and `https` are fetched.** `file:`, `ftp:`, `javascript:` and other schemes are refused.
+- **New `block_private_addresses` flag** (default `False`, so private and internal hosts are allowed as before). Set it to `True` to refuse URLs that resolve to loopback, private, link-local, CGNAT or other non-public addresses. This protects against a crawled page pointing the scraper at internal services, such as an `<img src="http://169.254.169.254/...">`. Limitation: the check runs before the connection is made, so it does not defend against DNS rebinding; keep the flag on for untrusted inputs and run the scraper with network egress controls where that matters.
+- **CSV formula injection is neutralised.** Cells beginning with `=`, `+`, `-`, `@`, tab or carriage return get a leading apostrophe, so spreadsheet programs will not evaluate scraped text as formulas.
+- **Safe image file names.** Names are restricted to letters, digits, `.`, `_` and `-`; Windows reserved names are prefixed; empty names fall back to `image`. Two images that would share a name in the same run are stored under different names instead of overwriting each other. Files left over from an earlier run with the same name are still overwritten, as before.
+- **Dependency floors.** `requests>=2.32.4` includes the fix for CVE-2024-47081 (a `.netrc` credential leak). `pillow>=10.3.0` includes the fix for CVE-2024-28219 (a buffer overflow in `_imagingcms`).
+- **CI hardening.** Every GitHub Action is pinned to a commit SHA. Workflows default to `permissions: contents: read`, checkout does not persist credentials, and the PyPI token is only available to the publish job.
+
+### Bug fixes
+- **Text extraction** joins inline markup correctly: `Hello <b>world</b>` gives `Hello world`, not `Helloworld`.
+- **Nested containers no longer duplicate text.** `<div><div><p>x</p></div></div>` yields `x` once, not three times.
+- **Line breaks.** `<br>` produces a line break in text output.
+- **`links` mode** now returns images, video, audio and `<source>` URLs as well as anchors (it previously returned only anchors, and the image and video branches never ran). URLs are returned absolute, not relative, and only `http`/`https` URLs are kept.
+- **`unseen` mode** includes `<script>` data as documented. It previously read only `<meta>` tags.
+- **Error pages are skipped.** 4xx and 5xx responses are no longer saved as page content.
+- **Each page is fetched once.** With `follow_child_links`, a page was previously downloaded twice (once for its content and once to find its links).
+- **Each image is downloaded once.** Images were previously requested twice (once to check the size and once to save). The check and the save could also see different responses.
+- **Images without a `src` are skipped.** They previously resolved to the page's own URL and triggered extra requests.
+- **Image names no longer crash the run.** A URL with an empty file name (such as `/?format=png`) used to raise `IsADirectoryError` and abort everything.
+- **Decompression bombs and corrupt images are skipped.** A header declaring an enormous image previously raised `DecompressionBombError` outside the error handling and aborted the run.
+- **Rate limit is global.** `rate_limit` is now the minimum spacing between requests across all threads. Previously each thread slept on its own, so `max_workers=5` could send five requests per interval.
+- **Fragments no longer cause repeat crawls.** `page#one` and `page#two` are the same page. Host case and default ports are normalised too.
+- **Output is deterministic.** Results come out in discovery order regardless of thread count, and duplicate detection runs after the crawl. Previously duplicate detection depended on thread timing and could let near-duplicates through.
+- **Failures no longer discard output.** One unreachable host or malformed URL used to abort the crawl before anything was written. Output is now written for every page that succeeded.
+- **Blank lines in links files** no longer crash the run.
+- **Character encoding.** Pages are decoded from their declared charset (HTTP header or `<meta charset>`). Previously requests decoded `text/*` responses without a charset as ISO-8859-1, which mangled UTF-8 pages.
+- **`<base href>`** is honoured when resolving relative links and images.
+- **Non-text responses** (images, PDFs and similar) are no longer parsed as HTML when they are reached as child links.
+- **`follow_child_links`** respects `max_depth`, which is new in this release.
+
+### New options
+- `scrape_text` and `scrape_images` (keyword-only): `timeout`, `max_page_bytes`, `max_depth`, `block_private_addresses`, `user_agent`, `max_redirects`.
+- `scrape_images` (keyword-only): `max_image_bytes`.
+- `scrape_text`: `output_file=None` skips the plain-text file.
+- `pywebscrapr.__version__` exposes the package version.
+
+### Packaging, CI and tests
+- The version is defined once, in `pywebscrapr/__init__.py`, and `setup.py` reads it. The `{{VERSION_PLACEHOLDER}}` substitution is gone.
+- `bs4` is replaced by `beautifulsoup4` in `requirements.txt` and `setup.py`. Build with `python -m build`.
+- The workflow runs the offline test suite on Python 3.9, 3.12 and 3.13 for every push and pull request. Publishing to PyPI happens only for tags, only after the tests pass, and only if the tag matches `__version__`.
+- `tests.py` is rewritten: 39 offline tests run against local fixture servers, so no internet access is needed. Set `PYWEBSCRAPR_LIVE_TESTS=1` to also run a live check against example.com.
+- `.gitignore` covers `output/` and `*.egg-info/`.
+
 ## Changes in 0.1.7:
 - Optimized `is_similar_regex` text similarity and duplicate checking by caching text normalization and word splitting, drastically improving duplicate text detection speed by up to 25x (~96% CPU time reduction).
 
@@ -45,17 +110,7 @@ pip install pywebscrapr
 
 ## Supported Python Versions
 
-PyWebScrapr supports the following Python versions:
-
-- Python 3.6
-- Python 3.7
-- Python 3.8
-- Python 3.9
-- Python 3.10
-- Python 3.11
-- Python 3.12/Later (Preferred)
-
-Please ensure that you have one of these Python versions installed before using PyWebScrapr. PyWebScrapr may not work as expected on lower versions of Python than the supported.
+PyWebScrapr supports Python 3.9 and later. Python 3.9, 3.12 and 3.13 are tested in CI.
 
 ## Features
 
@@ -92,6 +147,77 @@ links_array = ['https://example.com/image1.jpg', 'https://example.com/image2.png
 scrape_images(links_file=links_file, links_array=links_array, save_folder='images')
 ```
 
+### Options: timeouts, limits and safety
+
+Every request has a timeout and a size cap by default. Override them with keyword arguments:
+
+```python
+from pywebscrapr import scrape_images, scrape_text
+
+scrape_text(
+    links_array=['https://example.com'],
+    output_file='output.txt',
+    follow_child_links=True,
+    max_links_to_follow=20,      # child links only; seed links are always scraped
+    max_depth=2,                 # child-link depth (0 = seed pages only)
+    timeout=(5, 20),             # (connect, read) seconds; None disables timeouts
+    max_page_bytes=5 * 1024**2,  # largest page accepted
+    rate_limit=0.5,              # minimum seconds between requests, shared by all threads
+)
+
+scrape_images(
+    links_array=['https://example.com/gallery'],
+    save_folder='images',
+    max_image_bytes=10 * 1024**2,
+    block_private_addresses=True,  # refuse loopback, private and link-local targets
+    user_agent='MyCrawler/1.0 (+https://example.com/bot)',
+)
+```
+
+Skipped pages and images are logged. To see them, configure logging:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)  # logger name: 'pywebscrapr'
+```
+
+### Text scraping modes and output formats
+
+`scrape_text(..., elements_to_scrape=...)` accepts four modes:
+
+| Mode | What it returns |
+|---|---|
+| `'text'` (default) | Visible text, one line per block-level element. Inline markup is joined correctly (`Hello <b>world</b>` gives `Hello world`). `<br>` gives a line break. `<head>`, `<title>`, `<script>`, `<style>`, `<noscript>`, `<template>` and comments are excluded. |
+| `'content'` | The `content` attribute of `<meta>` tags. |
+| `'unseen'` | `<meta>` `content` values and `<script>` text. |
+| `'links'` | Absolute `http`/`https` URLs from `<a href>`, `<img src>`, `<video src>`, `<audio src>` and `<source src>`. |
+
+Output files:
+- **Plain text** (`output_file`, default `output.txt`; pass `None` to skip): pages separated by a blank line.
+- **CSV** (`csv_output_file`): columns `URL` and `Content`. Cells that start with `=`, `+`, `-`, `@`, tab or carriage return get a leading apostrophe, so spreadsheet programs do not run them as formulas.
+- **JSON** (`json_output_file`): a list of `{"URL": ..., "Content": ...}` objects.
+
+The `URL` in CSV and JSON output is the final address after redirects. Pages are written in discovery order, so output is the same however many `max_workers` you use. Only pages with content appear in CSV and JSON.
+
+### Upgrading from 0.1.7
+
+1. Install Python 3.9 or later. Earlier versions of the package declared 3.6+, but 0.1.8 requires 3.9+.
+2. Expect skipped pages and images in the logs instead of exceptions. Add `logging.basicConfig(level=logging.INFO)` to see downloads as well as skips.
+3. Review `max_links_to_follow`. It now counts child links only, and `0` means no child links.
+4. Review `rate_limit`. It is now the minimum spacing between requests across all threads, so the effective rate can be lower than before.
+5. Pass `user_agent=None` if you need the old User-Agent header.
+6. Only pass `timeout=None`, `max_page_bytes=None` or `max_image_bytes=None` if you accept the risk of unbounded waits or downloads.
+
+### Known limitations
+
+- `robots.txt` is not checked. Make sure your use of a site follows its terms and policies.
+- There are no retries or backoff. A failed request is skipped and logged.
+- Lazy-loaded images (`data-src`, `srcset`) are not followed. Only `src` is used.
+- SVG images are skipped, because Pillow cannot open them.
+- JavaScript is not run. Only the HTML the server returns is parsed.
+- Output is held in memory until the crawl finishes, and the output files are written at the end.
+- `block_private_addresses` checks addresses before connecting, so it does not protect against DNS rebinding. For untrusted input, also restrict network egress.
+
 ## Contributing
 
 Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to PyWebScrapr, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/pywebscrapr).