`, `<style>`, `<template>` and comments are excluded. Expect different (cleaner) output from earlier versions. * **Duplicate detection measures similarity against the larger of the two paragraphs.** The old measure was against the earlier paragraph, which let a short item such as "Home" delete unrelated content. Results for paragraphs of similar length are essentially unchanged. * **`max_links_to_follow` counts child links only.** Seed links are always scraped. Previously seeds counted toward the limit and `0` meant "unlimited"; now `0` means "follow no child links". * **Child links must stay on the same domain after redirects.** Seed links may still redirect anywhere. * **Result URLs are final URLs.** The `URL` field in CSV and JSON output is the address after redirects. * **Default User-Agent.** Requests now send `PyWebScrapr/0.1.8 (+https://github.com/infinitode/pywebscrapr)`. Pass `user_agent=None` to keep the requests default. * **Invalid options raise `ValueError` straight away**, for example an unknown `elements_to_scrape`, `max_workers < 1`, or a non-positive timeout. * **Image file extensions match the detected format** (for example `photo.jpeg` is saved as `photo.jpg` when the image really is a JPEG). * **Links files are read as UTF-8** (a byte-order mark is tolerated). Blank lines and surrounding whitespace are ignored. * **`output_file` can be `None`** to skip the plain-text file. The default is still `output.txt`. * Private helpers were replaced. They were never part of the documented API, but the old names are no longer importable. * **Security** * **Timeouts on every request.** The default is `timeout=(10, 30)` (connect, read seconds). Previously a server that never answered could hang the run forever. Pass `timeout=None` only if you accept that risk. * **Response size limits.** `max_page_bytes` (default 10 MB) and `max_image_bytes` (default 20 MB) are enforced from `Content-Length` and again while the body streams. Previously the whole response was loaded into memory with no cap. * **Redirects are checked hop by hop.** Each redirect target is validated for scheme and, when enabled, address. `max_redirects` (default 10) bounds the chain. * **Only `http` and `https` are fetched.** `file:`, `ftp:`, `javascript:` and other schemes are refused. * **New `block_private_addresses` flag** (default `False`, so private and internal hosts are allowed as before). Set it to `True` to refuse URLs that resolve to loopback, private, link-local, CGNAT or other non-public addresses. This protects against a crawled page pointing the scraper at internal services, such as an `<img src="http://169.254.169.254/...">`. Limitation: the check runs before the connection is made, so it does not defend against DNS rebinding; keep the flag on for untrusted inputs and run the scraper with network egress controls where that matters. * **CSV formula injection is neutralised.** Cells beginning with `=`, `+`, `-`, `@`, tab or carriage return get a leading apostrophe, so spreadsheet programs will not evaluate scraped text as formulas. * **Safe image file names.** Names are restricted to letters, digits, `.`, `_` and `-`; Windows reserved names are prefixed; empty names fall back to `image`. Two images that would share a name in the same run are stored under different names instead of overwriting each other. Files left over from an earlier run with the same name are still overwritten, as before. * **Dependency floors.** `requests>=2.32.4` includes the fix for CVE-2024-47081 (a `.netrc` credential leak). `pillow>=10.3.0` includes the fix for CVE-2024-28219 (a buffer overflow in `_imagingcms`). * **CI hardening.** Every GitHub Action is pinned to a commit SHA. Workflows default to `permissions: contents: read`, checkout does not persist credentials, and the PyPI token is only available to the publish job. * **Bug fixes** * **Text extraction** joins inline markup correctly: `Hello <b>world</b>` gives `Hello world`, not `Helloworld`. * **Nested containers no longer duplicate text.** `<div><div><p>x</p></div></div>` yields `x` once, not three times. * **Line breaks.** `<br>` produces a line break in text output. * **`links` mode** now returns images, video, audio and `<source>` URLs as well as anchors (it previously returned only anchors, and the image and video branches never ran). URLs are returned absolute, not relative, and only `http`/`https` URLs are kept. * **`unseen` mode** includes `<script>` data as documented. It previously read only `<meta>` tags. * **Error pages are skipped.** 4xx and 5xx responses are no longer saved as page content. * **Each page is fetched once.** With `follow_child_links`, a page was previously downloaded twice (once for its content and once to find its links). * **Each image is downloaded once.** Images were previously requested twice (once to check the size and once to save). The check and the save could also see different responses. * **Images without a `src` are skipped.** They previously resolved to the page's own URL and triggered extra requests. * **Image names no longer crash the run.** A URL with an empty file name (such as `/?format=png`) used to raise `IsADirectoryError` and abort everything. * **Decompression bombs and corrupt images are skipped.** A header declaring an enormous image previously raised `DecompressionBombError` outside the error handling and aborted the run. * **Rate limit is global.** `rate_limit` is now the minimum spacing between requests across all threads. Previously each thread slept on its own, so `max_workers=5` could send five requests per interval. * **Fragments no longer cause repeat crawls.** `page#one` and `page#two` are the same page. Host case and default ports are normalised too. * **Output is deterministic.** Results come out in discovery order regardless of thread count, and duplicate detection runs after the crawl. Previously duplicate detection depended on thread timing and could let near-duplicates through. * **Failures no longer discard output.** One unreachable host or malformed URL used to abort the crawl before anything was written. Output is now written for every page that succeeded. * **Blank lines in links files** no longer crash the run. * **Character encoding.** Pages are decoded from their declared charset (HTTP header or `<meta charset>`). Previously requests decoded `text/*` responses without a charset as ISO-8859-1, which mangled UTF-8 pages. * **`<base href>`** is honoured when resolving relative links and images. * **Non-text responses** (images, PDFs and similar) are no longer parsed as HTML when they are reached as child links. * **`follow_child_links`** respects `max_depth`, which is new in this release. * **New options** * `scrape_text` and `scrape_images` (keyword-only): `timeout`, `max_page_bytes`, `max_depth`, `block_private_addresses`, `user_agent`, `max_redirects`. * `scrape_images` (keyword-only): `max_image_bytes`. * `scrape_text`: `output_file=None` skips the plain-text file. * `pywebscrapr.__version__` exposes the package version. * **Packaging, CI and tests** * The version is defined once, in `pywebscrapr/__init__.py`, and `setup.py` reads it. The `{{VERSION_PLACEHOLDER}}` substitution is gone. * `bs4` is replaced by `beautifulsoup4` in `requirements.txt` and `setup.py`. Build with `python -m build`. * The workflow runs the offline test suite on Python 3.9, 3.12 and 3.13 for every push and pull request. Publishing to PyPI happens only for tags, only after the tests pass, and only if the tag matches `__version__`. * `tests.py` is rewritten: 39 offline tests run against local fixture servers, so no internet access is needed. Set `PYWEBSCRAPR_LIVE_TESTS=1` to also run a live check against example.com. * `.gitignore` covers `output/` and `*.egg-info/`. * **Documentation** * This documentation set now describes the 0.1.8 options, the four `elements_to_scrape` modes and the CSV/JSON output formats. * Added [Upgrading to 0.1.8](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md), [Options & Safety](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md), [Logging](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-logging.md) and [Known Limitations](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-known-limitations.md) pages, and refreshed the [`scrape_text` and `scrape_images` reference](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference/pywebscrapr-functions.md). * **0.1.7**: Optimized `is_similar_regex` text similarity and duplicate checking by caching text normalization and word splitting, drastically improving duplicate text detection speed by up to 25x (\~96% CPU time reduction). * **0.1.6**: * Added progress indicators to both `scrape_images` and `scrape_text` to provide real-time feedback on scraping progress. * Implemented multithreading to improve performance by scraping multiple pages concurrently. * Added a `rate_limit` parameter to both scraping functions to control the request frequency and prevent server overload. * Refactored the concurrency model to ensure that child links are also scraped concurrently. * **0.1.5**: Added new parameters to control following child links, and added a new export format, `json`. * **0.1.4**: Added new parameters to the `scrape_text` function for added control and flexibility. * **0.1.3**: Added support for handling different types of images on websites. Added improved error handling. * **0.1.2**: Updated PyPI project description. * **0.1.1**: New parameters for image extraction, and optimized extraction by using BeautifulSoup4's `SoupStrainer`. * **0.1.0**: Initial release. ## Installation and requirements ### Requirements | Requirement | Minimum | Why it matters | | ---------------- | ---------- | -------------------------------------------------------------------------------- | | Python | **3.9** | 0.1.8 dropped Python 3.6–3.8. The CI matrix runs 3.9, 3.12 and 3.13. | | `requests` | **2.32.4** | 2.32.4 includes the fix for CVE-2024-47081 (a `.netrc` credential leak). | | `beautifulsoup4` | **4.12** | The distribution name is `beautifulsoup4`; earlier releases listed it as `bs4`. | | `pillow` | **10.3.0** | 10.3.0 includes the fix for CVE-2024-28219 (a buffer overflow in `_imagingcms`). | Upgrade your Python interpreter first if it is older than 3.9 — the package will not install or run there. ### Install from PyPI ```bash pip install pywebscrapr ``` Verify the installed version: ```python import pywebscrapr print(pywebscrapr.__version__) # '0.1.8' ``` ### Install from source ```bash git clone https://github.com/infinitode/pywebscrapr.git cd pywebscrapr python -m pip install -r requirements.txt python -m pip install . ``` To build a distribution, use `python -m build` (the older `setup.py sdist bdist_wheel` route is gone). The version is defined once in `pywebscrapr/__init__.py` and read by `setup.py`, so there is no version placeholder to substitute. {% hint style="warning" %} PyWebScrapr 0.1.8 requires **Python 3.9 or later**. Python 3.6, 3.7 and 3.8 are no longer supported. {% endhint %} *** ## Quick start ### Scrape text ```python from pywebscrapr import scrape_text # Specify links in a file or a list links_file = 'links.txt' links_array = ['https://example.com/page1', 'https://example.com/page2'] # Scrape text and save it to 'output.txt' scrape_text(links_file=links_file, links_array=links_array, output_file='output.txt') ``` `links_file` takes precedence when both a file and an array are given. A links file is read as UTF-8; blank lines are ignored. ### Scrape images ```python from pywebscrapr import scrape_images # Specify links in a file or a list links_file = 'image_links.txt' links_array = ['https://example.com/image1.jpg', 'https://example.com/image2.png'] # Scrape images and save them to the 'images' folder scrape_images(links_file=links_file, links_array=links_array, save_folder='images') ``` ### Crawl a site with safety limits ```python from pywebscrapr import scrape_images, scrape_text scrape_text( links_array=['https://example.com'], output_file='output.txt', follow_child_links=True, max_links_to_follow=20, # child links only; seed links are always scraped max_depth=2, # 0 = seed pages only timeout=(5, 20), # (connect, read) seconds max_page_bytes=5 * 1024**2, # 5 MB rate_limit=0.5, # shared by all threads ) scrape_images( links_array=['https://example.com/gallery'], save_folder='images', max_image_bytes=10 * 1024**2, block_private_addresses=True, # refuse internal/loopback targets user_agent='MyCrawler/1.0 (+https://example.com/bot)', ) ``` ### See what was skipped ```python import logging logging.basicConfig(level=logging.INFO) # 'pywebscrapr' logger: INFO = downloads, WARNING = skipped items ``` ### Export CSV and JSON instead of plain text ```python from pywebscrapr import scrape_text scrape_text( links_array=['https://example.com'], output_file=None, # skip output.txt csv_output_file='pages.csv', json_output_file='pages.json', elements_to_scrape='links', ) ``` *** ## Documentation map * [Upgrading to 0.1.8](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md) — what changed since 0.1.7, and the migration checklist. * [Options & Safety](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md) — timeouts, size caps, redirects, crawl depth, rate limiting, `block_private_addresses` and `user_agent`. * [Text Modes & Output Formats](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-text-modes-and-output-formats.md) — the four `elements_to_scrape` modes, duplicate handling, and the plain-text, CSV and JSON outputs. * [Logging](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-logging.md) — the `pywebscrapr` logger, its levels and what each message means. * [Known Limitations](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-known-limitations.md) — what the package deliberately does not do. * [PyWebScrapr Reference](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference.md) — imports, signatures and module constants. * [PyWebScrapr Functions](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference/pywebscrapr-functions.md) — full parameter reference for `scrape_text` and `scrape_images`. *** ## Testing The test suite is offline by default: 39 tests run against local fixture servers, so no internet access is needed. ```bash python -m unittest -v tests ``` Set `PYWEBSCRAPR_LIVE_TESTS=1` to add a single live check against `example.com`: ```bash PYWEBSCRAPR_LIVE_TESTS=1 python -m unittest -v tests ``` CI runs the offline suite on Python 3.9, 3.12 and 3.13 for pull requests, pushes to `main` and tag pushes. Publishing to PyPI runs only for a tag push, only after the test jobs pass, and only when the tag matches `pywebscrapr.__version__`; the `PYPI_API_TOKEN` repository secret is available to the publish job alone. *** ## License PyWebScrapr is released under the **MIT License (Modified)**. The modified clause allows derivative works, but any substantial change must be clearly distinguished from the original and distributed under a different name. See the [LICENSE](https://github.com/infinitode/pywebscrapr/blob/main/LICENSE) file for the full text.

> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation.md). # PyWebScrapr Package Documentation Package documentation for PyWebScrapr 0.1.8, an open-source Python library for scraping text, links and images from web pages. Covers installation, safety limits, output formats, logging and the full ![Python Version](https://img.shields.io/badge/python-3.9%2B-blue.svg) ![PyPI Version](https://img.shields.io/pypi/v/pywebscrapr) ![License](https://img.shields.io/badge/license-MIT%20%28Modified%29-brightgreen.svg) ![Dependencies](https://img.shields.io/badge/dependencies-requests%20%C2%B7%20beautifulsoup4%20%C2%B7%20pillow-blue.svg) PyWebScrapr is an open-source Python library for **web scraping**. It extracts visible text, page metadata or links from web pages, or downloads their images, and can optionally crawl same-domain child links. Timeouts, response size caps, redirect handling, rate limiting, duplicate removal and CSV/JSON export are built in. {% hint style="info" %} **This documentation covers PyWebScrapr 0.1.8.** That release changes some defaults and behaviour: Python 3.9 or later is required, failed pages and images are skipped and logged instead of raising, `rate_limit` is now global, and every request has a timeout and a size cap. If you are upgrading from 0.1.7, read [Upgrading to 0.1.8](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md) first. {% endhint %} ## At a glance | Item | Value | | -------------------- | ------------------------------------------------------------------------------ | | Latest version | **0.1.8** (`pywebscrapr.__version__`) | | Python | **3.9 or later** (tested in CI on 3.9, 3.12 and 3.13) | | Runtime dependencies | `requests>=2.32.4`, `beautifulsoup4>=4.12`, `pillow>=10.3.0` | | Install | `pip install pywebscrapr` | | Public API | `scrape_text`, `scrape_images`, `normalize_text`, `is_similar_regex` | | Logger | `pywebscrapr` | | Repository | [github.com/infinitode/pywebscrapr](https://github.com/infinitode/pywebscrapr) | | License | MIT (Modified) | ## Changelog * **0.1.8 (Latest)**: Security, reliability and correctness release. It changes some defaults and behaviour, so read **Breaking changes** before upgrading. * **Breaking changes** * **Python 3.9 or later is required** (the package previously claimed 3.6+). Pillow 10.3+ and requests 2.32.4+ are the new minimums, and the CI matrix tests 3.9, 3.12 and 3.13. * **Failed pages and images are skipped and logged instead of raising.** A bad URL, timeout, DNS failure, HTTP error, oversized response or corrupt image no longer aborts the run or discards the output. Failures are logged as warnings and downloads as info messages on the `pywebscrapr` logger. Previously "Downloaded" and "Ignored" messages were printed; to see them again, use `logging.basicConfig(level=logging.INFO)`. * **Text output changes.** Words are no longer glued together, nested containers no longer repeat their text, and ``, ``, `<style>`, `<template>` and comments are excluded. Expect different (cleaner) output from earlier versions. * **Duplicate detection measures similarity against the larger of the two paragraphs.** The old measure was against the earlier paragraph, which let a short item such as "Home" delete unrelated content. Results for paragraphs of similar length are essentially unchanged. * **`max_links_to_follow` counts child links only.** Seed links are always scraped. Previously seeds counted toward the limit and `0` meant "unlimited"; now `0` means "follow no child links". * **Child links must stay on the same domain after redirects.** Seed links may still redirect anywhere. * **Result URLs are final URLs.** The `URL` field in CSV and JSON output is the address after redirects. * **Default User-Agent.** Requests now send `PyWebScrapr/0.1.8 (+https://github.com/infinitode/pywebscrapr)`. Pass `user_agent=None` to keep the requests default. * **Invalid options raise `ValueError` straight away**, for example an unknown `elements_to_scrape`, `max_workers < 1`, or a non-positive timeout. * **Image file extensions match the detected format** (for example `photo.jpeg` is saved as `photo.jpg` when the image really is a JPEG). * **Links files are read as UTF-8** (a byte-order mark is tolerated). Blank lines and surrounding whitespace are ignored. * **`output_file` can be `None`** to skip the plain-text file. The default is still `output.txt`. * Private helpers were replaced. They were never part of the documented API, but the old names are no longer importable. * **Security** * **Timeouts on every request.** The default is `timeout=(10, 30)` (connect, read seconds). Previously a server that never answered could hang the run forever. Pass `timeout=None` only if you accept that risk. * **Response size limits.** `max_page_bytes` (default 10 MB) and `max_image_bytes` (default 20 MB) are enforced from `Content-Length` and again while the body streams. Previously the whole response was loaded into memory with no cap. * **Redirects are checked hop by hop.** Each redirect target is validated for scheme and, when enabled, address. `max_redirects` (default 10) bounds the chain. * **Only `http` and `https` are fetched.** `file:`, `ftp:`, `javascript:` and other schemes are refused. * **New `block_private_addresses` flag** (default `False`, so private and internal hosts are allowed as before). Set it to `True` to refuse URLs that resolve to loopback, private, link-local, CGNAT or other non-public addresses. This protects against a crawled page pointing the scraper at internal services, such as an `<img src="http://169.254.169.254/...">`. Limitation: the check runs before the connection is made, so it does not defend against DNS rebinding; keep the flag on for untrusted inputs and run the scraper with network egress controls where that matters. * **CSV formula injection is neutralised.** Cells beginning with `=`, `+`, `-`, `@`, tab or carriage return get a leading apostrophe, so spreadsheet programs will not evaluate scraped text as formulas. * **Safe image file names.** Names are restricted to letters, digits, `.`, `_` and `-`; Windows reserved names are prefixed; empty names fall back to `image`. Two images that would share a name in the same run are stored under different names instead of overwriting each other. Files left over from an earlier run with the same name are still overwritten, as before. * **Dependency floors.** `requests>=2.32.4` includes the fix for CVE-2024-47081 (a `.netrc` credential leak). `pillow>=10.3.0` includes the fix for CVE-2024-28219 (a buffer overflow in `_imagingcms`). * **CI hardening.** Every GitHub Action is pinned to a commit SHA. Workflows default to `permissions: contents: read`, checkout does not persist credentials, and the PyPI token is only available to the publish job. * **Bug fixes** * **Text extraction** joins inline markup correctly: `Hello <b>world</b>` gives `Hello world`, not `Helloworld`. * **Nested containers no longer duplicate text.** `<div><div><p>x</p></div></div>` yields `x` once, not three times. * **Line breaks.** `<br>` produces a line break in text output. * **`links` mode** now returns images, video, audio and `<source>` URLs as well as anchors (it previously returned only anchors, and the image and video branches never ran). URLs are returned absolute, not relative, and only `http`/`https` URLs are kept. * **`unseen` mode** includes `<script>` data as documented. It previously read only `<meta>` tags. * **Error pages are skipped.** 4xx and 5xx responses are no longer saved as page content. * **Each page is fetched once.** With `follow_child_links`, a page was previously downloaded twice (once for its content and once to find its links). * **Each image is downloaded once.** Images were previously requested twice (once to check the size and once to save). The check and the save could also see different responses. * **Images without a `src` are skipped.** They previously resolved to the page's own URL and triggered extra requests. * **Image names no longer crash the run.** A URL with an empty file name (such as `/?format=png`) used to raise `IsADirectoryError` and abort everything. * **Decompression bombs and corrupt images are skipped.** A header declaring an enormous image previously raised `DecompressionBombError` outside the error handling and aborted the run. * **Rate limit is global.** `rate_limit` is now the minimum spacing between requests across all threads. Previously each thread slept on its own, so `max_workers=5` could send five requests per interval. * **Fragments no longer cause repeat crawls.** `page#one` and `page#two` are the same page. Host case and default ports are normalised too. * **Output is deterministic.** Results come out in discovery order regardless of thread count, and duplicate detection runs after the crawl. Previously duplicate detection depended on thread timing and could let near-duplicates through. * **Failures no longer discard output.** One unreachable host or malformed URL used to abort the crawl before anything was written. Output is now written for every page that succeeded. * **Blank lines in links files** no longer crash the run. * **Character encoding.** Pages are decoded from their declared charset (HTTP header or `<meta charset>`). Previously requests decoded `text/*` responses without a charset as ISO-8859-1, which mangled UTF-8 pages. * **`<base href>`** is honoured when resolving relative links and images. * **Non-text responses** (images, PDFs and similar) are no longer parsed as HTML when they are reached as child links. * **`follow_child_links`** respects `max_depth`, which is new in this release. * **New options** * `scrape_text` and `scrape_images` (keyword-only): `timeout`, `max_page_bytes`, `max_depth`, `block_private_addresses`, `user_agent`, `max_redirects`. * `scrape_images` (keyword-only): `max_image_bytes`. * `scrape_text`: `output_file=None` skips the plain-text file. * `pywebscrapr.__version__` exposes the package version. * **Packaging, CI and tests** * The version is defined once, in `pywebscrapr/__init__.py`, and `setup.py` reads it. The `{{VERSION_PLACEHOLDER}}` substitution is gone. * `bs4` is replaced by `beautifulsoup4` in `requirements.txt` and `setup.py`. Build with `python -m build`. * The workflow runs the offline test suite on Python 3.9, 3.12 and 3.13 for every push and pull request. Publishing to PyPI happens only for tags, only after the tests pass, and only if the tag matches `__version__`. * `tests.py` is rewritten: 39 offline tests run against local fixture servers, so no internet access is needed. Set `PYWEBSCRAPR_LIVE_TESTS=1` to also run a live check against example.com. * `.gitignore` covers `output/` and `*.egg-info/`. * **Documentation** * This documentation set now describes the 0.1.8 options, the four `elements_to_scrape` modes and the CSV/JSON output formats. * Added [Upgrading to 0.1.8](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md), [Options & Safety](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md), [Logging](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-logging.md) and [Known Limitations](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-known-limitations.md) pages, and refreshed the [`scrape_text` and `scrape_images` reference](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference/pywebscrapr-functions.md). * **0.1.7**: Optimized `is_similar_regex` text similarity and duplicate checking by caching text normalization and word splitting, drastically improving duplicate text detection speed by up to 25x (\~96% CPU time reduction). * **0.1.6**: * Added progress indicators to both `scrape_images` and `scrape_text` to provide real-time feedback on scraping progress. * Implemented multithreading to improve performance by scraping multiple pages concurrently. * Added a `rate_limit` parameter to both scraping functions to control the request frequency and prevent server overload. * Refactored the concurrency model to ensure that child links are also scraped concurrently. * **0.1.5**: Added new parameters to control following child links, and added a new export format, `json`. * **0.1.4**: Added new parameters to the `scrape_text` function for added control and flexibility. * **0.1.3**: Added support for handling different types of images on websites. Added improved error handling. * **0.1.2**: Updated PyPI project description. * **0.1.1**: New parameters for image extraction, and optimized extraction by using BeautifulSoup4's `SoupStrainer`. * **0.1.0**: Initial release. ## Installation and requirements ### Requirements | Requirement | Minimum | Why it matters | | ---------------- | ---------- | -------------------------------------------------------------------------------- | | Python | **3.9** | 0.1.8 dropped Python 3.6–3.8. The CI matrix runs 3.9, 3.12 and 3.13. | | `requests` | **2.32.4** | 2.32.4 includes the fix for CVE-2024-47081 (a `.netrc` credential leak). | | `beautifulsoup4` | **4.12** | The distribution name is `beautifulsoup4`; earlier releases listed it as `bs4`. | | `pillow` | **10.3.0** | 10.3.0 includes the fix for CVE-2024-28219 (a buffer overflow in `_imagingcms`). | Upgrade your Python interpreter first if it is older than 3.9 — the package will not install or run there. ### Install from PyPI ```bash pip install pywebscrapr ``` Verify the installed version: ```python import pywebscrapr print(pywebscrapr.__version__) # '0.1.8' ``` ### Install from source ```bash git clone https://github.com/infinitode/pywebscrapr.git cd pywebscrapr python -m pip install -r requirements.txt python -m pip install . ``` To build a distribution, use `python -m build` (the older `setup.py sdist bdist_wheel` route is gone). The version is defined once in `pywebscrapr/__init__.py` and read by `setup.py`, so there is no version placeholder to substitute. {% hint style="warning" %} PyWebScrapr 0.1.8 requires **Python 3.9 or later**. Python 3.6, 3.7 and 3.8 are no longer supported. {% endhint %} *** ## Quick start ### Scrape text ```python from pywebscrapr import scrape_text # Specify links in a file or a list links_file = 'links.txt' links_array = ['https://example.com/page1', 'https://example.com/page2'] # Scrape text and save it to 'output.txt' scrape_text(links_file=links_file, links_array=links_array, output_file='output.txt') ``` `links_file` takes precedence when both a file and an array are given. A links file is read as UTF-8; blank lines are ignored. ### Scrape images ```python from pywebscrapr import scrape_images # Specify links in a file or a list links_file = 'image_links.txt' links_array = ['https://example.com/image1.jpg', 'https://example.com/image2.png'] # Scrape images and save them to the 'images' folder scrape_images(links_file=links_file, links_array=links_array, save_folder='images') ``` ### Crawl a site with safety limits ```python from pywebscrapr import scrape_images, scrape_text scrape_text( links_array=['https://example.com'], output_file='output.txt', follow_child_links=True, max_links_to_follow=20, # child links only; seed links are always scraped max_depth=2, # 0 = seed pages only timeout=(5, 20), # (connect, read) seconds max_page_bytes=5 * 1024**2, # 5 MB rate_limit=0.5, # shared by all threads ) scrape_images( links_array=['https://example.com/gallery'], save_folder='images', max_image_bytes=10 * 1024**2, block_private_addresses=True, # refuse internal/loopback targets user_agent='MyCrawler/1.0 (+https://example.com/bot)', ) ``` ### See what was skipped ```python import logging logging.basicConfig(level=logging.INFO) # 'pywebscrapr' logger: INFO = downloads, WARNING = skipped items ``` ### Export CSV and JSON instead of plain text ```python from pywebscrapr import scrape_text scrape_text( links_array=['https://example.com'], output_file=None, # skip output.txt csv_output_file='pages.csv', json_output_file='pages.json', elements_to_scrape='links', ) ``` *** ## Documentation map * [Upgrading to 0.1.8](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md) — what changed since 0.1.7, and the migration checklist. * [Options & Safety](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md) — timeouts, size caps, redirects, crawl depth, rate limiting, `block_private_addresses` and `user_agent`. * [Text Modes & Output Formats](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-text-modes-and-output-formats.md) — the four `elements_to_scrape` modes, duplicate handling, and the plain-text, CSV and JSON outputs. * [Logging](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-logging.md) — the `pywebscrapr` logger, its levels and what each message means. * [Known Limitations](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-known-limitations.md) — what the package deliberately does not do. * [PyWebScrapr Reference](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference.md) — imports, signatures and module constants. * [PyWebScrapr Functions](/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference/pywebscrapr-functions.md) — full parameter reference for `scrape_text` and `scrape_images`. *** ## Testing The test suite is offline by default: 39 tests run against local fixture servers, so no internet access is needed. ```bash python -m unittest -v tests ``` Set `PYWEBSCRAPR_LIVE_TESTS=1` to add a single live check against `example.com`: ```bash PYWEBSCRAPR_LIVE_TESTS=1 python -m unittest -v tests ``` CI runs the offline suite on Python 3.9, 3.12 and 3.13 for pull requests, pushes to `main` and tag pushes. Publishing to PyPI runs only for a tag push, only after the test jobs pass, and only when the tag matches `pywebscrapr.__version__`; the `PYPI_API_TOKEN` repository secret is available to the publish job alone. *** ## License PyWebScrapr is released under the **MIT License (Modified)**. The modified clause allows derivative works, but any substantial change must be clearly distinguished from the original and distributed under a different name. See the [LICENSE](https://github.com/infinitode/pywebscrapr/blob/main/LICENSE) file for the full text.