> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md).

# Upgrading to 0.1.8

Upgrading from PyWebScrapr 0.1.7 to 0.1.8: what changed, what breaks, and the step-by-step migration checklist for Python version, options and behaviour.

0.1.8 is a security, reliability and correctness release. Most calls keep working, but some defaults changed: **Python 3.9+ is required**, failures are skipped instead of raised, `rate_limit` is global, and limits and timeouts are now on by default.

```bash
pip install --upgrade pywebscrapr
```

```python
import pywebscrapr
print(pywebscrapr.__version__)   # '0.1.8'
```

## Migration checklist

1. **Use Python 3.9 or later.** Upgrade the interpreter before the package; 3.6–3.8 are no longer supported.
2. **Expect skipped-item log lines instead of exceptions.** Add `logging.basicConfig(level=logging.INFO)` if you want to see them, or handle the absence of a page in your output.
3. **Review any `max_links_to_follow` values.** It now counts child links only, and `0` means "no child links" rather than "unlimited".
4. **Review any `rate_limit` values.** It is now shared across all threads, so the effective request rate can be lower than before.
5. **Pass `user_agent=None`** if you need the old requests default `User-Agent` header.
6. **Only pass `max_page_bytes=None`, `max_image_bytes=None` or `timeout=None`** if you accept unbounded downloads or waits. The timeouts and size caps are on by default for a reason.

```python
import logging

from pywebscrapr import scrape_text

logging.basicConfig(level=logging.INFO)   # see downloads and skips

scrape_text(
    links_array=['https://example.com'],
    follow_child_links=True,
    max_links_to_follow=20,   # 20 child links; seeds are always scraped
    rate_limit=0.5,           # global: at most two requests per second
    timeout=(10, 30),         # explicit, but these are the defaults
)
```

## What changed

| Area                  | 0.1.7                                                                                       | 0.1.8                                                    | What to do                                                                                                                                                                                   |
| --------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Python                | 3.6+ (metadata)                                                                             | **3.9+**                                                 | Upgrade Python before upgrading the package.                                                                                                                                                 |
| Failures              | Some errors aborted the run (for example a bad URL, a connection error, a corrupt image)    | Failed pages and images are **skipped and logged**       | Check the logs (see [Logging](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-logging.md)) if you relied on exceptions. |
| Logging output        | "Downloaded"/"Ignored" printed to stdout                                                    | Sent to the `pywebscrapr` logger (INFO/WARNING)          | Enable with `logging.basicConfig(level=logging.INFO)`.                                                                                                                                       |
| `max_links_to_follow` | Counted seed links too; `0` meant unlimited                                                 | Counts **child links only**; `0` means no child links    | If you used it to cap a crawl with many seeds, the cap is now larger by the number of seeds.                                                                                                 |
| `remove_duplicates`   | Similarity against the previously seen paragraph (short items could delete long paragraphs) | Similarity against the larger paragraph                  | Results for similar-length paragraphs are essentially unchanged. Re-check thresholds if you tuned them.                                                                                      |
| Text output           | Glued words, repeated nested text                                                           | Clean text                                               | Expect different output; this is a fix.                                                                                                                                                      |
| Redirects             | Child links followed to any domain                                                          | Child links must stay on the same domain after redirects | Seed links may still redirect anywhere.                                                                                                                                                      |
| `URL` in CSV/JSON     | Requested URL                                                                               | **Final URL** after redirects                            | Matters only if you map results back to input URLs.                                                                                                                                          |
| User-Agent            | requests default                                                                            | `PyWebScrapr/0.1.8 (...)`                                | Pass `user_agent=None` to keep the old header.                                                                                                                                               |
| Timeouts              | None (could hang forever)                                                                   | `(10, 30)`                                               | Override with `timeout=`.                                                                                                                                                                    |
| Size limits           | None                                                                                        | 10 MB pages, 20 MB images                                | Override with `max_page_bytes=` / `max_image_bytes=`.                                                                                                                                        |
| Charset               | requests guessed (ISO-8859-1 for `text/*` without a charset)                                | Taken from the HTTP header or `<meta charset>`           | UTF-8 pages no longer come out mojibake.                                                                                                                                                     |
| Images with no `src`  | Resolved to the page URL and re-requested                                                   | Skipped                                                  | None.                                                                                                                                                                                        |
| Images, each          | Downloaded twice                                                                            | Downloaded once                                          | None.                                                                                                                                                                                        |
| Child pages           | Fetched twice                                                                               | Fetched once                                             | None.                                                                                                                                                                                        |
| `rate_limit`          | Per thread (5 workers could send 5 requests per interval)                                   | Global                                                   | If you tuned `max_workers` against `rate_limit`, the effective rate is now lower and as documented.                                                                                          |
| Link file             | Read in platform encoding, blank lines included (a blank line crashed the run)              | UTF-8 (BOM tolerated), blanks skipped                    | None.                                                                                                                                                                                        |
| `output_file`         | Always written                                                                              | `None` skips it                                          | Default unchanged.                                                                                                                                                                           |
| `elements_to_scrape`  | Unknown value silently produced nothing                                                     | `ValueError`                                             | Fix typos.                                                                                                                                                                                   |
| Invalid options       | Some accepted silently                                                                      | `ValueError` early                                       | None for valid calls.                                                                                                                                                                        |
| Image file extension  | Kept from the URL                                                                           | From the detected format                                 | A `.jpeg` URL can now be saved as `.jpg`.                                                                                                                                                    |
| Private helpers       | Two private image helpers                                                                   | Replaced                                                 | Not part of the public API.                                                                                                                                                                  |

## New options to adopt

All of these are **keyword-only** — pass them by name.

| Option                    | Functions       | Default                   | Use it to                                                       |
| ------------------------- | --------------- | ------------------------- | --------------------------------------------------------------- |
| `timeout`                 | both            | `(10, 30)`                | Bound connect and read time per request.                        |
| `max_page_bytes`          | both            | `10485760` (10 MB)        | Cap page size before parsing.                                   |
| `max_image_bytes`         | `scrape_images` | `20971520` (20 MB)        | Cap image size before decoding.                                 |
| `max_depth`               | both            | `None`                    | Bound how deep a child-link crawl goes (`0` = seeds only).      |
| `block_private_addresses` | both            | `False`                   | Refuse loopback, private and link-local targets.                |
| `user_agent`              | both            | `PyWebScrapr/0.1.8 (...)` | Identify your bot, or pass `None` to keep the requests default. |
| `max_redirects`           | both            | `10`                      | Bound the redirect chain per request.                           |

`scrape_text` also accepts `output_file=None` to skip the plain-text file. See [Options & Safety](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md) for the full behaviour of each option.

## What stayed the same

* Both functions still return `None`.
* The four `elements_to_scrape` names are unchanged: `'text'`, `'content'`, `'unseen'`, `'links'`.
* `normalize_text(text)` and `is_similar_regex(new_text, seen_texts, threshold=0.8)` keep their signatures; only the similarity measure inside `is_similar_regex` changed.
* `print_progress=True` still prints `Scraped N pages: <url>` to stdout.
* `save_folder` is created if missing, and files from an earlier run with the same name are still overwritten.
* `output.txt` is still the default plain-text output.
* `scrape_images` still filters on `min_width`, `min_height`, `max_width` and `max_height`, where `0` or `None` means no limit.

{% hint style="info" %}
If you need the 0.1.7 behaviour while you migrate, pin the previous release with `pip install pywebscrapr==0.1.7`. Remember that 0.1.7 has no timeouts, no size caps and per-thread rate limiting.
{% endhint %}
