> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md).

# Options & Safety

PyWebScrapr 0.1.8 options and safety limits: request timeouts, page and image size caps, redirects, crawl depth, global rate limiting, private address blocking and the User-Agent header.

PyWebScrapr 0.1.8 has safe defaults: every request has a timeout, every response has a size cap, redirects are bounded, and failed items are skipped rather than aborting the run. All of the controls below are **keyword-only** arguments on `scrape_text` and `scrape_images`.

```python
scrape_text(
    links_array=['https://example.com'],
    timeout=(10, 30),
    max_page_bytes=10 * 1024**2,
    max_depth=2,
    follow_child_links=True,
)
```

{% hint style="info" %}
Private and internal addresses are **allowed by default** so that intranet scraping keeps working. Set `block_private_addresses=True` when you scrape pages you do not control. See [Private addresses and SSRF](#private-addresses-and-ssrf).
{% endhint %}

## Defaults at a glance

| Option                    | Used by         | Default                                                          | Meaning                                                          |
| ------------------------- | --------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------- |
| `timeout`                 | both            | `(10, 30)`                                                       | `(connect, read)` seconds per request. `None` disables timeouts. |
| `max_page_bytes`          | both            | `10485760` (10 MB)                                               | Largest page or child-page body accepted.                        |
| `max_image_bytes`         | `scrape_images` | `20971520` (20 MB)                                               | Largest image body accepted.                                     |
| `max_redirects`           | both            | `10`                                                             | Redirects followed per request.                                  |
| `max_depth`               | both            | `None`                                                           | Child-link depth; `None` is unlimited.                           |
| `block_private_addresses` | both            | `False`                                                          | Refuse URLs that resolve to non-public addresses.                |
| `user_agent`              | both            | `PyWebScrapr/0.1.8 (+https://github.com/infinitode/pywebscrapr)` | `User-Agent` header sent with every request.                     |
| `rate_limit`              | both            | `0`                                                              | Minimum seconds between requests, shared by all threads.         |
| `max_workers`             | both            | `5`                                                              | Worker threads.                                                  |
| `follow_child_links`      | both            | `False`                                                          | Crawl same-domain child links.                                   |
| `max_links_to_follow`     | both            | `None`                                                           | Maximum child links followed; `None` is unlimited.               |

The defaults are also available as module constants, so you can reference them in your own configuration:

| Constant                              | Value                                                              |
| ------------------------------------- | ------------------------------------------------------------------ |
| `pywebscrapr.DEFAULT_TIMEOUT`         | `(10, 30)`                                                         |
| `pywebscrapr.DEFAULT_MAX_PAGE_BYTES`  | `10485760`                                                         |
| `pywebscrapr.DEFAULT_MAX_IMAGE_BYTES` | `20971520`                                                         |
| `pywebscrapr.DEFAULT_MAX_REDIRECTS`   | `10`                                                               |
| `pywebscrapr.DEFAULT_USER_AGENT`      | `'PyWebScrapr/0.1.8 (+https://github.com/infinitode/pywebscrapr)'` |

## Timeouts

`timeout` bounds how long a single request may take. It accepts:

* a number, used as both the connect and the read timeout, for example `timeout=15`;
* a `(connect, read)` tuple, for example `timeout=(5, 20)`;
* `None`, which disables timeouts.

The default is `(10, 30)`. Values must be positive numbers; anything else raises `ValueError`. `timeout=None` is allowed but not recommended: a server that never answers can then hang the run indefinitely.

```python
scrape_text(
    links_array=['https://example.com'],
    timeout=(5, 20),   # 5 s to connect, 20 s for each read
)
```

A request that times out is skipped and logged as a warning; the rest of the run continues.

## Response size caps

Two caps keep a single response from exhausting memory:

* `max_page_bytes` — the largest HTML page accepted. Default **10 MB** (`10485760` bytes). Applies to seed pages and child pages in both functions.
* `max_image_bytes` — the largest image accepted. Default **20 MB** (`20971520` bytes). Applies to `scrape_images`.

Each cap is enforced twice: from the `Content-Length` header when the server sends one, and again while the body streams, so a missing or inaccurate header does not bypass the limit. An oversized response is skipped and logged; it does not abort the run.

Pass `None` to disable a cap (not recommended), or a different integer number of bytes. Values must be `None` or an integer of at least `1`.

```python
scrape_text(
    links_array=['https://example.com'],
    max_page_bytes=5 * 1024**2,     # 5 MB
)

scrape_images(
    links_array=['https://example.com/gallery'],
    max_image_bytes=10 * 1024**2,   # 10 MB
)
```

## Redirects

Redirects are followed **one hop at a time** rather than by the HTTP client, so every target is validated before it is fetched:

* only `http` and `https` are fetched — `file:`, `ftp:`, `javascript:` and other schemes are refused;
* when `block_private_addresses=True`, each hop's address is checked as well;
* `max_redirects` (default `10`) bounds the chain. `0` means no redirects are followed at all.

A chain longer than `max_redirects` is skipped and logged.

The `URL` field written to CSV and JSON is the **final** address after redirects, not the address you passed in. See [Text Modes & Output Formats](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-text-modes-and-output-formats.md#output-formats).

{% hint style="warning" %}
Seed links may redirect to any domain. **Child links must stay on the same domain after redirects**; a child that redirects elsewhere is skipped and logged.
{% endhint %}

## Crawl depth and child links

Crawling is opt-in. With `follow_child_links=False` (the default) only the links you supply are scraped.

| Option                | Default | Behaviour                                                                                                                                                        |
| --------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `follow_child_links`  | `False` | Follow same-domain links found on scraped pages.                                                                                                                 |
| `max_depth`           | `None`  | Maximum child-link depth. `0` scrapes seed pages only, `1` adds their direct children, `None` is unlimited.                                                      |
| `max_links_to_follow` | `None`  | Maximum number of **child links** to follow. Seed links are always scraped, so this never limits your input list. `0` means no child links; `None` is unlimited. |

```python
scrape_text(
    links_array=['https://example.com/docs'],
    follow_child_links=True,
    max_depth=2,              # seeds, their children, and their children's children
    max_links_to_follow=50,   # 50 child links in total, across the whole crawl
)
```

Notes on crawling:

* Child links must resolve to the same host and port as the page that linked to them. A different host name or an explicitly different port counts as a different domain.
* `page#section` and `page#other` are treated as the same page: fragments are dropped. Host case and default ports (`:80` for `http`, `:443` for `https`) are normalised too.
* `<base href>` is honoured when resolving relative links.
* `max_links_to_follow` counts **child links only**. It used to count seeds as well, and `0` used to mean unlimited. Review the value when you upgrade — see [Upgrading to 0.1.8](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md).
* Pages are written in discovery order (seed order, then child links) regardless of `max_workers`.

## Rate limiting and concurrency

`rate_limit` is the **minimum spacing between requests, shared across all threads**. It accepts integers and floats, and `0` (the default) disables it.

```python
scrape_text(
    links_array=['https://example.com'],
    rate_limit=0.5,   # at most two requests per second, in total
    max_workers=5,    # five threads may still work in parallel between slots
)
```

A `rate_limit` of `0.5` means no two requests start less than half a second apart, however many workers are running. In 0.1.7 the delay was applied per thread, so five workers could send five requests per interval.

`max_workers` must be an integer of at least `1` and defaults to `5`. It changes how quickly a crawl completes, not the order of the output. `print_progress=True` prints `Scraped N pages: <url>` to stdout as pages complete and is independent of logging.

## Private addresses and SSRF

By default, private and internal hosts are allowed — this is deliberate, so that scraping an intranet or a local development server keeps working.

Set `block_private_addresses=True` to refuse any URL that resolves to loopback, private, link-local, CGNAT or another non-public address. The check applies to **seed pages, child pages, every redirect hop and image downloads**, so a crawled page cannot steer the scraper at an internal service:

```html
<!-- Without the flag, this could make the scraper fetch an internal endpoint -->
<img src="http://169.254.169.254/latest/meta-data/">
```

```python
scrape_images(
    links_array=['https://untrusted.example.com'],
    block_private_addresses=True,
)
```

{% hint style="danger" %}
**Limitation:** the address check runs *before* the connection, so it does not protect against DNS rebinding. For untrusted input, keep the flag on **and** run the scraper with network egress controls.
{% endhint %}

The rest of the security posture:

* Only `http` and `https` are fetched. Other schemes are refused.
* Redirects are followed one hop at a time, with `max_redirects` (default `10`) as the bound.
* CSV exports are formula-escaped: cells that begin with `=`, `+`, `-`, `@`, tab or carriage return get a leading apostrophe. See [Output formats](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-text-modes-and-output-formats.md#output-formats).
* Images are saved under safe file names derived from the URL, with the extension of the **detected** format. See [`scrape_images`](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-reference/pywebscrapr-functions.md#scrape-images).

## User agent

Every request sends `PyWebScrapr/0.1.8 (+https://github.com/infinitode/pywebscrapr)` by default. Set your own string to identify your bot, or pass `None` to let `requests` use its default `User-Agent` header.

```python
scrape_text(
    links_array=['https://example.com'],
    user_agent='MyCrawler/1.0 (+https://example.com/bot)',
)
```

## Option validation

Invalid options fail fast with a `ValueError` instead of being silently ignored.

| Option                              | Accepted values                                                                    |
| ----------------------------------- | ---------------------------------------------------------------------------------- |
| `max_workers`                       | integer `>= 1`                                                                     |
| `rate_limit`                        | number `>= 0` (floats allowed)                                                     |
| `timeout`                           | positive number, a two-item `(connect, read)` tuple of positive numbers, or `None` |
| `max_links_to_follow`               | `None` or integer `>= 0`                                                           |
| `max_depth`                         | `None` or integer `>= 0`                                                           |
| `max_redirects`                     | integer `>= 0`                                                                     |
| `max_page_bytes`, `max_image_bytes` | `None` or integer `>= 1`                                                           |
| `similarity_threshold`              | number greater than `0` and at most `1`                                            |
| `elements_to_scrape`                | `'text'`, `'content'`, `'unseen'` or `'links'`                                     |

At least one of `links_file` or `links_array` must be provided, otherwise `scrape_text` and `scrape_images` raise `ValueError`.

## Complete example

```python
from pywebscrapr import scrape_images, scrape_text

scrape_text(
    links_array=['https://example.com'],
    output_file='output.txt',
    follow_child_links=True,
    max_links_to_follow=20,      # child links only; seed links are always scraped
    max_depth=2,                 # 0 = seed pages only
    timeout=(5, 20),             # (connect, read) seconds
    max_page_bytes=5 * 1024**2,  # 5 MB
    rate_limit=0.5,              # shared by all threads
)

scrape_images(
    links_array=['https://example.com/gallery'],
    save_folder='images',
    max_image_bytes=10 * 1024**2,
    block_private_addresses=True,   # refuse internal/loopback targets
    user_agent='MyCrawler/1.0 (+https://example.com/bot)',
)
```
