> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-known-limitations.md).

# Known Limitations

What PyWebScrapr 0.1.8 deliberately does not do: robots.txt, retries, JavaScript rendering, lazy-loaded images, SVG, streaming output and DNS rebinding protection.

PyWebScrapr is a small, dependency-light scraper for the HTML a server returns. The following are deliberate limitations, not bugs — plan around them.

## Crawling and fetching

* **`robots.txt` is not checked.** The package does not read or obey `robots.txt`. Make sure your use of a site follows its terms and policies, and honour the site's rules yourself.
* **There are no retries or backoff.** A failed request is skipped and logged. Transient errors — a timeout, a 503, a dropped connection — are not retried. If you need retries, call the scraper again for the affected URLs.
* **Only `http` and `https` are fetched.** Other schemes are refused.
* **Child links must stay on the same host and port**, including after redirects. Seed links may redirect anywhere.
* **Private addresses are allowed by default.** Set `block_private_addresses=True` to refuse loopback, private and link-local targets. The address check runs before the connection, so it does **not** protect against DNS rebinding; for untrusted input, also restrict network egress. See [Private addresses and SSRF](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md#private-addresses-and-ssrf).

## Parsing and images

* **JavaScript is not run.** Only the HTML the server returns is parsed. Content rendered client-side (React, Vue, infinite scroll and similar) is not visible to the scraper.
* **Lazy-loaded images are not followed.** Only an `<img>` element's `src` is used. `data-src`, `srcset`, `data-srcset`, `<picture>` `source` sets and CSS background images are ignored.
* **`data:` image URIs are skipped**, as are images without a `src`.
* **SVG images are skipped**, because Pillow cannot open them. Only raster formats Pillow recognises (PNG, JPEG, GIF, WEBP, BMP, TIFF and so on) are saved.
* **Very small image responses are treated as error pages.** A response under 500 bytes is skipped rather than saved.
* **Non-text child pages are not parsed.** If a child link returns an image, PDF or other non-textual type, it is skipped.

## Output

* **Output is held in memory** until the crawl finishes. Large crawls therefore use memory proportional to the total scraped text.
* **Output files are written at the end of the run**, once, and are not written atomically. If the process is killed mid-write, the file can be incomplete.
* **Files from an earlier run are overwritten.** Images with the same name as a file already in `save_folder` are replaced, exactly as in 0.1.7. Within a single run, name clashes get a short hash suffix so two new images never overwrite each other.
* **CSV and JSON only contain pages with content.** Empty pages are skipped in every format.

## Versions and testing

* **Python 3.6 to 3.8 are not supported** and were not tested. The package requires Python 3.9 or later; CI covers 3.9, 3.12 and 3.13.
* **The live test is off by default.** Setting `PYWEBSCRAPR_LIVE_TESTS=1` adds one test that fetches `https://example.com/`, so it needs internet access. The other 39 tests run fully offline against local fixture servers.

See also [Upgrading to 0.1.8](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-upgrading.md) for the behaviour changes in this release and [Options & Safety](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-options-and-safety.md) for the limits you can configure.
