> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation/pywebscrapr-logging.md).

# Logging

How PyWebScrapr 0.1.8 reports downloads and skipped pages or images through the standard pywebscrapr logger, the levels it uses, and how to enable INFO.

PyWebScrapr does not raise on a failed page or image: it **skips the item and logs it**. That makes logging the place to see what a run actually did.

## The logger

| Item        | Value                                                                                |
| ----------- | ------------------------------------------------------------------------------------ |
| Logger name | `pywebscrapr`                                                                        |
| Framework   | Python's standard [`logging`](https://docs.python.org/3/library/logging.html) module |
| Levels used | `INFO` for successful downloads and intentional skips, `WARNING` for failures        |
| Handlers    | None added by the library; records propagate to the root logger                      |

Because the library only emits records, nothing appears for `INFO` messages unless you configure logging. With no configuration at all, Python's last-resort handler still shows `WARNING` and above on stderr.

## Enable INFO output

```python
import logging

logging.basicConfig(level=logging.INFO)

from pywebscrapr import scrape_text

scrape_text(links_array=['https://example.com'], follow_child_links=True)
```

That single line shows downloads as well as skipped items. In 0.1.7, "Downloaded" and "Ignored" messages were printed to stdout; in 0.1.8 they are `INFO` records on the `pywebscrapr` logger.

## What gets logged

| Level     | Message                                                                    | Meaning                                                                                                                                                                   |
| --------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INFO`    | `Downloaded: <url> -> <path>`                                              | An image passed validation and was written to `<path>`.                                                                                                                   |
| `INFO`    | `Ignored due to size constraints: <url>`                                   | The image's dimensions fall outside `min_width`/`min_height`/`max_width`/`max_height`.                                                                                    |
| `INFO`    | `Skipped non-image content: <url> (<content-type>)`                        | The response was not an `image/*` type.                                                                                                                                   |
| `INFO`    | `Skipped due to small content size: <url>`                                 | The image response was too small to be a real image.                                                                                                                      |
| `INFO`    | `Skipped non-text content: <url> (<content-type>)`                         | A page returned a non-textual type (an image, PDF and so on) and was not parsed as HTML.                                                                                  |
| `WARNING` | `Skipped <url>: <reason>`                                                  | A page could not be scraped: unsupported scheme, blocked private address, DNS failure, timeout, HTTP 4xx/5xx, oversized response, too many redirects, or a parsing error. |
| `WARNING` | `Skipped image <url>: <reason>`                                            | An image could not be downloaded or saved: connection error, oversized file, corrupt or unrecognised image data, decompression bomb.                                      |
| `WARNING` | `Skipped <url>: child link redirected to a different domain (<final-url>)` | A child link left the site after redirects and was not scraped.                                                                                                           |
| `WARNING` | `Skipped invalid URL <url>: <reason>`                                      | A link could not be normalised, for example a malformed port.                                                                                                             |
| `WARNING` | `Skipped link that is not a string: <entry>`                               | A `links_array` entry was not a string and was ignored.                                                                                                                   |

{% hint style="info" %}
`print_progress=True` still writes `Scraped N pages: <url>` to **stdout**. It is a progress indicator, not a log record, and stays separate from the logger.
{% endhint %}

## Recipes

### Info level for the whole program

```python
import logging
logging.basicConfig(level=logging.INFO)
```

### Info level for PyWebScrapr only

Useful when your own application logs a lot at `INFO`:

```python
import logging

logging.basicConfig(level=logging.WARNING)
logging.getLogger('pywebscrapr').setLevel(logging.INFO)
```

### Log to a file

```python
import logging

logger = logging.getLogger('pywebscrapr')
logger.setLevel(logging.INFO)

handler = logging.FileHandler('scrape.log', encoding='utf-8')
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
logger.addHandler(handler)
logger.propagate = False   # avoid duplicate records from the root logger
```

### Silence the library

```python
import logging
logging.getLogger('pywebscrapr').setLevel(logging.CRITICAL)
```

Skipped items still do not raise; they are simply not reported.

{% hint style="warning" %}
Log records contain full URLs, including query strings. If those can contain tokens or personal data, route the `pywebscrapr` logger to a filtered handler or keep its level at `WARNING` (or higher) in shared log sinks.
{% endhint %}
