> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/pywebscrapr-package-documentation.md).

# PyWebScrapr Package Documentation

## Changelog

* **0.1.6 (Latest)**: 
  * Added progress indicators to both `scrape_images` and `scrape_text` to provide real-time feedback on scraping progress.
  * Implemented multithreading to improve performance by scraping multiple pages concurrently.
  * Added a `rate_limit` parameter to both scraping functions to control the request frequency and prevent server overload.
  * Refactored the concurrency model to ensure that child links are also scraped concurrently.
* **0.1.5**: Added new parameters to control following child links, and added a new export format, `json`.
* **0.1.4**: Added new parameters to the `scrape_text` function for added control and flexibility.
* **0.1.3**: Added support for handling different types of images on websites. Added improved error handling.
* **0.1.2**: Updated PYPI project description.
* **0.1.1**: New parameters for image extraction, and optimized extraction by using BeautifulSoup4's `SoupStrainer`.
* **0.1.0**: Initial release.

## Installation

You can install PyWebScrapr using PyPi, please make sure that you are using Python 3.6 or later before installing PyWebScrapr:

```bash
pip install pywebscrapr
```

***

## Example Usage

### Text scraping

```python
from pywebscrapr import scrape_text

# Specify links in a file or list
links_file = 'links.txt'
links_array = ['https://example.com/page1', 'https://example.com/page2']

# Scrape text and save to the 'output.txt' file
scrape_text(links_file=links_file, links_array=links_array, output_file='output.txt')
```

### Image scraping

```python
from pywebscrapr import scrape_images

# Specify links in a file or list
links_file = 'image_links.txt'
links_array = ['https://example.com/image1.jpg', 'https://example.com/image2.png']

# Scrape images and save to the 'images' folder
scrape_images(links_file=links_file, links_array=links_array, save_folder='images')
```