## Query

CrawlerRunConfig arun configuration object per crawl run.

## Response

### Sources:
Important: you can fetch the full content of any source using the fetch_url_content tool

#### (unclecode/crawl4ai/docs/md_v2/complete-sdk-reference.md)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/complete-sdk-reference.md] (Score: 0.71)
- .
10. **`enable_rate_limiting`**:
   - If `True`, enables rate limiting for batch processing.
   - Requires `rate_limit_config` to be set.
11. **`memory_threshold_percent`**:
    - The memory threshold (as a percentage) to monitor.
    - If exceeded, the crawler will pause or slow down.
12. **`check_interval`**:
    - The interval (in seconds) to check system resources.
    - Affects how often memory and CPU usage are monitored.
13. **`max_session_permit`**:
    - The maximum number of concurrent crawl sessions.
    - Helps prevent overwhelming the system.
14. **`url_matcher`** & **`match_mode`**:
    - Enable URL-specific configurations when used with `arun_many()`.
    - Set `url_matcher` to patterns (glob, function, or list) to match specific URLs.
    - Use `match_mode` (OR/AND) to control how multiple patterns combine.
15. **`display_mode`**:
    - The display mode for progress information (`DETAILED`, `BRIEF`, etc.).
    - Affects how much information is printed during the crawl.
### Helper Methods
The `clone()` method is particularly useful for creating variations of your crawler configuration:
```python
# Create a base configuration
base_config = CrawlerRunConfig(

- ## 3. Basic Configuration (Light Introduction)
1. **`BrowserConfig`**: Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.).
2. **`CrawlerRunConfig`**: Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.).
```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_conf = BrowserConfig(headless=True)  # or False to see the browser
    run_conf = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS
    )

    async with AsyncWebCrawler(config=browser_conf) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_conf
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())
```
> IMPORTANT: By default cache mode is set to `CacheMode.BYPASS` to have fresh content. Set `CacheMode.ENABLED` to enable caching.
## 4. Generating Markdown Output
- **`result.markdown`**:
- **`result.markdown.fit_markdown`**:
  The same content after applying any configured **content filter** (e.g., `PruningContentFilter`).
- # `arun()` Parameter Guide (New Approach)
In Crawl4AI’s **latest** configuration model, nearly all parameters that once went directly to `arun()` are now part of **`CrawlerRunConfig`**. When calling `arun()`, you provide:
```python
await crawler.arun(
    url="https://example.com",
    config=my_run_config
)
```
Below is an organized look at the parameters that can go inside `CrawlerRunConfig`, divided by their functional areas. For **Browser** settings (e.g., `headless`, `browser_type`), see [BrowserConfig](./parameters.md).
## 1. Core Usage
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    run_config = CrawlerRunConfig(
        verbose=True,            # Detailed logging
        cache_mode=CacheMode.ENABLED,  # Use normal read/write cache
        check_robots_txt=True,   # Respect robots.txt rules
        # ... other parameters
    )

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_config
        )

        # Check if blocked by robots.txt
        if not result.success and result.status_code == 403:
            print(f"Error: {result.error_message}")
```
- `verbose=True` logs each crawl step.
- `cache_mode` decides how to read/write the local crawl cache.
## 2. Cache Control
**`cache_mode`** (default: `CacheMode.ENABLED`)
Use a built-in enum from `CacheMode`:
- `ENABLED`: Normal caching—reads if available, writes if missing.
- ### B) **Caching & Session**
| **Parameter**           | **Type / Default**     | **What It Does**                                                                                |
|-------------------------|------------------------|------------------------------------------------------------------------------------------------------------------------------|
| **`cache_mode`**        | `CacheMode or None`    | Controls how caching is handled (`ENABLED`, `BYPASS`, `DISABLED`, etc.). If `None`, typically defaults to `ENABLED`.          |
| **`session_id`**        | `str or None`          | Assign a unique ID to reuse a single browser session across multiple `arun()` calls.                                          |
| **`bypass_cache`**      | `bool` (False)         | If `True`, acts like `CacheMode.BYPASS`.                                                                                |
| **`disable_cache`**     | `bool` (False)         | If `True`, acts like `CacheMode.DISABLED`.                                                                                |
| **`no_cache_read`**     | `bool` (False)         | If `True`, acts like `CacheMode.WRITE_ONLY` (writes cache but never reads).                                                  |
| **`no_cache_write`**    | `bool` (False)         | If `True`, acts like `CacheMode.READ_ONLY` (reads cache but never writes).                                                   |
### C) **Page Navigation & Timing**

- asyncio.run(main())
```
- We define a **`BrowserConfig`** with Firefox, no headless, and `verbose=True`.
- We define a **`CrawlerRunConfig`** that **bypasses cache**, uses a **CSS** extraction schema, has a `word_count_threshold=15`, etc.
- We pass them to `AsyncWebCrawler(config=...)` and `arun(url=..., config=...)`.
## 7. Best Practices & Migration Notes
1. **Use** `BrowserConfig` for **global** settings about the browser’s environment.
2. **Use** `CrawlerRunConfig` for **per-crawl** logic (caching, content filtering, extraction strategies, wait conditions).
3. **Avoid** legacy parameters like `css_selector` or `word_count_threshold` directly in `arun()`. Instead:
   ```python
   run_cfg = CrawlerRunConfig(css_selector=".main-content", word_count_threshold=20)
   result = await crawler.arun(url="...", config=run_cfg)
   ```
## 8. Summary
- **Constructor** accepts **`BrowserConfig`** (or defaults).
- **`arun(url, config=CrawlerRunConfig)`** is the main method for single-page crawls.
- **`arun_many(urls, config=CrawlerRunConfig)`** handles concurrency across multiple URLs.
- For advanced lifecycle control, use `start()` and `close()` explicitly.
- If you used `AsyncWebCrawler(browser_type="chromium", css_selector="...")`, move browser settings to `BrowserConfig(...)` and content/crawl logic to `CrawlerRunConfig(...)`.

#### (unclecode/crawl4ai/docs/md_v2/api/parameters.md)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/api/parameters.md] (Score: 0.69)
- ### C) **Caching & Session**

| **Parameter**           | **Type / Default**     | **What It Does**                                                                                |
|-------------------------|------------------------|------------------------------------------------------------------------------------------------------------------------------|
| **`cache_mode`**        | `CacheMode or None`    | Controls how caching is handled (`ENABLED`, `BYPASS`, `DISABLED`, etc.). If `None`, typically defaults to `ENABLED`.          |
| **`session_id`**        | `str or None`          | Assign a unique ID to reuse a single browser session across multiple `arun()` calls.                                          |
| **`bypass_cache`**      | `bool` (False)         | **Deprecated.** If `True`, acts like `CacheMode.BYPASS`. Use `cache_mode` instead.                                           |
| **`disable_cache`**     | `bool` (False)         | **Deprecated.** If `True`, acts like `CacheMode.DISABLED`. Use `cache_mode` instead.                                         |
| **`no_cache_read`**     | `bool` (False)         | **Deprecated.** If `True`, acts like `CacheMode.WRITE_ONLY` (writes cache but never reads). Use `cache_mode` instead.        |
| **`no_cache_write`**    | `bool` (False)         | **Deprecated.** If `True`, acts like `CacheMode.READ_ONLY` (reads cache but never writes). Use `cache_mode` instead.         |
| **`shared_data`**       | `dict or None` (None)  | Shared data to be passed between hooks and accessible across crawl operations.                                                |

Use these for controlling whether you read or write from a local content cache. Handy for large batch crawls or repeated site visits.

---

### D) **Page Navigation & Timing**

#### (unclecode/crawl4ai/docs/md_v2/api/arun.md)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/api/arun.md] (Score: 0.68)
- # `arun()` Parameter Guide (New Approach)

In Crawl4AI’s **latest** configuration model, nearly all parameters that once went directly to `arun()` are now part of **`CrawlerRunConfig`**. When calling `arun()`, you provide:

```python
await crawler.arun(
    url="https://example.com",
    config=my_run_config
)
```

Below is an organized look at the parameters that can go inside `CrawlerRunConfig`, divided by their functional areas. For **Browser** settings (e.g., `headless`, `browser_type`), see [BrowserConfig](./parameters.md).

---

## 1. Core Usage

```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    run_config = CrawlerRunConfig(
        verbose=True,            # Detailed logging
        cache_mode=CacheMode.ENABLED,  # Use normal read/write cache
        check_robots_txt=True,   # Respect robots.txt rules
        # ... other parameters
    )

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_config
        )

        # Check if blocked by robots.txt
        if not result.success and result.status_code == 403:
            print(f"Error: {result.error_message}")
```

**Key Fields**:
- `verbose=True` logs each crawl step.
- `cache_mode` decides how to read/write the local crawl cache.

---

## 2. Cache Control

**`cache_mode`** (default: `CacheMode.ENABLED`)
Use a built-in enum from `CacheMode`:

#### (unclecode/crawl4ai/llms.txt)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/llms.txt] (Score: 0.68)
- - [arun.md](https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/api/arun.md): Crawl4AI `arun()` Parameter Guide
The Crawl4AI documentation provides a comprehensive guide to the `arun()` parameters using the new approach with `CrawlerRunConfig`. The configuration model has changed, and nearly all parameters that once went directly to `arun()` are now part of `CrawlerRunConfig`. When calling `arun()`, you provide a URL and a `config` object of type `CrawlerRunConfig`. The guide is divided into sections covering core usage, cache control, content processing and selection, page navigation and timing, session management, screenshot and media options, extraction strategy, and best practices.

The guide provides various code examples to demonstrate how to use the `CrawlerRunConfig` object. For instance:

```python
run_config = CrawlerRunConfig(
    verbose=True,
    cache_mode=CacheMode.ENABLED,
    check_robots_txt=True,
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://example.com",
        config=run_config
    )
```

Additionally, the document includes a comprehensive example that combines many parameters:

- ```python
run_config = CrawlerRunConfig(
    # Core
    verbose=True,
    cache_mode=CacheMode.ENABLED,
    check_robots_txt=True,

    # Content
    word_count_threshold=10,
    css_selector="main.content",
    excluded_tags=["nav", "footer"],
    exclude_external_links=True,

    # Page & JS
    js_code="document.querySelector('.show-more')?.click();",
    wait_for="css:.loaded-block",
    page_timeout=30000,

    # Extraction
    extraction_strategy=JsonCssExtractionStrategy(schema),

    # Session
    session_id="persistent_session",

    # Media
    screenshot=True,
    pdf=True,

    # Anti-bot
    simulate_user=True,
    magic=True,
)
```

The document also mentions other code examples, including text processing, content selection, link handling, media filtering, and JavaScript execution.

KEY TERMS:
1. CrawlerRunConfig
2. AsyncWebCrawler
3. CacheMode
4. BrowserConfig
5. JsonCssExtractionStrategy
6. arun()
7. Caching
8. Content Processing
9. Page Navigation
10. Extraction Strategy

- [arun_many.md](https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/api/arun_many.md): `arun_many` Reference for Concurrent URL Crawling
The `arun_many` function is used for concurrent or batch crawling of multiple URLs. It is similar to the `arun` function but allows for advanced concurrency control through dispatchers and supports streaming results. The function takes a list of URLs, an optional configuration, and an optional dispatcher as parameters. It returns a `RunManyReturn` object containing a list of `CrawlResult` objects or an async generator if streaming is enabled. Key features include concurrency control, rate limiting, and memory-based adaptive throttling.

```python
# Basic Example (Batch Mode)
results = await crawler.arun_many(
    urls=["https://site1.com", "https://site2.com"],
    config=CrawlerRunConfig(stream=False)  # Default behavior
)

#### (unclecode/crawl4ai/docs/md_v2/core/quickstart.md)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/core/quickstart.md] (Score: 0.67)
- ## 3. Basic Configuration (Light Introduction)

Crawl4AI’s crawler can be heavily customized using two main classes:

1. **`BrowserConfig`**: Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.).
2. **`CrawlerRunConfig`**: Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.).

Below is an example with minimal usage:

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_conf = BrowserConfig(headless=True)  # or False to see the browser
    run_conf = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS
    )

    async with AsyncWebCrawler(config=browser_conf) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_conf
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())
```

> IMPORTANT: By default cache mode is set to `CacheMode.BYPASS` to have fresh content. Set `CacheMode.ENABLED` to enable caching.

We’ll explore more advanced config in later tutorials (like enabling proxies, PDF output, multi-tab sessions, etc.). For now, just note how you pass these objects to manage crawling.

---

## 4. Generating Markdown Output

#### (unclecode/crawl4ai/docs/md_v2/api/async-webcrawler.md)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/api/async-webcrawler.md] (Score: 0.66)
- - We define a **`BrowserConfig`** with Firefox, no headless, and `verbose=True`.
- We define a **`CrawlerRunConfig`** that **bypasses cache**, uses a **CSS** extraction schema, has a `word_count_threshold=15`, etc.
- We pass them to `AsyncWebCrawler(config=...)` and `arun(url=..., config=...)`.

---

## 7. Best Practices & Migration Notes

1. **Use** `BrowserConfig` for **global** settings about the browser’s environment.
2. **Use** `CrawlerRunConfig` for **per-crawl** logic (caching, content filtering, extraction strategies, wait conditions).
3. **Avoid** legacy parameters like `css_selector` or `word_count_threshold` directly in `arun()`. Instead:

   ```python
   run_cfg = CrawlerRunConfig(css_selector=".main-content", word_count_threshold=20)
   result = await crawler.arun(url="...", config=run_cfg)
   ```

4. **Context Manager** usage is simplest unless you want a persistent crawler across many calls.

---

## 8. Summary

**AsyncWebCrawler** is your entry point to asynchronous crawling:

- **Constructor** accepts **`BrowserConfig`** (or defaults).
- **`arun(url, config=CrawlerRunConfig)`** is the main method for single-page crawls.
- **`arun_many(urls, config=CrawlerRunConfig)`** handles concurrency across multiple URLs.
- For advanced lifecycle control, use `start()` and `close()` explicitly.

**Migration**:

- If you used `AsyncWebCrawler(browser_type="chromium", css_selector="...")`, move browser settings to `BrowserConfig(...)` and content/crawl logic to `CrawlerRunConfig(...)`.

This modular approach ensures your code is **clean**, **scalable**, and **easy to maintain**. For any advanced or rarely used parameters, see the [BrowserConfig docs](../api/parameters.md).

#### (unclecode/crawl4ai/docs/md_v2/core/simple-crawling.md)[https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/md_v2/core/simple-crawling.md] (Score: 0.66)
- # Access extracted media and links
print(result.media)        # Dictionary of found media (images, videos, audio)
print(result.links)        # Dictionary of internal and external links
```

## Adding Basic Options

Customize your crawl using `CrawlerRunConfig`:

```python
run_config = CrawlerRunConfig(
    word_count_threshold=10,        # Minimum words per content block
    exclude_external_links=True,    # Remove external links
    remove_overlay_elements=True,   # Remove popups/modals
    process_iframes=True           # Process iframe content
)

result = await crawler.arun(
    url="https://example.com",
    config=run_config
)
```

## Handling Errors

Always check if the crawl was successful:

```python
run_config = CrawlerRunConfig()
result = await crawler.arun(url="https://example.com", config=run_config)

if not result.success:
    print(f"Crawl failed: {result.error_message}")
    print(f"Status code: {result.status_code}")
```

## Logging and Debugging

Enable verbose logging in `BrowserConfig`:

```python
browser_config = BrowserConfig(verbose=True)

async with AsyncWebCrawler(config=browser_config) as crawler:
    run_config = CrawlerRunConfig()
    result = await crawler.arun(url="https://example.com", config=run_config)
```

## Complete Example

Here's a more comprehensive example demonstrating common usage patterns:


The configuration class passed **per crawl run** to `arun()` is:

## `CrawlerRunConfig`

This is distinct from **`BrowserConfig`**, which configures global browser behavior (headless mode, browser type, user agent, etc.).

- **`BrowserConfig`** → passed once to `AsyncWebCrawler(config=...)` — controls the browser environment.
- **`CrawlerRunConfig`** → passed to each `arun(url=..., config=...)` call — controls per-crawl logic such as caching (`cache_mode`), content filtering, extraction strategies (e.g., `JsonCssExtractionStrategy`), page navigation/timing, JS execution, session management, and screenshot/PDF options.

Example usage:
```python
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

browser_conf = BrowserConfig(headless=True)
run_conf = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

async with AsyncWebCrawler(config=browser_conf) as crawler:
    result = await crawler.arun(url="https://example.com", config=run_conf)
```