# Ahrefs Login & Scraping – funktionierender Ansatz

## Problem
- `agent-browser` (CLI) wird von Cloudflare Turnstile auf app.ahrefs.com **nach dem Login** blockiert
- Login-Seite selbst geht manchmal, aber Dashboard/interne Seiten → Cloudflare Challenge
- Headless Browser ohne Anti-Detection werden erkannt

## Lösung: Playwright + playwright-stealth

### Installation (einmalig)
```bash
pip install --break-system-packages playwright playwright-stealth
playwright install chromium
# Xvfb für headless mit Display:
apt-get install -y xvfb
```

### Login-Flow (state speichern)
```python
import asyncio, json
from playwright.async_api import async_playwright
from playwright_stealth import Stealth

async def main():
    stealth = Stealth()
    async with stealth.use_async(async_playwright()) as p:
        browser = await p.chromium.launch(headless=True, args=['--no-sandbox'])
        ctx = await browser.new_context(
            user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36',
            viewport={'width': 1280, 'height': 900}
        )
        page = await ctx.new_page()
        
        await page.goto('https://app.ahrefs.com/user/login', wait_until='networkidle', timeout=30000)
        await page.fill('input[name="email"]', 'mkiesewetter9@googlemail.com')
        await page.fill('input[name="password"]', 'mkiesewetter9@googlemail.com')
        await page.click('button[type="submit"]')
        await page.wait_for_url('**/dashboard**', timeout=15000)
        
        # State speichern für spätere Sessions
        await ctx.storage_state(path='/tmp/ahrefs-state.json')
        print('Logged in & state saved')
        await browser.close()

asyncio.run(main())
```

### Daten abrufen (mit gespeichertem State)
```python
import asyncio, json
from playwright.async_api import async_playwright
from playwright_stealth import Stealth

async def get_table_data(page):
    """Scrollt und extrahiert Tabellen-Zeilen"""
    await page.evaluate('window.scrollTo(0, 800)')
    await page.wait_for_timeout(3000)
    rows = await page.evaluate('''() => {
        const rows = document.querySelectorAll('[class*="tableRow"], [class*="TableRow"], [role="row"], tr');
        const data = [];
        rows.forEach(r => {
            const t = r.innerText.trim();
            if (t && t.length > 5 && t.length < 500 && t.indexOf('Filter') === -1 && t.indexOf('Spalten') === -1) {
                data.push(t);
            }
        });
        return data;
    }''')
    return rows

async def main():
    stealth = Stealth()
    async with stealth.use_async(async_playwright()) as p:
        browser = await p.chromium.launch(headless=True, args=['--no-sandbox'])
        ctx = await browser.new_context(
            user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36',
            viewport={'width': 1280, 'height': 900}
        )
        # Gespeicherten State laden
        cookies = json.load(open('/tmp/ahrefs-state.json'))['cookies']
        await ctx.add_cookies(cookies)
        page = await ctx.new_page()
        
        base = 'mode=subdomains&projectId=2032747&target=manonamission.de%2F&volume_type=monthly'
        
        # Beispiel: Top Pages
        await page.goto(f'https://app.ahrefs.com/site-explorer/top-pages?{base}', wait_until='networkidle', timeout=30000)
        await page.wait_for_timeout(5000)
        rows = await get_table_data(page)
        for r in rows:
            print(r)
        
        await browser.close()

asyncio.run(main())
```

### Wichtige URLs (mit Projekt-Parametern)
- **Dashboard:** `https://app.ahrefs.com/dashboard`
- **Site Explorer Overview:** `/site-explorer/overview?{base}`
- **Organic Keywords:** `/site-explorer/organic-keywords?{base}`
- **Top Pages:** `/site-explorer/top-pages?{base}`
- **Backlinks:** `/site-explorer/backlinks?{base}`
- **Organic Competitors:** `/site-explorer/organic-competitors?{base}`
- **Opportunities:** `/site-explorer/opportunities?{base}`
- **Content Gap:** `/site-explorer/content-gap?{base}`

### Wichtige Parameter
- `projectId=2032747` – manonamission.de Projekt-ID
- `target=manonamission.de%2F` – Ziel-Domain
- `mode=subdomains` – inkl. Subdomains
- `volume_type=monthly` – monatliches Suchvolumen

### Xvfb (falls nötig)
```bash
Xvfb :99 -screen 0 1280x1024x24 &
export DISPLAY=:99
```
Normalerweise nicht nötig – Playwright headless läuft ohne Display.

### Was NICHT funktioniert
- ❌ `agent-browser` CLI → Cloudflare Block nach Login
- ❌ `web_fetch` → kein JavaScript, useless für SPA
- ❌ Direkte URL-Konstruktion ohne `projectId` → "Bitte überprüfe deine Domain"
- ❌ `Stealth().use_async(browser.new_context(...))` → falsche Syntax
- ✅ `Stealth().use_async(async_playwright())` → korrekt!

### State-Haltbarkeit
- `/tmp/ahrefs-state.json` überlebt Reboots NICHT (tmpfs)
- Nach Server-Reboot muss neu eingeloggt werden
- Falls Session abgelaufen → Login-Flow wiederholen