132 lines
4.9 KiB
Markdown
132 lines
4.9 KiB
Markdown
# Ahrefs Login & Scraping – funktionierender Ansatz
|
||
|
||
## Problem
|
||
- `agent-browser` (CLI) wird von Cloudflare Turnstile auf app.ahrefs.com **nach dem Login** blockiert
|
||
- Login-Seite selbst geht manchmal, aber Dashboard/interne Seiten → Cloudflare Challenge
|
||
- Headless Browser ohne Anti-Detection werden erkannt
|
||
|
||
## Lösung: Playwright + playwright-stealth
|
||
|
||
### Installation (einmalig)
|
||
```bash
|
||
pip install --break-system-packages playwright playwright-stealth
|
||
playwright install chromium
|
||
# Xvfb für headless mit Display:
|
||
apt-get install -y xvfb
|
||
```
|
||
|
||
### Login-Flow (state speichern)
|
||
```python
|
||
import asyncio, json
|
||
from playwright.async_api import async_playwright
|
||
from playwright_stealth import Stealth
|
||
|
||
async def main():
|
||
stealth = Stealth()
|
||
async with stealth.use_async(async_playwright()) as p:
|
||
browser = await p.chromium.launch(headless=True, args=['--no-sandbox'])
|
||
ctx = await browser.new_context(
|
||
user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36',
|
||
viewport={'width': 1280, 'height': 900}
|
||
)
|
||
page = await ctx.new_page()
|
||
|
||
await page.goto('https://app.ahrefs.com/user/login', wait_until='networkidle', timeout=30000)
|
||
await page.fill('input[name="email"]', 'mkiesewetter9@googlemail.com')
|
||
await page.fill('input[name="password"]', 'mkiesewetter9@googlemail.com')
|
||
await page.click('button[type="submit"]')
|
||
await page.wait_for_url('**/dashboard**', timeout=15000)
|
||
|
||
# State speichern für spätere Sessions
|
||
await ctx.storage_state(path='/tmp/ahrefs-state.json')
|
||
print('Logged in & state saved')
|
||
await browser.close()
|
||
|
||
asyncio.run(main())
|
||
```
|
||
|
||
### Daten abrufen (mit gespeichertem State)
|
||
```python
|
||
import asyncio, json
|
||
from playwright.async_api import async_playwright
|
||
from playwright_stealth import Stealth
|
||
|
||
async def get_table_data(page):
|
||
"""Scrollt und extrahiert Tabellen-Zeilen"""
|
||
await page.evaluate('window.scrollTo(0, 800)')
|
||
await page.wait_for_timeout(3000)
|
||
rows = await page.evaluate('''() => {
|
||
const rows = document.querySelectorAll('[class*="tableRow"], [class*="TableRow"], [role="row"], tr');
|
||
const data = [];
|
||
rows.forEach(r => {
|
||
const t = r.innerText.trim();
|
||
if (t && t.length > 5 && t.length < 500 && t.indexOf('Filter') === -1 && t.indexOf('Spalten') === -1) {
|
||
data.push(t);
|
||
}
|
||
});
|
||
return data;
|
||
}''')
|
||
return rows
|
||
|
||
async def main():
|
||
stealth = Stealth()
|
||
async with stealth.use_async(async_playwright()) as p:
|
||
browser = await p.chromium.launch(headless=True, args=['--no-sandbox'])
|
||
ctx = await browser.new_context(
|
||
user_agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36',
|
||
viewport={'width': 1280, 'height': 900}
|
||
)
|
||
# Gespeicherten State laden
|
||
cookies = json.load(open('/tmp/ahrefs-state.json'))['cookies']
|
||
await ctx.add_cookies(cookies)
|
||
page = await ctx.new_page()
|
||
|
||
base = 'mode=subdomains&projectId=2032747&target=manonamission.de%2F&volume_type=monthly'
|
||
|
||
# Beispiel: Top Pages
|
||
await page.goto(f'https://app.ahrefs.com/site-explorer/top-pages?{base}', wait_until='networkidle', timeout=30000)
|
||
await page.wait_for_timeout(5000)
|
||
rows = await get_table_data(page)
|
||
for r in rows:
|
||
print(r)
|
||
|
||
await browser.close()
|
||
|
||
asyncio.run(main())
|
||
```
|
||
|
||
### Wichtige URLs (mit Projekt-Parametern)
|
||
- **Dashboard:** `https://app.ahrefs.com/dashboard`
|
||
- **Site Explorer Overview:** `/site-explorer/overview?{base}`
|
||
- **Organic Keywords:** `/site-explorer/organic-keywords?{base}`
|
||
- **Top Pages:** `/site-explorer/top-pages?{base}`
|
||
- **Backlinks:** `/site-explorer/backlinks?{base}`
|
||
- **Organic Competitors:** `/site-explorer/organic-competitors?{base}`
|
||
- **Opportunities:** `/site-explorer/opportunities?{base}`
|
||
- **Content Gap:** `/site-explorer/content-gap?{base}`
|
||
|
||
### Wichtige Parameter
|
||
- `projectId=2032747` – manonamission.de Projekt-ID
|
||
- `target=manonamission.de%2F` – Ziel-Domain
|
||
- `mode=subdomains` – inkl. Subdomains
|
||
- `volume_type=monthly` – monatliches Suchvolumen
|
||
|
||
### Xvfb (falls nötig)
|
||
```bash
|
||
Xvfb :99 -screen 0 1280x1024x24 &
|
||
export DISPLAY=:99
|
||
```
|
||
Normalerweise nicht nötig – Playwright headless läuft ohne Display.
|
||
|
||
### Was NICHT funktioniert
|
||
- ❌ `agent-browser` CLI → Cloudflare Block nach Login
|
||
- ❌ `web_fetch` → kein JavaScript, useless für SPA
|
||
- ❌ Direkte URL-Konstruktion ohne `projectId` → "Bitte überprüfe deine Domain"
|
||
- ❌ `Stealth().use_async(browser.new_context(...))` → falsche Syntax
|
||
- ✅ `Stealth().use_async(async_playwright())` → korrekt!
|
||
|
||
### State-Haltbarkeit
|
||
- `/tmp/ahrefs-state.json` überlebt Reboots NICHT (tmpfs)
|
||
- Nach Server-Reboot muss neu eingeloggt werden
|
||
- Falls Session abgelaufen → Login-Flow wiederholen
|