screamingFrog/AGENTS.md

Repository Guidelines

Project Structure & Module Organization

The codebase is intentionally lean. index.php bootstraps the crawl by instantiating webanalyse and handing off the crawl identifier. Core crawling logic lives in webanalyse.php, which houses HTTP fetching, link extraction, and database persistence. Use setnew.php to reset seed data inside the screaming_frog schema before a rerun. Keep new helpers in their own PHP files under this root so the autoload includes stay predictable; group SQL migrations or fixtures under a database/ folder if you add them. IDE settings reside in .idea/.

Build, Test, and Development Commands

Docker Development

The project runs in Docker containers. Use these commands:

# Start containers
docker-compose up -d

# Stop containers
docker-compose down

# Rebuild containers
docker-compose up -d --build

# View logs
docker-compose logs -f php

Running Tests

The project uses PHPUnit for automated testing:

# Run all tests (Unit + Integration)
docker-compose exec php sh -c "php /var/www/html/vendor/bin/phpunit /var/www/tests/"

# Or use the composer shortcut
docker-compose exec php composer test

Test Structure:

• tests/Unit/ - Unit tests for individual components
• tests/Integration/ - Integration tests for full crawl workflows
• All tests run in isolated database transactions

Static Code Analysis

PHPStan is configured at Level 8 (strictest) to ensure type safety:

# Run PHPStan analysis
docker-compose exec php sh -c "php -d memory_limit=512M /var/www/html/vendor/bin/phpstan analyse -c /var/www/phpstan.neon"

# Or use the composer shortcut
docker-compose exec php composer phpstan

PHPStan Configuration:

• Level: 8 (maximum strictness)
• Analyzes: src/ and tests/
• Excludes: vendor/
• Config file: phpstan.neon

All code must pass PHPStan Level 8 with zero errors before merging.

Code Style Checking

PHP_CodeSniffer enforces PSR-12 coding standards:

# Check code style
docker-compose exec php composer phpcs

# Automatically fix code style issues
docker-compose exec php composer phpcbf

PHPCS Configuration:

• Standard: PSR-12
• Analyzes: src/ and tests/
• Excludes: vendor/
• Auto-fix available via phpcbf

Run phpcbf before committing to automatically fix most style violations.

Coding Style & Naming Conventions

Follow PSR-12 style cues already in use: 4-space indentation, brace-on-new-line for functions, and declare(strict_types=1); at the top of entry scripts. Favour descriptive camelCase for methods (getMultipleWebsites) and snake_case only for direct SQL field names. Maintain mysqli usage for consistency, and gate new configuration through constants or clearly named environment variables.

Testing Guidelines

Automated Testing

The project has a comprehensive test suite using PHPUnit:

• Write tests first: Follow TDD principles when adding new features
• Unit tests (tests/Unit/): Test individual classes and methods in isolation
• Integration tests (tests/Integration/): Test full crawl workflows with real HTTP requests
• Database isolation: Tests use transactions that roll back automatically
• Coverage: Aim for high test coverage on critical crawl logic

Quality Gates

Before committing code, ensure:

1. All tests pass: docker-compose exec php composer test
2. PHPStan analysis passes: docker-compose exec php composer phpstan
3. Code style is correct: docker-compose exec php composer phpcs
4. Auto-fix style issues: docker-compose exec php composer phpcbf

Pre-commit Checklist:

• ✅ Tests pass
• ✅ PHPStan Level 8 with 0 errors
• ✅ PHPCS PSR-12 compliance (warnings acceptable)

Manual Testing

For UI changes, manually test the crawler interface at http://localhost:8080. Verify:

• Job creation and status updates
• Page and link extraction accuracy
• Error handling for invalid URLs or network issues

Commit & Pull Request Guidelines

Author commit messages in the present tense with a concise summary (Add link grouping for external URLs). Group related SQL adjustments with their PHP changes in the same commit. For pull requests, include: a short context paragraph, reproduction steps, screenshots of key output tables when behaviour changes, and any follow-up tasks. Link tracking tickets or issues so downstream agents can trace decisions.

Security & Configuration Notes

Database credentials are currently hard-coded for local XAMPP usage. If you introduce environment-based configuration, document expected .env keys and ensure credentials are excluded from version control. Never commit production connection details or raw crawl exports.