Smoke Test Token Extraction - Troubleshooting Guide¶

The original smoke tests (scripts/test-api-flows.sh) relied on docker logs command to extract email verification and password reset tokens from container logs. This approach failed when tests ran inside Docker containers (test and CI environments) because the Docker CLI was not available within containers, and containers cannot introspect their own logs.

Investigation revealed that pytest's built-in caplog fixture provides a superior solution for log capture during test execution. The caplog fixture captures application logs in-memory without requiring external tools, works in all environments (dev, test, CI/CD), and provides better error messages and debugging capabilities.

Implementation replaced shell script token extraction with pure Python log parsing using caplog. The solution achieved 96% test success rate (22/23 tests passing, 1 skipped due to unrelated API bug), eliminated external dependencies, and enabled tests to run consistently across all environments. The smoke test suite now validates the complete authentication flow (registration → verification → login → password reset → logout) with no environment-specific workarounds.

Initial Problem¶

Symptoms¶

Smoke tests failed to extract tokens from logs when running inside Docker containers, resulting in test failures during email verification and password reset flows.

Environment: Test containers (dashtam-test-app) and CI/CD pipeline (GitHub Actions)

# Error attempting to extract tokens
docker logs dashtam-test-app 2>&1 | grep "verify-email?token="
# Error: Cannot connect to Docker daemon (not available inside container)

Working Environments: Local development (tests running on host machine with Docker CLI available)

Expected Behavior¶

Tests should extract email verification and password reset tokens from application logs to continue authentication flow testing (verify email, reset password).

Actual Behavior¶

Tests failed when attempting to execute docker logs command from within test containers:

• Docker CLI not available inside containers
• Container cannot access its own logs via Docker API
• Shell script token extraction logic failed silently

Impact¶

• Severity: High (blocked critical smoke tests)
• Affected Components: Smoke test suite, CI/CD pipeline, test environment
• User Impact: Unable to run comprehensive authentication flow tests in containerized environments

Investigation Steps¶

Document each investigation attempt chronologically.

Step 1: Container Environment Analysis¶

Hypothesis: Docker CLI is missing or misconfigured inside test container

Investigation:

Checked if Docker CLI was available inside container and attempted to run Docker commands:

# Inside test container
docker compose -f compose/docker-compose.test.yml exec app bash
which docker  # Check if Docker CLI exists
docker ps     # Try to list containers

Findings:

• Docker CLI not installed in test container (by design - containers are isolated)
• Even if installed, containers cannot access host Docker daemon without socket mounting
• Socket mounting creates security risks and environment coupling

Result: ✅ Issue found - Docker logs approach fundamentally incompatible with containerized testing

Step 2: Alternative Log Capture Methods¶

Hypothesis: pytest or FastAPI provides built-in log capture mechanisms

Investigation:

Researched pytest logging capabilities and FastAPI testing patterns:

# Explored pytest caplog fixture
def test_with_caplog(caplog):
    # caplog captures logs during test execution
    with caplog.at_level(logging.INFO):
        # Run code that logs
        pass
    # Access captured logs
    for record in caplog.records:
        print(record.message)

Findings:

• pytest's caplog fixture captures logs in-memory during test execution
• No external tools required (pure Python)
• Works in all environments (containers, CI/CD, local)
• Provides structured access to log records (level, message, logger name)

Result: ✅ Solution identified - caplog fixture eliminates Docker dependency

Step 3: Token Extraction Pattern Design¶

Hypothesis: Simple regex can extract tokens from log messages

Investigation:

Analyzed EmailService log format and designed extraction logic:

# EmailService logs in development mode
logger.info(f"Verification URL: https://localhost:3000/verify-email?token={token}")

# Extraction function
def extract_token_from_caplog(caplog, pattern: str) -> str:
    for record in caplog.records:
        if pattern in record.message:
            regex_pattern = pattern.replace("?", "\\?")  # Escape special chars
            match = re.search(rf"{regex_pattern}([^&\s\"]+)", record.message)
            if match:
                return match.group(1)
    raise AssertionError(f"Token not found in logs")

Findings:

• Simple string matching first, then regex extraction
• Need to escape special URL characters (?, &)
• Clear error messages when token not found

Result: ✅ Extraction pattern proven effective

Root Cause Analysis¶

Primary Cause¶

Problem:

The original smoke tests used docker logs command to extract tokens from container stdout/stderr. This approach assumes the Docker CLI is available and the test process has access to the Docker daemon. When tests run inside containers (standard for test/CI environments), these assumptions break down.

# Problematic shell script approach
VERIFICATION_TOKEN=$(docker logs dashtam-test-app 2>&1 | \
    grep "verify-email?token=" | \
    grep -oP 'token=\K[^&\s"]+' | \
    tail -1)

Why This Happens:

1. Docker-in-Docker limitation: Containers cannot access host Docker daemon by default
2. CLI availability: Docker CLI not installed in application containers (security best practice)
3. Log access: Containers cannot introspect their own logs via Docker API
4. Environment coupling: Solution only works when Docker CLI available on host

Impact:

Tests worked locally (Docker CLI on host) but failed in containerized test/CI environments, creating environment-specific behavior and blocking automated testing pipelines.

Contributing Factors¶

Factor 1: Shell Script Complexity¶

Shell scripts for log parsing are brittle, have poor error messages, and are difficult to debug when extraction fails.

Factor 2: Race Conditions¶

Multiple tests running in parallel could race to extract tokens from shared log stream, causing intermittent failures.

Factor 3: Maintenance Burden¶

Shell script token extraction logic separate from Python test code, requiring context switching and duplication of regex patterns.

Solution Implementation¶

Approach¶

Replace Docker log extraction with pytest's built-in caplog fixture:

1. Use caplog.at_level() context manager to capture logs during test execution
2. Extract tokens from captured log records using Python regex
3. Store extracted tokens in shared test data dictionary for use across test functions
4. Eliminate all shell script dependencies for token extraction

Changes Made¶

Change 1: Token Extraction Function¶

Before:

# Shell script extraction in scripts/test-api-flows.sh
VERIFICATION_TOKEN=$(docker logs dashtam-test-app 2>&1 | \
    grep "verify-email?token=" | \
    grep -oP 'token=\K[^&\s"]+' | \
    tail -1)

After:

# Pure Python extraction in tests/smoke/test_complete_auth_flow.py
def extract_token_from_caplog(caplog, pattern: str) -> str:
    """Extract token from pytest's captured logs.

    Args:
        caplog: pytest's log capture fixture
        pattern: URL pattern to search for (e.g., "verify-email?token=")

    Returns:
        Extracted token string

    Raises:
        AssertionError: If token not found in logs
    """
    for record in caplog.records:
        if pattern in record.message:
            regex_pattern = pattern.replace("?", "\\?")
            match = re.search(rf"{regex_pattern}([^&\s\"]+)", record.message)
            if match:
                return match.group(1)
    raise AssertionError(f"Token not found in logs with pattern: {pattern}")

Rationale:

Pure Python implementation eliminates Docker CLI dependency, provides better error messages, and works consistently in all environments.

Change 2: Fixture with Caplog Integration¶

Before:

# Old fixture without log capture
@pytest.fixture(scope="module")
def smoke_test_user(client):
    # Registration but no token extraction
    response = client.post("/api/v1/auth/register", json={...})
    return {"email": email}  # Missing verification token

After:

# New fixture with caplog for token extraction
_smoke_test_user_data = {}  # Module-level shared state

@pytest.fixture(scope="function")
def smoke_test_user(client, unique_test_email, test_password, caplog):
    """Create and register smoke test user with token extraction.

    Uses module-level shared dictionary to maintain state across tests
    while using function-scoped caplog fixture.
    """
    if _smoke_test_user_data:
        return _smoke_test_user_data

    _smoke_test_user_data.update({
        "email": unique_test_email,
        "password": test_password,
        # ... other fields
    })

    # Capture logs during registration
    with caplog.at_level(logging.INFO):
        response = client.post("/api/v1/auth/register", json={...})

    # Extract token AFTER caplog block closes
    _smoke_test_user_data["verification_token"] = extract_token_from_caplog(
        caplog, "verify-email?token="
    )

    return _smoke_test_user_data

Rationale:

Combines module-level state sharing with function-scoped caplog to capture logs during each test while maintaining user data across test sequence.

Change 3: Log Capture in Test Functions¶

Before:

# Manual Docker logs extraction in shell script
docker logs dashtam-test-app 2>&1 | grep "password-reset" | ...

After:

# Automatic log capture during test execution
def test_09_request_password_reset(client, smoke_test_user, caplog):
    """Test password reset request sends email with reset token."""
    with caplog.at_level(logging.INFO):
        response = client.post(
            "/api/v1/auth/password-resets",
            json={"email": smoke_test_user["email"]},
        )

    assert response.status_code == 200

    # Extract reset token from captured logs
    reset_token = extract_token_from_caplog(caplog, "password-reset?token=")
    smoke_test_user["reset_token"] = reset_token

Rationale:

Log capture happens automatically during test execution, no external commands needed, clean separation between test execution and log parsing.

Implementation Steps¶

1. Step 1: Added caplog parameter to fixtures and test functions

# Import logging module
import logging

# Add caplog to function signatures
def smoke_test_user(client, unique_test_email, test_password, caplog):

1. Step 2: Wrapped API calls in caplog.at_level() context manager

with caplog.at_level(logging.INFO):
    response = client.post("/api/v1/auth/register", json={...})

1. Step 3: Implemented token extraction function

def extract_token_from_caplog(caplog, pattern: str) -> str:
    # Extraction logic

1. Step 4: Updated all test functions to use caplog extraction

# Removed all shell script token extraction
# Updated Python tests with caplog approach

1. Step 5: Removed shell script dependencies

# Smoke tests now pure pytest (no bash scripts needed)
make test-smoke  # Works in all environments

Verification¶

Test Results¶

Before Fix:

# Tests failed in containerized environments
docker compose -f compose/docker-compose.test.yml exec app pytest tests/smoke/
# Error: Cannot extract tokens (Docker CLI not available)
# Result: Tests blocked, unable to complete auth flow

After Fix:

# All tests pass in all environments
$ make test-smoke
======================== test session starts =========================
tests/smoke/test_complete_auth_flow.py::test_00_health_check PASSED
tests/smoke/test_complete_auth_flow.py::test_01_api_docs_accessible PASSED
tests/smoke/test_complete_auth_flow.py::test_02_user_registration PASSED
tests/smoke/test_complete_auth_flow.py::test_03_extract_verification_token PASSED
tests/smoke/test_complete_auth_flow.py::test_04_email_verification PASSED
...
tests/smoke/test_complete_auth_flow.py::test_18_access_token_still_works_after_logout PASSED

=================== 22 passed, 1 skipped in 2.45s ====================

Verification Steps¶

1. Test in original failing environment (test container)

docker compose -f compose/docker-compose.test.yml up -d
docker compose -f compose/docker-compose.test.yml exec app \
  uv run pytest tests/smoke/test_complete_auth_flow.py -v

Result: ✅ 22/23 passing (1 skipped due to unrelated API bug)

1. Test in CI/CD environment (GitHub Actions)

# .github/workflows/test.yml runs smoke tests
# No environment-specific workarounds needed

Result: ✅ Passing in all CI runs

1. Test in development environment (local)

make test-smoke

Result: ✅ Passing

Regression Testing¶

Full test suite regression check to ensure caplog integration didn't break other tests:

# Run all tests
make test

# Results
295 tests passing (smoke, unit, integration, API)
76% code coverage maintained
Zero regressions introduced

Lessons Learned¶

Technical Insights¶

1. pytest's caplog fixture is production-ready

The caplog fixture is mature, well-documented, and designed exactly for this use case. It eliminates the need for external log parsing tools and provides structured access to log records.

1. Fixture scope matters with caplog

Caplog is function-scoped by design (to avoid log pollution between tests). When tests need to share state, use module-level dictionaries with function-scoped fixtures rather than fighting pytest's scoping model.

1. Log capture timing is critical

Token extraction must happen AFTER the with caplog.at_level() block closes. Attempting extraction inside the block can miss logs that haven't been flushed yet.

Process Improvements¶

1. Avoid Docker-dependent testing patterns

Tests should use language-native tools (pytest fixtures, mocks, etc.) rather than relying on Docker CLI or shell scripts. This ensures tests work consistently across all environments.

1. Prefer Python over shell scripts for testing

Python provides better error messages, type safety, and debugging capabilities compared to shell scripts. Keep test logic in the same language as the application.

1. Test in target environment early

The Docker logs approach worked locally but failed in containers. Testing in the actual deployment environment (containers, CI/CD) earlier would have caught this issue sooner.

Best Practices¶

• Use pytest's built-in fixtures (caplog, monkeypatch, tmp_path) before reaching for external tools
• Design tests to be environment-agnostic (no assumptions about Docker CLI availability)
• Extract reusable test utilities into well-documented helper functions
• Keep test data extraction logic close to test code (same language, same file)
• Prefer in-memory test data over external file/log parsing when possible

Future Improvements¶

Short-Term Actions¶

1. Fix GET /password-resets/{token} endpoint

Timeline: Next sprint (low priority - endpoint is optional for UX only)

Owner: Backend team

Details: Update endpoint to iterate through unused tokens and compare with bcrypt, matching pattern used in email verification endpoint.

1. Add smoke test documentation to main README

Timeline: Current sprint

Owner: Documentation team

Details: Link to tests/smoke/README.md from main project README for visibility.

Long-Term Improvements¶

1. Expand smoke test coverage

Add additional critical path tests: - Provider connection flow (OAuth) - Account data retrieval - Transaction sync

1. Implement smoke test monitoring

Run smoke tests on production-like staging environment before each deployment: - Pre-deployment gate in CI/CD pipeline - Alert on failures before production rollout

Monitoring & Prevention¶

Add pre-commit check to prevent Docker CLI dependencies in tests:

# .git/hooks/pre-commit
# Reject commits with Docker CLI in test code
if grep -r "docker logs" tests/; then
    echo "Error: Tests must not depend on Docker CLI"
    exit 1
fi

References¶

Related Documentation:

• Smoke Test README (tests/smoke/README.md in project root) - Complete smoke test documentation
• Testing Guide - Comprehensive testing strategy
• Testing Best Practices - Testing patterns

External Resources:

• pytest caplog documentation - Official caplog fixture docs
• FastAPI Testing Guide - FastAPI TestClient patterns

Related Issues:

• GitHub PR #XX - Smoke test caplog implementation (if exists)
• Original shell script: scripts/test-api-flows.sh (deprecated)

Document Information¶

Template: troubleshooting-template.md Created: 2025-10-06 Last Updated: 2025-10-18