Troubleshooting Guide

Common issues and solutions when working with fastapi-topaz.

Connection Issues

"Connection refused" to Topaz

Symptoms:

ConnectionRefusedError: [Errno 111] Connection refused

Solutions:

1. Check Topaz is running:

   curl http://localhost:8383/api/v2/info
   # or for gRPC:
   grpcurl -plaintext localhost:8282 list
   

2. Verify URL in config:

   # Correct:
   AuthorizerOptions(url="localhost:8282")
   
   # Wrong (don't include protocol):
   AuthorizerOptions(url="http://localhost:8282")
   

3. Check network/firewall:

   # If using Docker:
   docker-compose ps
   # Ensure containers are on same network
   

TLS handshake failures against a plaintext Topaz

Symptoms: the client hangs or fails with SSL errors when Topaz runs without TLS (e.g. topaz start --no-tls or a plaintext docker-compose setup).

Solution: the client uses a TLS gRPC channel by default. For a local, non-TLS Topaz, enable the insecure channel:

config = TopazConfig(
    ...
    insecure=True,  # plaintext gRPC channel - local development only
)

Never enable insecure=True against a production Topaz. If Topaz runs with TLS and a self-signed certificate, keep the default and point AuthorizerOptions(cert_file_path=...) at the CA certificate instead.

"Deadline exceeded" / Timeout

Symptoms:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED

Solutions:

1. Increase timeout:

   AuthorizerOptions(url="localhost:8282", timeout=10)
   

2. Enable circuit breaker:

   config = TopazConfig(
       ...
       check_timeout=5.0,  # gRPC deadline in seconds per authorization call
       circuit_breaker=CircuitBreaker(
           fallback="cache_then_deny",
       ),
   )
   

3. Check Topaz health:

   # Topaz may be overloaded or policy evaluation is slow
   docker logs topaz-container
   

---

Authorization Issues

Always Getting 403 Forbidden

Symptoms: Every request returns 403, even for admins.

Debug steps:

1. Enable debug logging:

   import logging
   logging.getLogger("fastapi_topaz").setLevel(logging.DEBUG)
   

2. Check identity extraction:

   def identity_provider(request: Request) -> Identity:
       user_id = request.headers.get("X-User-ID")
       print(f"DEBUG: Extracted user_id={user_id}")  # Add debug
   
       if not user_id:
           return Identity(type=IdentityType.IDENTITY_TYPE_NONE)
   
       return Identity(type=IdentityType.IDENTITY_TYPE_SUB, value=user_id)
   

3. Verify policy path:

   # Print what policy path is being checked
   policy_path = config.policy_path_for("GET", "/documents/{id}")
   print(f"Policy path: {policy_path}")  # e.g., myapp.GET.documents.__id
   

4. Test policy directly:

   # Use topaz CLI to test
   topaz decision --decision allowed \
       --policy-path myapp.GET.documents \
       --identity-type sub \
       --identity alice
   

Seeing Which Policy Denied a Request

For development and debugging, expose_deny_reason=True makes 403 responses include the evaluated policy path and check source instead of a bare "Forbidden":

config = TopazConfig(
    ...,
    expose_deny_reason=True,  # dev/debug only
)

Denied responses then carry a structured body — {"detail": "Forbidden", "policy": "myapp.GET.documents", "source": "middleware"} from the middleware or dependency paths, with object_type and relation added for ReBAC denials. A custom middleware on_denied callback still takes precedence.

Warning: this leaks your policy structure to API clients. Never enable it in production.

Policy Not Found

Symptoms:

policy "myapp.GET.documents" not found

Solutions:

1. Check policy_instance_name matches:

   config = TopazConfig(
       ...
       policy_instance_name="myapp",  # Must match Topaz config
   )
   

2. Verify policy is loaded:

   # List loaded policies
   topaz policy list
   

3. Check Rego package name:

   # policy.rego
   package myapp  # Must match policy_path_root
   
   GET.documents.allowed := true
   

Hyphens in Policy Path (Invalid Rego Identifier)

Symptoms: Topaz returns errors about invalid policy paths, or policies don't match, when your API routes contain hyphens (e.g., /aircraft-programs).

Cause: Rego identifiers cannot contain hyphens. The - character is parsed as the minus operator. Route /aircraft-programs generates policy path myapp.GET.aircraft-programs, which is invalid Rego.

Solution:

Use policy_path_normalizer to replace hyphens with underscores:

from fastapi_topaz import TopazConfig, normalize_hyphens

config = TopazConfig(
    ...
    policy_path_normalizer=normalize_hyphens,
)

# Verify:
config.policy_path_for("GET", "/aircraft-programs")
# → "myapp.GET.aircraft_programs"

ReBAC Always Denied

Symptoms: ReBAC checks always return false, even for owners.

Debug steps:

1. Check resource context:

   # Add logging to see what's being sent
   def resource_context_provider(request: Request) -> dict:
       ctx = {
           "owner_id": request.state.document.owner_id,
           "is_public": request.state.document.is_public,
       }
       print(f"DEBUG: resource_context={ctx}")
       return ctx
   

2. Verify object_id resolution:

   # Explicit object_id
   require_rebac_allowed(config, "document", "can_read", object_id="123")
   
   # Or from path params (default)
   # Uses request.path_params["id"]
   

3. Check directory relations:

   # Query Topaz directory
   topaz directory get-relation \
       --subject-type user \
       --subject-id alice \
       --relation owner \
       --object-type document \
       --object-id 123
   

---

Caching Issues

Changes Not Taking Effect

Symptoms: Updated permissions in Topaz aren't reflected immediately.

Solutions:

1. Clear cache:

   await config.decision_cache.clear()
   

2. Reduce TTL for development:

   # Short TTL for development
   DecisionCache(ttl_seconds=5)
   
   # Longer TTL for production
   DecisionCache(ttl_seconds=60)
   

3. Bypass the cache for specific checks: use a second TopazConfig without decision_cache for those calls:

   uncached = TopazConfig(..., decision_cache=None)
   allowed = await uncached.is_allowed(request, "myapp.GET.documents")
   

Cache Not Working

Symptoms: Cache hit rate is 0%, every request hits Topaz.

Debug steps:

1. Verify cache is configured:

   config = TopazConfig(
       ...
       decision_cache=DecisionCache(),  # Must be set!
   )
   

2. Check cache keys are consistent:

   # Cache key includes resource_context
   # If context changes, cache won't hit
   def resource_context_provider(request: Request) -> dict:
       return {
           "id": request.path_params.get("id"),
           # Don't include timestamps or random values!
       }
   

3. Enable metrics:

   config = TopazConfig(
       ...
       metrics=PrometheusMetrics(),
   )
   # Check topaz_cache_hits_total vs topaz_cache_misses_total
   

---

Circuit Breaker Issues

Circuit Opens Too Quickly

Symptoms: Circuit breaker opens after minor network blips.

Solutions:

circuit_breaker=CircuitBreaker(
    failure_threshold=10,     # Increase from default 5
    recovery_timeout=60,      # Longer recovery time
    success_threshold=3,      # More successes needed to close
)

Fallback Not Working

Symptoms: When circuit is open, requests fail instead of using fallback.

Debug steps:

1. Check fallback strategy:

   circuit_breaker=CircuitBreaker(
       fallback="cache_then_deny",  # Uses stale cache first
   )
   

2. Verify stale cache is enabled:

   circuit_breaker=CircuitBreaker(
       serve_stale_cache=True,
       stale_cache_ttl=300,  # 5 minutes
   )
   

3. Check circuit status:

   @app.get("/health")
   async def health():
       if config.circuit_breaker:
           status = config.circuit_breaker.status()
           return {
               "circuit_state": status.state,
               "failure_count": status.failure_count,
           }
   

---

Middleware Issues

Middleware Not Applied

Symptoms: Routes aren't being protected by TopazMiddleware.

Solutions:

1. Check middleware order:

   # Middleware is added in reverse order
   # TopazMiddleware should be added LAST
   app.add_middleware(CORSMiddleware, ...)
   app.add_middleware(TopazMiddleware, config=config)  # Added last, runs first
   

2. Check exclusion patterns:

   TopazMiddleware(
       app,
       config=config,
       exclude_paths=[r"^/health$", r"^/docs.*"],  # Regex patterns
   )
   

3. Check route is matched:

   # Middleware only protects routes that exist
   # 404s pass through without authorization
   

Routes Excluded Unexpectedly

Symptoms: Some routes aren't being protected.

Debug steps:

1. Check @skip_middleware decorator:

   # This route is excluded:
   @app.get("/special")
   @skip_middleware
   async def special():
       ...
   

2. Check router dependencies:

   # This entire router is excluded:
   router = APIRouter(dependencies=[Depends(SkipMiddleware)])
   

---

Integration Environment

Start with make int-doctor. It prints what the Makefile detected and fails with a fix-it message for each missing prerequisite.

Every Authenticated Call Returns 401

Cause: The webapp container started without OIDC credentials. Compose reads env_file (integration-tests/.env.oidc) only when it creates a container. docker-compose restart keeps the old environment.

Solution: make int-tf-apply recreates the webapp for you. If you ran Terraform or compose by hand, recreate it:

cd integration-tests && docker-compose up -d --force-recreate webapp

Login Redirect Fails / Cannot Resolve authentik-server

Cause: The OIDC issuer is http://authentik-server:9000. The e2e client and your browser follow redirects to that name from the host.

Solution:

echo "127.0.0.1 authentik-server" | sudo tee -a /etc/hosts

Podman: "Cannot connect to the Docker daemon"

Cause: docker-compose is looking for /var/run/docker.sock.

Solution: The Makefile sets DOCKER_HOST to the Podman machine socket when DOCKER_HOST is unset. Check that the machine is running (podman machine start). For compose commands run outside make, export it yourself:

export DOCKER_HOST="unix://$(podman machine inspect --format '{{.ConnectionInfo.PodmanSocket.Path}}')"

To pin a different socket, compose command, or Terraform binary, copy local.mk.example to local.mk.

---

Common Error Messages

Error	Cause	Solution
Identity has no value	identity_provider returned None/empty	Check header extraction, authentication
policy_path must not be empty	Empty policy path	Check policy_path_root configuration
Semaphore released too many times	Bug in custom code	Check async context managers
Unexpected 403, or 503 Authorization service unavailable	Authorizer call failed with no circuit-breaker fallback (e.g. INVALID_ARGUMENT for a missing policy); the status follows on_error	Check the ERROR log "Authorization check failed in ... for policy ..."; add the policy or configure a CircuitBreaker

---

Getting Help

If you're still stuck:

1. Enable full debug logging:

   import logging
   logging.basicConfig(level=logging.DEBUG)
   

2. Check Topaz logs:

   docker logs topaz-container -f
   

3. Simplify to minimal example:

   # Create minimal reproduction case
   from fastapi import FastAPI, Depends
   from fastapi_topaz import TopazConfig, require_policy_allowed
   # ... minimal config ...
   

4. Report issues: GitHub Issues

See Also

• Testing - Mock authorization in tests
• Circuit Breaker - Resilience configuration
• Observability - Monitoring and debugging