API Data Integration Planner

Prompt Template

You are a data integration architect. Plan an API data integration.

API(s) to integrate: {{apis}}
Data needed: {{data_needed}}
Destination system: {{destination}}
Update frequency: {{frequency: real-time/hourly/daily}}
Volume estimate: {{volume}}

## API Assessment
| API | Auth Method | Rate Limits | Data Format | Pagination | Docs Quality |

## Integration Architecture
### Data Flow
Source API → [extraction] → [transformation] → [loading] → Destination

### Authentication Strategy
- Auth type: [API key/OAuth2/JWT/Basic]
- Token management: [refresh strategy]
- Secret storage: [approach]

### Extraction Design
- Endpoint mapping:
  | Data Entity | Endpoint | Method | Parameters |
- Pagination handling:
- Incremental extraction: [strategy for getting only new/changed data]

### Rate Limit Management
- Limits: [X requests per Y seconds]
- Throttling strategy:
- Backoff algorithm:
- Queue management:

### Error Handling
| Error Code | Meaning | Response | Retry? |
| 429 | Rate limited | Backoff and retry | Yes |
| 401 | Auth expired | Refresh token | Yes |
| 500 | Server error | Log and retry | 3x |
| 404 | Not found | Log and skip | No |

### Data Transformation
| Source Field | Target Field | Transformation | Validation |

## Monitoring
| Metric | Alert Threshold | Response |
| API response time | | |
| Error rate | | |
| Data freshness | | |
| Volume anomalies | | |

## Failover Plan
What happens when the API is down?

Tips

• Always implement exponential backoff for rate limits — hammering a rate-limited API will get you banned
• Store API credentials in environment variables or a secrets manager, never in code
• Design for API changes — version your integration and monitor for deprecation notices
• Test with production-like volumes, not just a few records