api-data-sources

Research and document all authoritative APIs and data sources for: $ARGUMENTS

$ARGUMENTS should include:

A description of the app, project, or feature to research
The target domain / industry (e.g. real estate, fitness, publishing)
Optionally: budget constraints or pricing tier preferences (free-only, paid OK, enterprise)
Optionally: data freshness requirements (real-time, daily, weekly, static)
Optionally: a project slug for the output directory name
Optionally: a custom output directory (default: _docs/api-sources/)
Empty — ask the user to describe their project and use case

Purpose

This skill produces a complete API and data-source research package — a set of markdown files that fully document every authoritative data source available for a given project, how to access each one, what data it returns, what it costs, and exactly how to integrate it all into a React + Supabase + Vercel + AI Agents application.

What it produces:

A project directory under _docs/api-sources/[project-slug]/
_OVERVIEW.md — executive summary of all discovered sources
apis/[api-name].md — one detailed file per API
data-sources/[source-name].md — one file per non-API data source
_DATA_CATALOG.md — human-readable inventory of all available data types
_INTEGRATION_BLUEPRINT.md — step-by-step guide for React + Supabase + Vercel + AI integration

Phase 1 — Elicitation & Scoping

Before researching anything, understand the project deeply enough to filter results.

If the user's description is brief or ambiguous, ask up to 4 targeted questions. Skip any that are already answered by the description.

What does the app do? — What is the core user-facing functionality? What problem does it solve?
What data does the app need? — What types of information must the app display, process, or act on? (e.g. property listings, workout plans, book metadata, weather data)
What's the budget posture? — Free-tier only? Willing to pay per-request? Enterprise contracts OK? This filters which APIs to recommend.
What's the data freshness requirement? — Does data need to be real-time (websockets, streaming), near-real-time (polling every few minutes), daily batch, or static/occasional?

After scoping, write a Scoping Summary (internal, not output) that captures:

The core domain(s) to search
The data types needed (entities, fields, relationships)
Budget constraints
Freshness requirements
Any explicit user preferences for specific providers

Phase 2 — API & Data Source Discovery

2.1 — Search Strategy

For each data type identified in the scoping summary, search systematically:

Search for official/authoritative APIs — Use web search with queries like:
- "[domain] API" site:rapidapi.com OR site:programmableweb.com
- "[data type] API" developer documentation
- "[domain] open data" API
- "[domain] REST API" OR "[domain] GraphQL API"
Check major API aggregators — RapidAPI, ProgrammableWeb, API List, Public APIs (github.com/public-apis/public-apis)
Check government/institutional open data — data.gov, EU Open Data Portal, World Bank, census.gov, etc.
Check domain-specific authoritative sources — Industry databases, professional associations, research institutions
Identify non-API data sources — RSS/Atom feeds, CSV/JSON open datasets, web scraping targets (with legality notes), file downloads, existing databases

2.2 — Source Classification

Categorize every discovered source into one of these types:

Type	Description
REST API	Standard HTTP request/response API
GraphQL API	GraphQL endpoint
WebSocket / Streaming	Real-time data push
Webhook	Event-driven push notifications
RSS / Atom Feed	Syndication feed
Open Dataset	Downloadable CSV, JSON, XML, or database dump
Web Scraping Target	Structured data on a public page (note ToS/legality)
Supabase / DB Source	Existing data the user already has in Supabase or another DB
File-Based	PDFs, spreadsheets, documents containing structured data
AI-Generated	Data that would be synthesized or enriched by an AI agent

2.3 — Evaluation Criteria

For each source, assess:

Authority — Is this the canonical/official source for this data? (prefer first-party APIs)
Reliability — How stable is the API? Is there an SLA? When was it last updated?
Coverage — Does it provide all the fields/entities needed, or only a subset?
Cost — Free tier limits? Per-request pricing? Monthly caps?
Developer Experience — Quality of docs, SDKs, community support
Terms of Service — Any restrictions on storage, display, commercial use, or redistribution?

Rank sources as: Primary (recommended), Alternative (viable backup), or Supplementary (fills a gap not covered by primary).

Phase 3 — Per-Source Documentation

3.1 — API Documentation Template

For each API, create a file at apis/[api-name].md with this structure:

# [API Name]

> [One-line description of what this API provides]

| Field | Value |
|-------|-------|
| **Provider** | [Company/org name] |
| **Type** | REST / GraphQL / WebSocket / etc. |
| **Base URL** | `https://api.example.com/v2` |
| **Auth Method** | API Key / OAuth 2.0 / Bearer Token / None |
| **Official Docs** | [link to official documentation] |
| **Status Page** | [link if available] |
| **Role** | Primary / Alternative / Supplementary |

## Pricing & Quotas

| Tier | Price | Rate Limit | Monthly Quota | Notes |
|------|-------|------------|---------------|-------|
| Free | $0 | X req/sec | Y req/month | [any restrictions] |
| Pro | $X/mo | ... | ... | ... |
| Enterprise | Contact | ... | ... | ... |

> **Cost estimate for this project:** Based on [estimated usage pattern], expect ~[X] requests/month → [tier] tier → $[Y]/month.

## Getting Started

### Step 1 — Create an Account
[Exact steps: go to [URL], click sign up, verify email]

### Step 2 — Get API Credentials
[Exact steps: navigate to developer dashboard, create app, copy API key]

### Step 3 — Make Your First Request

```bash
curl -X GET "https://api.example.com/v2/endpoint" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

// TypeScript / fetch
const response = await fetch('https://api.example.com/v2/endpoint', {
  headers: {
    'Authorization': `Bearer ${process.env.API_KEY}`,
    'Content-Type': 'application/json',
  },
});
const data = await response.json();

Step 4 — Verify the Response

[Expected response shape — actual JSON example if available]

Key Endpoints

[Endpoint Name]

Method: GET / POST / etc.
Path: /v2/resource
Description: [what it returns]
Parameters: [key params with types and descriptions]
Response Shape:

{
  "field": "type — description"
}

[Repeat for each relevant endpoint]

Data Fields Available

Field	Type	Description	Example
`id`	string	Unique identifier	`"abc123"`
...	...	...	...

Rate Limits & Error Handling

Rate limit: [X requests per second/minute]
Rate limit header: X-RateLimit-Remaining
Retry strategy: [exponential backoff recommended]
Common errors:

Status Meaning Action

401 Invalid/expired API key Refresh credentials

429 Rate limited Back off, retry after Retry-After header

500 Server error Retry with backoff

Status	Meaning	Action
401	Invalid/expired API key	Refresh credentials
429	Rate limited	Back off, retry after `Retry-After` header
500	Server error	Retry with backoff

Terms of Service Notes

[Any restrictions on caching, storage, redistribution, attribution requirements]
[Link to full ToS]

SDK / Libraries

Official SDK: [npm package name if available]
Community libraries: [notable wrappers]


### 3.2 — Non-API Data Source Template

For each non-API source, create a file at `data-sources/[source-name].md`:

```markdown
# [Source Name]

> [One-line description]

| Field | Value |
|-------|-------|
| **Provider** | [Organization] |
| **Type** | Open Dataset / RSS Feed / Scraping Target / File |
| **URL** | [access URL] |
| **Format** | CSV / JSON / XML / HTML / PDF |
| **Update Frequency** | Real-time / Daily / Weekly / Monthly / Static |
| **License** | [license type — CC-BY, MIT, proprietary, etc.] |
| **Role** | Primary / Alternative / Supplementary |

## How to Access

[Step-by-step instructions — download, subscribe, or fetch]

## Data Fields Available

| Field | Type | Description |
|-------|------|-------------|
| ... | ... | ... |

## Ingestion Strategy

[How to get this data into Supabase — manual upload, scheduled cron, Edge Function, etc.]

## Legal & Attribution

[Any attribution requirements, usage restrictions, or licensing terms]

Phase 4 — Data Catalog

Create _DATA_CATALOG.md — a human-readable inventory of all data types available across all sources, filtered to the user's use case.

Structure:

# Data Catalog — [Project Name]

> All data types available for [use case description], organized by category.

## [Category 1 — e.g. "Properties" or "User Profiles"]

| Data Point | Type | Source(s) | Freshness | Notes |
|-----------|------|-----------|-----------|-------|
| [field name] | string/number/etc. | [API name(s)] | real-time/daily/etc. | [any notes] |
| ... | ... | ... | ... | ... |

## [Category 2 — e.g. "Market Data"]

| Data Point | Type | Source(s) | Freshness | Notes |
|-----------|------|-----------|-----------|-------|
| ... | ... | ... | ... | ... |

[Repeat for each logical category of data]

## Coverage Matrix

Shows which sources provide which data categories:

| Source | [Cat 1] | [Cat 2] | [Cat 3] | ... |
|--------|---------|---------|---------|-----|
| [API 1] | ✅ | ✅ | ❌ | ... |
| [API 2] | ❌ | ✅ | ✅ | ... |
| [Dataset 1] | ✅ | ❌ | ❌ | ... |

## Gaps & Recommendations

[Any data the user needs that isn't available from discovered sources, with suggestions for alternatives — AI generation, user-submitted data, manual entry, etc.]

Rules for the data catalog:

Only include data types that are relevant to the stated use case — don't list every field an API returns
Group by user-facing category, not by source
Use plain language — a non-technical stakeholder should be able to read this and understand what data is available
Call out gaps explicitly — if a needed data type isn't available from any source, say so

Phase 5 — Integration Blueprint

Create _INTEGRATION_BLUEPRINT.md — a step-by-step integration guide for wiring all sources into a React + Supabase + Vercel + AI Agents application.

Structure:

# Integration Blueprint — [Project Name]

> Step-by-step guide to integrate [N] APIs and [M] data sources into a React + Supabase + Vercel + AI Agents stack.

## Architecture Overview

[High-level description of how data flows through the stack]

[ASCII or Mermaid diagram showing]: External APIs → Supabase Edge Functions → Supabase DB → React Client ↕ AI Agents (OpenAI)


## Step 1 — Supabase Setup

### 1.1 — Database Schema

```sql
-- Tables derived from the data catalog
CREATE TABLE [entity] (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  -- fields mapped from API responses
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

[One CREATE TABLE per major entity, with comments explaining field origins]

1.2 — Row Level Security

-- RLS policies for each table
ALTER TABLE [entity] ENABLE ROW LEVEL SECURITY;

CREATE POLICY "..." ON [entity]
  FOR SELECT USING (auth.uid() = user_id);

1.3 — Environment Variables

# .env.local
SUPABASE_URL=your-project-url
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-key

# API keys for each external source
[API_NAME]_API_KEY=your-key
OPENAI_API_KEY=your-key

Step 2 — API Integration Layer

2.1 — Supabase Edge Functions

For each external API, create a Supabase Edge Function that:

Accepts a request from the client
Adds the API key server-side (never expose keys to the client)
Calls the external API
Transforms the response to match the DB schema
Optionally caches in Supabase

// supabase/functions/[api-name]/index.ts
import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'
import { createClient } from 'https://esm.sh/@supabase/supabase-js@2'

serve(async (req) => {
  // [Implementation specific to each API]
})

[One code block per Edge Function with full implementation]

2.2 — Data Transformation

// lib/transformers/[api-name].ts
// Maps external API response shapes to internal DB schema
export function transform[Entity](apiResponse: ExternalType): InternalType {
  return {
    // field mappings
  }
}

2.3 — Caching Strategy

Source	Cache Duration	Strategy	Rationale
[API 1]	1 hour	Supabase table + stale-while-revalidate	[why]
[API 2]	No cache	Direct passthrough	[why — e.g. real-time pricing]
[Dataset]	24 hours	Full table refresh via cron	[why]

Step 3 — React Client Integration

3.1 — Supabase Client Setup

// lib/supabase.ts
import { createClient } from '@supabase/supabase-js'

export const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!
)

3.2 — Data Hooks

// hooks/use[Entity].ts
import { useQuery } from '@tanstack/react-query' // or SWR, or Supabase realtime

export function use[Entity](params: Params) {
  return useQuery({
    queryKey: ['entity', params],
    queryFn: async () => {
      const { data, error } = await supabase
        .from('entity')
        .select('*')
        .match(params)
      if (error) throw error
      return data
    },
  })
}

[One hook per major data entity]

3.3 — Edge Function Calls (for fresh external data)

// lib/api/[api-name].ts
export async function fetch[Data](params: Params) {
  const { data, error } = await supabase.functions.invoke('[api-name]', {
    body: params,
  })
  if (error) throw error
  return data
}

Step 4 — AI Agent Integration

4.1 — Agent Architecture

[Describe how AI agents fit into the data flow]:

Which data gets sent to the AI agent as context
What the agent produces (summaries, recommendations, transformations, etc.)
Whether the agent runs on-demand (user-triggered) or in the background (automated)

4.2 — OpenAI Integration

// lib/ai/agent.ts
import OpenAI from 'openai'

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
})

export async function run[Agent](context: ContextType) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      { role: 'system', content: `[System prompt specific to this use case]` },
      { role: 'user', content: JSON.stringify(context) },
    ],
  })
  return completion.choices[0].message.content
}

4.3 — Agent as Supabase Edge Function

// supabase/functions/ai-[task]/index.ts
// Wraps the AI call in an Edge Function so the API key stays server-side

4.4 — Cost Estimation

Model	Input Cost	Output Cost	Est. Tokens/Request	Est. Monthly Cost
gpt-4o	$2.50/1M	$10/1M	[estimate]	$[estimate]
gpt-4o-mini	$0.15/1M	$0.60/1M	[estimate]	$[estimate]

Recommendation: [Which model to use and why, based on the use case complexity vs. cost]

Step 5 — Vercel Deployment

5.1 — Environment Variables

Set these in the Vercel dashboard under Project → Settings → Environment Variables:

Variable	Where to Get It
`NEXT_PUBLIC_SUPABASE_URL`	Supabase Dashboard → Settings → API
`NEXT_PUBLIC_SUPABASE_ANON_KEY`	Same
`[API_NAME]_API_KEY`	[Provider dashboard URL]
`OPENAI_API_KEY`	platform.openai.com

5.2 — API Routes (if using Next.js API routes instead of / in addition to Edge Functions)

// app/api/[endpoint]/route.ts

5.3 — Cron Jobs (if needed)

// vercel.json
{
  "crons": [
    {
      "path": "/api/cron/refresh-[data]",
      "schedule": "0 */6 * * *"
    }
  ]
}

5.4 — Deployment Checklist

All environment variables set in Vercel
Supabase Edge Functions deployed (supabase functions deploy)
Database migrations applied (supabase db push)
RLS policies verified
API rate limits tested under expected load
AI agent costs estimated and budget alerts set
Error monitoring configured (Sentry / Vercel Analytics)

Cost Summary

Service	Tier	Monthly Cost	Notes
Supabase	[Free/Pro]	$[X]	[what's included]
[API 1]	[Tier]	$[X]	[based on estimated usage]
[API 2]	[Tier]	$[X]	...
OpenAI	Pay-as-you-go	$[X]	[based on model + est. usage]
Vercel	[Hobby/Pro]	$[X]	[based on traffic estimate]
Total		$[X]/mo


---

## Execution Steps

### Step 1 — Scope the Project
Read `$ARGUMENTS`. If insufficient, ask the 4 elicitation questions. Produce an internal scoping summary.

### Step 2 — Research APIs & Data Sources
Use web search extensively. For each domain/data-type identified in scoping, search for authoritative APIs. Visit official documentation pages to verify endpoints, pricing, and auth methods. Check API aggregators for alternatives.

### Step 3 — Create the Output Directory
Create `_docs/api-sources/[project-slug]/`, `apis/`, and `data-sources/` subdirectories.

### Step 4 — Write _OVERVIEW.md
Summarize all discovered sources, their roles (primary/alternative/supplementary), and the overall data strategy.

### Step 5 — Write Individual API Files
One file per API in `apis/`. Follow the API documentation template exactly. **Include working code examples** — don't leave placeholders. If you cannot verify an endpoint, note it as "unverified."

### Step 6 — Write Individual Data Source Files
One file per non-API source in `data-sources/`. Follow the data source template.

### Step 7 — Write _DATA_CATALOG.md
Compile the full data inventory. Organize by user-facing category. Include the coverage matrix. Call out gaps.

### Step 8 — Write _INTEGRATION_BLUEPRINT.md
Produce the full integration guide. Every code block should be as close to production-ready as possible. Include actual SQL schemas, actual TypeScript types, actual Edge Function boilerplate.

### Step 9 — Report
Summarize what was created using the output format below.

---

## Output Format

After generating all files, output:

API & Data Source Research: [Project Name]

Sources Discovered

APIs: [count] ([count] primary, [count] alternative, [count] supplementary)
Data Sources: [count]
Data Types Cataloged: [count] across [count] categories

Files Created

_docs/api-sources/[slug]/_OVERVIEW.md — Executive summary
_docs/api-sources/[slug]/apis/[name].md — [one-line desc]
...
_docs/api-sources/[slug]/data-sources/[name].md — [one-line desc]
...
_docs/api-sources/[slug]/_DATA_CATALOG.md — Full data inventory
_docs/api-sources/[slug]/_INTEGRATION_BLUEPRINT.md — React + Supabase + Vercel + AI guide

Estimated Monthly Cost

Service	Cost
[itemized]	$X
Total	$X/mo

Coverage Gaps

[Any data the user needs that no source provides]

Next Steps

Review _OVERVIEW.md for the recommended source selection
Sign up for API keys (links in each API file)
Follow _INTEGRATION_BLUEPRINT.md step by step
Start with the primary APIs; add alternatives if you hit rate limits or need more coverage


---

## Anti-Patterns

- **Don't recommend APIs you haven't verified.** If you can't confirm an API exists and has current documentation, don't include it. Note any uncertainty explicitly.
- **Don't skip pricing.** Every API file must include pricing information. If pricing isn't publicly available, say "Contact sales — pricing not public" rather than guessing.
- **Don't recommend scraping when an API exists.** Always prefer official APIs over scraping. Only suggest scraping as a last resort, and always note ToS implications.
- **Don't write placeholder code.** Code examples should be as close to copy-paste-ready as possible. Use realistic variable names, proper error handling, and TypeScript types.
- **Don't conflate data types with API endpoints.** The data catalog should be organized by what the user *sees* (entities, fields), not by which API returns them.
- **Don't ignore free tiers.** Many projects can run entirely on free tiers. Always document the free tier first, then paid upgrades.
- **Don't ignore rate limits.** Rate limits are a production concern. Always document them and include retry/backoff strategies.
- **Don't be vague about auth.** "Uses API key" is not enough. Document exactly where to put the key (header, query param, bearer token) with a working example.
- **Don't forget AI cost estimation.** AI agents can be the most expensive part of the stack. Always include token/cost estimates for the specific use case.