gemini

Google Gemini AI models for multimodal tasks. Use for multimodal AI.

SKILL.md

GitHub

Gemini

Gemini is Google's native multimodal foundation model family, featuring native audio, image, and video comprehension alongside million-token context windows and fast tool-calling.

When to Use

  • Massive Multimodal Understanding: Gemini 1.5 Pro / Flash processing up to 2 million tokens of text, hours of video, audio, and large codebases.
  • Structured JSON Extraction: Enforcing rigid response schemas with Pydantic and native response_schema.
  • High-Speed Real-Time Inference: Gemini 1.5 Flash for sub-second, cost-effective conversational assistants and summarization.
  • Code & Repository Analysis: Analyzing entire GitHub repositories in a single context window.

Quick Start

from google import genai

client = genai.Client()

response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents="Explain quantum computing in one short paragraph.",
)

print(response.text)

Core Concepts

Structured Output with Pydantic & Google GenAI SDK

Enforcing type-safe JSON schema responses:

from google import genai
from google.genai import types
from pydantic import BaseModel, Field
import os

client = genai.Client(api_key=os.environ.get("GEMINI_API_KEY"))

class InvoiceExtraction(BaseModel):
    invoice_number: str
    vendor: str
    total_amount: float = Field(description="Total invoice amount in USD")
    items: list[str]
    is_paid: bool

response = client.models.generate_content(
    model='gemini-1.5-flash',
    contents='Invoice #INV-2026-908 from CloudCorp for $1,450.00. Included 2 Cloud Servers and 1 Storage bucket. Paid via Wire.',
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=InvoiceExtraction,
        temperature=0.1
    ),
)

extracted: InvoiceExtraction = InvoiceExtraction.model_validate_json(response.text)
print(f"Extracted Invoice: {extracted.invoice_number}, Total: ${extracted.total_amount}")

Native Video & Audio Multimodal Analysis

Processing video files directly using the File API:

from google import genai

client = genai.Client()

# Upload video file to File API
video_file = client.files.upload(file="presentation.mp4")

# Query video content directly
response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents=[
        video_file,
        "Summarize the key decisions made between timestamp 04:15 and 08:30 in this meeting."
    ]
)

print(response.text)

Context Caching for Massive Contexts

Reducing cost on recurring queries against massive datasets:

from google import genai
from google.genai import types
import datetime

client = genai.Client()

# Create cached content for a massive documentation corpus
cache = client.caches.create(
    model='gemini-1.5-pro',
    config=types.CreateCachedContentConfig(
        contents=[large_documentation_file],
        ttl=datetime.timedelta(hours=2),
        display_name='api_documentation_v2'
    )
)

# Query against cache at significantly reduced per-token cost
response = client.models.generate_content(
    model='gemini-1.5-pro',
    contents='What are the rate limits for the batch endpoint?',
    config=types.GenerateContentConfig(cached_content=cache.name)
)

Common Patterns

Multimodal Document Analysis (PDF / Audio / Video)

Problem: Processing multi-page PDF documents or audio files without complex local extraction libraries.

Solution: Upload file using the Google GenAI File API:

# Upload document directly to Gemini File API
doc_file = client.files.upload(file="contract.pdf")

response = client.models.generate_content(
    model="gemini-2.0-flash",
    contents=[
        doc_file,
        "Extract the governing law, termination notice period, and liability cap."
    ]
)

print(response.text)

Best Practices

Do:

  • Target the modern Google GenAI SDK (google-genai) instead of legacy deprecated libraries.
  • Use gemini-1.5-flash for high-volume, low-latency tasks and gemini-1.5-pro for deep reasoning.
  • Enforce structured outputs with response_schema and Pydantic models for reliable API integrations.
  • Leverage Context Caching for prompts exceeding 32k tokens that are reused across multiple calls.

Don't:

  • Upload raw video/audio base64 payloads inline; use the client.files.upload() API.
  • Leave sensitive credentials exposed in client-side code; proxy requests through a secure backend.
  • Use high temperature settings (> 0.4) when strict schema compliance or factual retrieval is required.

Troubleshooting

Error Cause Solution
google.genai.errors.APIError: 400 Invalid argument Unsupported file format or invalid model name. Verify model identifier (e.g. gemini-2.0-flash or gemini-1.5-pro).
ResourceExhausted: 429 Quota exceeded Rate limit hit on free tier requests per minute. Implement client-side rate limiting or upgrade to paid tier API key.
Missing GEMINI_API_KEY environment variable Environment variable not exported in shell. Export export GEMINI_API_KEY="AIzaSy...".

References