Generate images and video | AI-103 | Episode 23

Generative AI goes beyond text: Microsoft Foundry can deploy specialized models that generate or modify images and videos from prompts and reference media. The key is understanding which model, API, and workflow fits the task.

Choose the right model

Models have different capabilities. For image generation, select a model supporting the Text-to-Image inference task, such as GPT Image or FLUX. Video generation requires a video-capable model such as Sora. Test models and prompts in the Foundry playground before integrating them into code.

Image generation

Flow: Prompt / Reference → Image Model → Generate/Edit → Image

Use images.generate() to create images from text. Typical controls include the model, prompt, number of images, and output size. Depending on the API/model, generated image data can be returned Base64-encoded and decoded by the application.

# TEXT → IMAGE
image = client.images.generate(
    model="image-model",
    prompt="A rover exploring Mars",
    size="1024x1024"
)

Know the distinction: generation creates new visual content, while editing/inpainting modifies an existing image using a prompt and reference media or mask.

Video generation

Three workflows are especially important:

Text → Video = generate a new scene
Video + Prompt → Video = remix an existing video
Image + Prompt → Video = animate/reference an image

# 1. TEXT → VIDEO
video = client.videos.create(
    model="sora-2",
    prompt="Mountain lake at sunrise",
    seconds="4"
)

# Generation is asynchronous → poll status
while video.status not in ["completed", "failed", "cancelled"]:
    video = client.videos.retrieve(video.id)

# 2. VIDEO → REMIXED VIDEO
remix = client.videos.remix(
    video_id=video.id,
    prompt="Change to warm sunset lighting"
)

# 3. IMAGE → VIDEO
from_image = client.videos.create(
    model="sora-2",
    prompt="Add gentle camera movement",
    input_reference=open("lake.png", "rb"),
    seconds="4"
)

# Completed video → retrieve output
content = client.videos.download_content(video.id)

Video generation is asynchronous: create starts the job → retrieve polls its status → download_content retrieves the completed output. Supported resolution and duration values are model-specific, so don't assume arbitrary values.

Remember the API mapping

images.generate() → Text-to-Image
videos.create(prompt) → Text-to-Video
videos.remix(video_id, prompt) → Remix existing video
videos.create(input_reference=image) → Image-to-Video
videos.retrieve(video_id) → Check generation status
videos.download_content(video_id) → Retrieve completed video

Exam mindset: first identify the required modality and operation (generate, edit/remix, or reference-based generation), then select a compatible model, configure its supported generation parameters, and use the corresponding API workflow.