Sora AI: Revolutionising Video Generation Models

The present report sheds light on Sora - a generative model developed for video data that overcomes the challenges often presented by conventional models. Traditional models generally limit themselves to specific categories of visual data, shorter videos, or standard-sized videos. On the contrary, Sora's capabilities portray a much wider range, yielding videos and images across various durations, resolutions, and aspect ratios. It can even generate high-definition videos of up to a minute.

Referring to successful large language models that have benefitted from training on internet-scale data, Sora creates a unified representation of diverse visual data by turning them into what are termed "patches". These patches, frequently demonstrated as effective representations for visual data models, result in a highly scalable and effective manifestation for training generative models on diverse video and image types.

Sora, a diffusion model, is trained to predict the original “clean” patches using input noisy patches along with conditioning information like text prompts. As training progresses, the quality of video generation improves!. It retains the ability to generate a variety of video styles with varying durations, resolutions, and aspect ratios.

What sets Sora apart from other video generation approaches is that it does not limit videos to a standard size but trains on data at their native size, leading to several advantages. Sora gives the flexibility to generate videos of wide-ranging sizes and aspect ratios, allowing swift prototyping at lower sizes before producing at full resolution. Training videos at their native sizes have shown to improve composition and framing, leading to improved results.

Furthermore, Sora does not limit itself to text-to-video generation; it can be prompted with pre-existing images or videos for an array of video and image editing tasks such as creating seamless video loops, animating static images, or extending videos in time either forward or backward.

While Sora demonstrates impressive capabilities, it is not devoid of limitations. Experimentally, it’s found that Sora struggles to generate objects that have very specific shapes like humans or animals, and fails to realistically animate videos. These areas need improvements for better results.

Disclaimer: The above article was written with the assistance of AI. The original sources can be found on OpenAI.