Rajan Agarwal

I study Software Engineering at the University of Waterloo. This fall, I will work on RL for web agents as a Research Engineer at the Amazon AGI Lab. Previously, I built multimodal video editing agents for hollywood at Kino AI and low-level train safety systems at Hitachi Rail.

I am a deeply technical person. I'm constantly building, learning and breaking things. I'm obsessed with learning how things work and designing novel solutions to problems I can't get out of my head. Right now, I'm most curious about multimodal models and coding agents.

Multimodal Video Search

Multimodal Video Search

2025

Building multimodal machine learning systems to help filmmakers streamline their creative process. Working on video understanding, generation, and editing.

Demo Video
Distributed Training

Distributed Training

2024

A decentralized cross-device model training system with model and tensor parallelism to reduce compute needed to train large models.

Codebase
AI Agents as Citizens

AI Agents as Citizens

2024

City simulation of Los Angeles with AI Agents, simulating human behaviour and optimizing transit routing with RL.

Project Breakdown
Self-Driving Delivery

Self-Driving Delivery

2023

Built an autonomous model tank that navigates campus walking paths and delivers small parcels. Implemented path planning, simulations, object detection, PID control, and CV.

Project Breakdown
Multilingual Understanding

Multilingual Understanding

2024

Research at Cohere for AI to represent different languages as different modalities for training multilingual language models.

Project Breakdown
Humanoid Robot

Humanoid Robot

2024

Leading behaviour and interaction software for a humanoid robot design team in Waterloo. Built GPU-optimized 3D voxel grids for awareness models.

Website
Shapeshift

Shapeshift

2023

Deep learning pipeline that analyzes seismic frequencies and local policy to design affordable earthquake-resistant buildings.

Website
Self Sustaining Startup

Self Sustaining Startup

2024

Built an entirely self-sustaining startup on AI Agents, which built the startup from scratch, made business plans, did branding, built a full-stack app and put a job posting on LinkedIn.

Project Breakdown
Offline Mesh Network

Offline Mesh Network

2022

An offline mesh network written in Swift to allow for cross-device transfer of files entirely offline, creating a chain of encrypted nodes.

Bracket Bots

Bracket Bots

2024

Helping with the software behind Bracket Bots, a self-balancing robot for under $200 that can roam around your house.

Website
LLM Memory Architecture

LLM Memory Architecture

2023

Long-term memory with multimodal knowledge graphs to search 7 days of video and audio within 5 seconds. Winners @ Hack the North 2023.

Project Breakdown
Driverless Car

Driverless Car

2024

Self-driving car design team, WATonomous, in Waterloo. I'm working on a low-latency data-driven controls model for the car.

Project Breakdown
Recommendations System

Recommendations System

2024

Worked with Aviato to build a recommendations system for recruiting top AI talent amongst 300k+ engineers.

Website
Generative UI

Generative UI

2024

Worked with Tempo Labs to build generative UI agents in multiple programming languages concurrently.

Website
Language Diffusion Model

Language Diffusion Model

2024

Inpainting with diffusion language models to fill in missing text. In Progress.

Image Generation

Image Generation

2023

Trained a smaller VQGAN+CLIP to generate images from text and poetry prompts. Scaled up inference to run on MacBooks efficiently.

Project Breakdown
Audio2Vec

Audio2Vec

2024

256 Dimension audio embeddings for semantic audio analysis using waveforms and Fourier Transforms, trained with contrastive learning.

Project Breakdown
NYC Train Safety

NYC Train Safety

2024

Worked on safety simulation software for New York trains with custom network protocols.

Company Website
Crowdsourcing Policy

Crowdsourcing Policy

2024

Generating policy recommendations with AI agents from citizen complaints.

Twitter Thread
Matchmaking

Matchmaking

2024

Experimented with graphs and embeddings to match people to events, used on 800+ person events in Waterloo and built in Stanford at TreeHacks.

GitHub Repo
Public Funding Research

Public Funding Research

2022

Graphing 500K+ papers and dynamically distributing funding given to ideas based on relevance. Winners @ Hack the North 2022.

Devpost
Visual Reasoning

Visual Reasoning

2024

Foundational models for visual reasoning in robotics to handle unseen objects and dynamic environments. Demoed at W24 Socratica Symposium.

Celeritas Research

Celeritas Research

2024

Technical Advisor for a high school team trying to win the F1Tenth Driverless Competition, using Reinforcement Learning policies and ViT models for autonomous driving.

Website
IDE for Formal Proofs

IDE for Formal Proofs

2024

Built Axiom, a multiplayer IDE for formal proofs, for the SE212 Logic and Computation course at the University of Waterloo.

GitHub Repo
Compiler from Scratch

Compiler from Scratch

2024

Built a compiler from scratch in C++, implementing a lexer, parser, and compiler for a simpler version of C.

Camp Social

Camp Social

2022

Full-stack engineering intern at Camp Social. Built core microservices and APIs with Ruby on Rails, Typescript and Solidity for efficient smart contract generation.

Hack Club

Hack Club

2022

Built a full-stack web app for multilingual applications to join Hack Club, resulting in 7000+ sign ups globally and featured in the Github Newsletter.