From Cloud Giants to Pocket AI

I’ve been experimenting with open-weight and open-source models since they came out. The idea that I could run a mini ChatGPT on my laptop was always too intriguing.

I started with LLaMa 2 (a large language model by Meta) back at Stanford in 2023. I wasn’t aware of Ollama (a tool to easily run local LLMs) back then, so I took the hard route—trying to figure out Hugging Face (a popular AI model repository and library) for the first time. After spending two days straight struggling because I couldn’t understand simple code (and ChatGPT had no clue how to make it run), I finally found out about Ollama on Reddit. After that, I never looked back when it came to downloading new models - it got simpler.

I started using local LLMs for small tasks, and it turns out they weren’t bad for basic text editing and email refinement - things I used to go to ChatGPT for. (To be honest, I used to go to ChatGPT for every minor inconvenience :) … still do.)

Gradually, I started using them more. Later on, I learned about RAG (Retrieval-Augmented Generation, a technique that improves AI accuracy by connecting it to a database) and the cool things you could do if you had a vector database (a database where words are stored as numbers) at your disposal. It was really amusing to see how a simple vector search could help an LLM answer questions with high precision. This basically solved my problem of not being able to edit documents using my local LLaMa. I ended up diving deeper into RAG and fell into the whirlpool of agentic RAG and other emerging techniques. Needless to say, I implemented these in multiple hackathons and college projects (which ultimately got labelled as "just chatbots" haha).

In July 2024, I took on an interesting and challenging research project: comparing LLaMa 3.1 8B (an 8-billion-parameter version of Meta’s LLaMa 3) hosted locally versus a LLaMa instance connected to ChromaDB (a vector database). I vectorized (turned words into numbers in a way that keeps their original meaning and context intact) a dataset with 1.2 million data points, which took some effort to optimize - I had to figure out a producer-consumer method (using my CPU and GPU at the same time to reduce compute workload) to ensure all points were vectorized within two hours (that too, locally :)). The goal was to test how feasible it would be for small hospitals to locally vectorize their patient data and run it with a local LLM, compared to sending sensitive patient data to external servers (which, to say the least, is very risky). The dataset contained synthetic patient data, and my objective was to generate differential diagnoses based on the symptoms presented by these "patients."

I tested multiple locally hosted LLMs: LLaMa 3.1 8B, LLaMa 3.2 1B (a 1-billion-parameter version), LLaMa 3.2 3B—and interestingly, some of these got released while I was in the middle of this research. Couldn't have been better timing.

Turns out, RAG outperformed standard LLMs. (I even created a new scoring metric to evaluate this - it was actually pretty fun.)

This made me realize that while I do enjoy the fast responses from ChatGPT (and it knows me so well now), I’d definitely prefer keeping my documents locally and asking questions directly. While I firmly believe that the internet (especially ChatGPT) already knows everything about me and there’s not much I can "hide," I’d still like to have local models process some documents for me. Eventually, I’d love to run local agents (AI-powered programs that can reason, plan, and take actions independently based on custom instructions and data) that can analyze this data and maybe even automate some of my boring weekly emails.

It’s an exciting prospect - one that seems to be getting lost in the race towards billion-dollar data centers. And while I do think we need these data centers to develop better LLMs, I also believe that, in the future, every person with a powerful laptop will be running 30B-parameter LLMs with ease. Eventually, we’ll all shift to local instances of these models to safeguard our personal data (at least, whatever is left of it :)).

The thing is, most people using local LLMs today are tech savvy computer science majors—not the general public. That is why I wanted to write this blog: to make sure everyone knows how easy it is to run a local LLM and get it to handle basic tasks. You can even try out the super cool DeepSeek-R1 models locally.

LM Studio (a GUI-based tool for running local LLMs) has been a game-changer for me since it saves me from dealing with the CLI when running models locally. I highly recommend people at least try it out. For my projects and codes, I love Ollama because it integrates very easily with my RAG pipelines on LangChain.

The future of AI isn’t just in massive cloud data centers - it’s in our own hands, running on our own machines. As models improve and hardware catches up, we won’t just use AI; we’ll own it.

The age of Personal AI is coming: where AI runs locally, adapts to our needs and memories, and works for us without sending data to external servers or big tech.

From Cloud Giants to Pocket AI

Here are some resources that helped me set up my local AI models: