BuffBuddy

Answers to basic student questions at CU (registration holds, tuition, financial aid, housing, IT) are spread across dozens of university sites. A general chatbot either sends you a link or makes something up. I wanted one that pulls the actual steps, deadlines and phone numbers out of official pages, and that refuses when it hasn't got a source.

A crawler walks colorado.edu pages, respects robots.txt, and skips pages that haven't changed since last time by comparing a hash of the content. The text is split into chunks of about 350 words with some overlap, embedded locally with bge-small, and stored in ChromaDB. A few pages block crawlers, so I wrote those up by hand instead of trying to get around the block.

When you ask something, a FastAPI server pulls the top six chunks and drops any that are too far from the question. If nothing survives, it declines. Otherwise it streams an answer from Claude with the sources attached. There's also a demo sign-in mode that loads a fake student record and shows what's still outstanding, like holds or missing immunization forms.

The whole thing depends on how strict that relevance cutoff is. I built a set of 57 labeled questions: 38 it should answer and 19 it should refuse, including some tricky near misses. The eval script only scores retrieval, so it doesn't cost anything to run. At the cutoff I picked, the right source shows up in the top six every time, and it makes the right answer or refuse call on 91% of questions.

A slightly stricter cutoff scored 95%, but it only won because of a few specific questions in the set, so I didn't take it. The misses that are left are mostly questions about the wrong school, like Colorado State tuition, which a scope rule in the prompt should fix.

Runs locally. It was never deployed.