HomefellowBench
Attention
Choose what deserves attention today or during the coming week: commitments, preparation, decisions and unfinished work. Leave out what is already handled.
What should be on my mind?
Homefellow research · In development
Less to keep
in your head.
A benchmark for AI assistance with managing a home and family.
We’re building Homefellow to help families manage everyday life. This benchmark measures how well a system understands household information, recognizes what needs attention and chooses what is worth asking.
Why this matters
Useful help starts with
understanding the situation.
Household plans are scattered across conversations, messages, calendars and documents. Keeping them straight means remembering what changed, what is already handled and what needs to happen next.
We want to measure whether a system can use that information to offer help worth someone's time. A reminder can be too late. A question can ask for something already supplied. A long list can bury the important item.
One suite, three tasks
Different decisions.
Separate results.
Questions
Choose a useful question that someone can reasonably answer. Learn enough to help without turning their household into a form to fill out.
What would be worth asking?
Information gathering
Ask someone to find a document or detail when it would make useful help possible. Account for the effort of finding and supplying it.
What would be worth finding?
An empty list or a decision not to ask can be the right result. Each task has its own scorecard. There is no combined overall score.
The same starting point
Original sources.
Your system's approach.
Enter one task or several.
Choose your own models and memory.
- 01 / Receive
A source history
Synthetic conversations, messages, documents and calendar entries, with dates and audience permissions.
- 02 / Understand
Your system
Read the inputs, reconcile changes and decide what matters. The benchmark does not supply a resolved household profile.
- 03 / Respond
A requested task
Return attention items, a question or an information request. Evaluation considers the evidence available at that moment.
The repository includes 60 fictional households and an inventory of 407 concepts across 39 domains. These are development materials. They are not yet 60 runnable benchmark cases.
Browse the corpus ↗How we plan to evaluate
Does the help fit
the evidence and the moment?
The first rubric focuses on attention.
These are the questions it must answer.
Start with one caseMake the judgment
Make the judgment
possible to inspect.
Each reference item needs a reason and supporting sources. Several lists may be reasonable. We will test different time windows and changes such as a task being completed.
Read the evaluation plan ↗- Supported
- Do the sources justify the item? Are uncertainty and audience permissions respected?
- Timely
- Does it belong in this period, including preparation for something later?
- Useful
- Does it help with something that matters and remains unresolved?
- Selective
- Does the list make good use of limited attention, without repeats or unnecessary work?
Initial grading will use models, with human review optional. We will record evidence, model versions and disagreements. Model-judged results will be labeled provisional; agreement between models alone does not establish that people find the help useful.
Where the project stands
Testing the first
attention case.
The first attention case now connects original sources, a standalone local model, provisional assessments and an inspection report. Four snapshots and contrasting authored outputs help us test whether the evaluator distinguishes useful help from mistakes.
The focus is reliable measurement. Judgments remain provisional; there is no overall score, validated ranking or leaderboard. The corpus and five earlier question examples remain development material.
Explore the earlier question examples →Contribute
Show us a case
we should account for.
A missing preparation step. A reminder that should have stopped. A question that asks too much. Concrete examples help us define what the benchmark should reward.
Use fictional examples. Please do not submit private household records or correspondence.
Explore locally
git clone https://github.com/johncantrell97/homefellow-bench.git
cd homefellow-bench
python3 scripts/check_corpus.py
python3 scripts/build.py --check
python3 -m http.server 8080 --directory distRepository access is currently limited during development. With access, open localhost:8080 after these commands. Python 3.10 or newer; no API keys or model calls are needed.