Searching for a place for 10: why did 44 open places never show up?

This morning I read one line of a bug-fix log and went straight back to my old SQL, afraid some of it has the same bug.

The bug sits in a stay-search tool. Ask for a place for 10 and the tool answers 13, which sounds like the whole search. In fact 44 more open places were never looked at, only because the host hadn't written down a capacity, and the answer doesn't mention them at all.

I measured the numbers here on 5 November against the real tool, for the nights of 6 to 8 November. Figures taken from the log are marked as the log's. AI helped with the measuring and the draft.

Tiếng Việt: Tìm chỗ ở cho 10 người, sao 44 chỗ còn trống chẳng hiện ra?.

Where it goes wrong

There is a small directory of places to stay that I work on. It has search_stays, a function an AI assistant calls on a guest's behalf: give it dates and a party size, get back the places open for all those nights. A function like this is called an MCP tool, after the standard that lets AI call outside functions. Suppose a guest asks for somewhere for 10 people. The tool answers count: 13, total: 13, meaning 13 places, and that's all.

Those 13 are real. "That's all" is what's wrong. I counted every place open on those nights:

Open places	Count
Capacity stated, holds 10	13
Capacity stated, holds fewer than 10	12
No capacity stated	44
All together	69

The tool only looked at the first two rows. Whether the 44 in the third row fit, nobody knows, because it never looked at them. (The log counted 68 and 43 at the time of the fix. One off because calendars keep moving.)

Someone on a web page who sees too few results can drop the filter and look again. An AI assistant has only the number in hand, so it tells the guest "13 places fit" with great confidence. The guest has no idea there are 44 more to ask about.

Why those 44 vanished

When a host hasn't stated a capacity, that cell in the database is NULL, meaning "not known", which is not the number 0. The old code filtered with capacity >= 10. Ask whether "not known" is above 10 and the database answers "unknown", and WHERE keeps only the rows that come out true. So filtering >= 10 plus filtering the opposite still doesn't cover the table. I rebuilt the table from the numbers I had just measured and ran it on SQLite: 69 rows, capacity >= 10 keeps 13, not capacity >= 10 keeps 12, and 44 rows sit on neither side.

The funniest case is travelling alone: for one person anywhere fits, and the tool still answers 25 and leaves 44 behind. Try dragging below. Each dot is an open place, with the capacities measured this morning. Wherever you drag, the 44 stays still, because those places never enter the comparison at all.

• 1 guest: 25 fit, 44 unknown
• 10 guests: 13 fit, 12 too small, 44 unknown
• 20 guests: 3 fit, 22 too small, 44 unknown
• 30 guests: 0 fit, 25 too small, 44 unknown

Three times in three days, one shape

This tool was only written before dawn on 3 November. By the morning of the 5th it had tripped three times, each time in the same way.

• 3 November, morning: bad input. Passing type: "BUFFET", which is not a kind of stay, returned success with 0 places instead of an error.
• 3 November, evening: a cut. 14 places matched, the default limit was 10, and the answer said count: 10 with no sign of the other 4.
• 5 November, morning: missing data. The 44 places above.

All three are partial answers that talk like whole ones. Each time the log fixed exactly one spot and the next one stayed, because the rows returned really were correct, so nothing in the result looked wrong. A bug of this kind only shows when someone asks the reverse question: how much was left behind?

The fix admits the number, and the guest still can't see the place

The fix does the hard part right: the filter moved from the database up into code, and the answer gained unchecked: { noCapacity: 44, note }. The note says plainly that those places could not be checked, never that they fit, and suggests create_inquiry to ask the host. I like that sentence because it refuses to guess. The log stops there.

I reran it only to check the numbers, and caught on a very ordinary question: 44 places, which ones? The answer counts 44 and names none of them, and asking a host takes a name. So I dropped the filter and asked for as much as I could: 25 of 69, limit tops out at 25, and there is no second page. The list sorts by capacity ascending, Postgres puts NULL last, and exactly 25 places state a capacity. So the longest list is exactly those 25, with none of the 44.

The answer advises raising limit or narrowing, so I did: 60 calls, all five kinds of stay, eight bedroom levels from 1 to 10. Together they reached 48 of 69. 21 open places did not appear in any call. There is still a long way round: read each host, then ask for each unit's calendar, about 283 calls, the very thing this tool exists to avoid.

So counting what you left behind is half the road, and the other half is showing the way to it. I sent these numbers to whoever tends that site, and which fix to pick is theirs to choose.

How to check your own filter

This runs on both Postgres and SQLite; change the table and column names. If fits plus too_small does not equal total, your filter is leaving rows behind and not saying so.

select count(*) as total,
  count(*) filter (where capacity >= 10) as fits,
  count(*) filter (where not capacity >= 10) as too_small,
  count(*) filter (where capacity is null) as unknown
from stays;

For a tool that returns a list, above all one an AI calls, I now ask three things:

1. How many matched, and how many did I hand over?
2. How many rows could the filter not check, and does the answer say so?
3. Is there a way for the caller to reach those rows? This one takes calling it as an outsider, because in the code the advice "raise limit" still reads as perfectly fine.

One more place to trip: whether NULL comes first or last depends on the database. With the same order by capacity asc, the tool on Postgres puts NULL last, and SQLite, when I ran it, puts NULL first. Cutting a list after a sort like that is choosing for people, sometimes without knowing it.

What I still don't know

• Of those 44, how many can really hold 10 people? Perhaps most are small and 13 turns out nearly right. Nobody knows until the hosts write the number down.
• Does a real assistant read unchecked, or still only total? I have not tried one. I have not read the logs either, so I don't know whether a real guest ever heard "13 places": the guest at the start of this note is a scenario.
• One more layer of filtering is still silent. A place with too old a calendar does not appear; the tool's description says so, but the answer does not count them. From outside I can see 258 listings in the directory and 69 places open, and cannot tell a fully booked place from a stale calendar.
• The SQL above only catches NULL. A column that writes 0 or an empty string for "unknown" slips through quietly, and I have no tidy check for that yet.

Has a filter of yours ever answered a little too neatly? Write me a letter below.