When Every Fact Is True but the Sentence Is Wrong

While building VyomaVeda, I ran into a kind of LLM hallucination I wasn't expecting.

I compute the entire birth chart in code and use the model only to turn it into natural language. All the reasoning is done before the model sees anything, so I assumed I was mostly safe from made-up facts.

I was wrong.

The model gets the planet placements, and separately, which planet rules each house. Every fact I hand it is correct. But every now and then it puts two of them together and states a third fact I never gave it.

Here's the one that caught me. On a Leo ascendant chart, the model had two correct facts in front of it: Saturn is in the 4th house, and Mars rules the 4th house. It combined them and wrote:

"Saturn is your 4th house lord."

From a Leo ascendant the 4th lord is Mars, not Saturn, so the sentence is false. But look at what actually happened. The model didn't invent Saturn, and it didn't invent the 4th house. Both are real, and both were sitting in front of it. It invented the relationship between them.

Why my guard missed it

I already had a guard against made-up facts. It checked every entity the model named against the chart. Is Saturn a real planet here? Yes. Is the 4th house one we can talk about? Yes, Saturn is sitting in it. Every noun in the sentence was a real, computed fact, so the guard let it through.

That's the gap. The guard checked the facts one at a time and never checked the claim they were being used to make. "Saturn rules the 4th" isn't a statement about Saturn or about the 4th house on their own. It's a statement about the relationship between them, and a check that looks at entities one at a time can't see a false relationship built out of true ones.

I spent a while looking for a bug in my calculations. There wasn't one. The chart was right, the payload was right. The model was combining correct things into a wrong thing.

Check the claim, not the words

The truth was already in the engine. For any ascendant, I can say which planet actually rules each house.

def house_lords_for_lagna(lagna):
    """house -> the set of planets that really rule it, for this ascendant."""
    lords = {}
    for house in range(1, 13):
        sign = get_house_sign(lagna, house)
        lords[house] = {SIGN_RULERS[sign]}   # e.g. {"Mars"} for the 4th from Leo
    return lords

So the fix was to stop trusting the sentence and rebuild the claim inside it. Pull every "planet rules house N" assertion out of the prose, turn each one into a plain (planet, house) pair, and check that pair against the real map.

def lordship_mismatches(text, house_lords):
    bad = []
    for claim in extract_lordship_claims(text):   # -> (planet, house) pairs
        owners = house_lords.get(claim.house)
        if not owners:
            continue                # house not in the map: never invent a violation
        if claim.planet not in owners:
            bad.append(claim)       # "Saturn" isn't in {"Mars"} -> caught
    return bad

"Saturn isn't in the set of rulers for the 4th house" is a one-line lookup. The lookup was never the hard part. Getting from a sentence a person can phrase ten ways down to that one clean pair was.

Normalize the claim instead of matching it

My first instinct was to write patterns for the claim itself. Match "Saturn is the 4th house lord." Match "the 4th house lord is Saturn." Match "Saturn rules the 4th." That never ends, and I had two older validators built that way, each with its own blind spot. One missed the word "house" in the middle of the phrase. The other only worked when the planet sat right next to the house.

So instead of matching whole claims, the code does three small things. It finds an anchor, a span that asserts lordship of a numbered house ("4th house lord", "lord of the 4th", "rules the 4th"). It binds that anchor to a planet in the same clause. And it emits a plain (planet, house) pair. Every phrasing collapses to the same pair, and a new way of writing the claim needs one new anchor, not a whole new rule for judging a sentence.

The fix had its own hole

A planet often rules two houses, and prose says so in one breath: "Saturn rules the 6th and 7th." My anchor only caught the first house, so "Saturn rules the 6th and 4th" passed clean on a Leo chart. The 6th was right, and the false half was never even pulled out to check.

No test caught it. A real read did. Chart tours use that phrasing constantly, and one live answer had five of them. So the extractor learned to follow the "and" and bind every house in the list to the same planet. That's the usual shape of these fixes: you close the hole you can see, and the next one turns up in something a real person read.

One rule that matters

If a house is missing from the map, the check does nothing. Absence means the map couldn't be built, and a guard that flagged a violation from missing data would just be inventing a fact of its own. It only fires when it can prove the relationship is wrong, never when it can't tell.

The takeaway

The model wasn't making up new facts. It was making up new relationships between correct ones. That's the harder failure to catch, because every piece of it checks out, and a guard that validates entities one at a time will pass it every time.

So the check has to move up a level, from the words to the claim. Rebuild the assertion the sentence is making, reduce it to something you can compare against what you computed, and test that. In my case the claim was "planet X rules house N," and the answer was a dictionary I already had.