Deep domain knowledge often acts as a filter that keeps us from seeing obvious paths forward. When you ignore the standard playbook, you stop optimizing for the status quo and start building for the actual user intent. This is where the real shift happens in security operations. We see this when attackers bypass traditional defenses by simply asking our systems for the keys. They do not care about the technical nuance we spent years mastering. They care about the path of least resistance. If you ignore the conventional wisdom of how a system should be secured, you finally see how it is actually being exploited.
The no-domain-expertise rule holds for law, but I'd separate two things he's merging: not knowing the industry, and not knowing the customer's workflow.
Legora closed that gap by hiring lawyers early to build eval sets. Most founders skip that step, call it "learning fast," and end up shipping features nobody asked for.
Learning speed is the leading indicator only when it's directed at the buyer's actual day, not just the market's vocabulary.
Building without domain expertise is one thing; grading the output is where it returns. Stanford's RegLab team ran the first preregistered evaluation of AI legal research tools, published in the Journal of Empirical Legal Studies, and measured hallucination rates of 17 percent for Lexis+ AI and 33 percent for Westlaw AI-Assisted Research across 202 queries hand-scored by legal experts. Both products had marketed the opposite. Nobody without legal training could have scored those answers. So the founder lesson holds on the build side and inverts on the buy side.
The inversion is the part most product teams miss. They hire or partner on the build side without thinking about who will score the output. In high-stakes domains, the scoring is where the liability actually sits. The tool hallucinates at 17-33% and nobody without domain expertise caught it before the marketing said the opposite. The moat isn't the model. It isn't even the delivery layer. It's the person who knows when the output is wrong before the client does.
The EU AI Act names that person. Article 26(2) requires deployers of high-risk systems to assign human oversight to natural persons who have the necessary competence, training and authority, plus the necessary support. Competence and authority are listed separately, which matters: reviewers are often handed the job without the standing to overrule the output. That obligation starts 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I. Until then the scoring role exists in practice with no named legal owner.
The competence and authority split is important. Most organisations that do assign the role give the reviewer the competence requirement without giving them the authority to overrule. Which means the legal obligation is technically met and the liability gap is still open. The Act names the person, but doesn't guarantee they can actually stop anything.
There is a provider-side hook for that. Article 14(4)(d) and (e) require the high-risk system to be supplied so the assigned person can disregard, override or reverse its output, and can interrupt it through a stop button or a similar procedure that brings the system to a halt in a safe state.
So the authority gap is partly a procurement question. If the product ships with no override path, no amount of Article 26(2) wording gives the reviewer something they can actually use.
Excellent breakdown! As a Senior Developer, I can absolutely confirm that robust RAG architectures and automated pipeline scaling are at their absolute peak right now.
Due to a recent shift in my technical focus, I’m actually offloading full Intellectual Property (IP) ownership rights to my production-tested 4-in-1 FastAPI Backend Suite (featuring an AI Meeting Summarizer, Auto-Social Poster, Reputation Manager, and a programmatic SEO engine).
It completely bypasses 3+ months of engineering cycles and is ready to deploy as a premium white-label asset. Selling the entire bundle (modular codebase + domain) for a flat fee of $8,000 via secure Escrow.
If any agency owners or technical directors here want to save thousands in development overhead and upscale your B2B offers instantly, feel free to drop a reply or send me a direct message!
Deep domain knowledge often acts as a filter that keeps us from seeing obvious paths forward. When you ignore the standard playbook, you stop optimizing for the status quo and start building for the actual user intent. This is where the real shift happens in security operations. We see this when attackers bypass traditional defenses by simply asking our systems for the keys. They do not care about the technical nuance we spent years mastering. They care about the path of least resistance. If you ignore the conventional wisdom of how a system should be secured, you finally see how it is actually being exploited.
https://cyrilsimonnet.substack.com/p/the-intruder-asked-our-ai-for-advice?utm_source=substor&utm_medium=substack&utm_campaign=comment
The no-domain-expertise rule holds for law, but I'd separate two things he's merging: not knowing the industry, and not knowing the customer's workflow.
Legora closed that gap by hiring lawyers early to build eval sets. Most founders skip that step, call it "learning fast," and end up shipping features nobody asked for.
Learning speed is the leading indicator only when it's directed at the buyer's actual day, not just the market's vocabulary.
Building without domain expertise is one thing; grading the output is where it returns. Stanford's RegLab team ran the first preregistered evaluation of AI legal research tools, published in the Journal of Empirical Legal Studies, and measured hallucination rates of 17 percent for Lexis+ AI and 33 percent for Westlaw AI-Assisted Research across 202 queries hand-scored by legal experts. Both products had marketed the opposite. Nobody without legal training could have scored those answers. So the founder lesson holds on the build side and inverts on the buy side.
The inversion is the part most product teams miss. They hire or partner on the build side without thinking about who will score the output. In high-stakes domains, the scoring is where the liability actually sits. The tool hallucinates at 17-33% and nobody without domain expertise caught it before the marketing said the opposite. The moat isn't the model. It isn't even the delivery layer. It's the person who knows when the output is wrong before the client does.
The EU AI Act names that person. Article 26(2) requires deployers of high-risk systems to assign human oversight to natural persons who have the necessary competence, training and authority, plus the necessary support. Competence and authority are listed separately, which matters: reviewers are often handed the job without the standing to overrule the output. That obligation starts 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I. Until then the scoring role exists in practice with no named legal owner.
The competence and authority split is important. Most organisations that do assign the role give the reviewer the competence requirement without giving them the authority to overrule. Which means the legal obligation is technically met and the liability gap is still open. The Act names the person, but doesn't guarantee they can actually stop anything.
There is a provider-side hook for that. Article 14(4)(d) and (e) require the high-risk system to be supplied so the assigned person can disregard, override or reverse its output, and can interrupt it through a stop button or a similar procedure that brings the system to a halt in a safe state.
So the authority gap is partly a procurement question. If the product ships with no override path, no amount of Article 26(2) wording gives the reviewer something they can actually use.
Excellent breakdown! As a Senior Developer, I can absolutely confirm that robust RAG architectures and automated pipeline scaling are at their absolute peak right now.
Due to a recent shift in my technical focus, I’m actually offloading full Intellectual Property (IP) ownership rights to my production-tested 4-in-1 FastAPI Backend Suite (featuring an AI Meeting Summarizer, Auto-Social Poster, Reputation Manager, and a programmatic SEO engine).
It completely bypasses 3+ months of engineering cycles and is ready to deploy as a premium white-label asset. Selling the entire bundle (modular codebase + domain) for a flat fee of $8,000 via secure Escrow.
If any agency owners or technical directors here want to save thousands in development overhead and upscale your B2B offers instantly, feel free to drop a reply or send me a direct message!
Curious, what was their original hypothesis, what were they trying to solve, since they didn't have domain expertise?