Heyboxx

Navigating the LLM Hype Cycle

It's reasonably hard to manage a product life-cycle while Sam Altman is announcing the end of capitalism every 6 months. Practically our biggest problem was trying to make our system fully agentic before the technology was ready. Our customers were not naturally sales people, and we were trying to get them to use a product which would be more easily recognised by a BDR or someone in Rev Ops. We thought we could solve this by having the product run entirely autonomously, and then later had to strip back and replace features one by one until we found a balance between automation and quality/reliability.

LLMs in a production environment

Having a system be correct 90% of the time is great for demos, but in production 90% accuracy effectively useless. There were many steps involved in getting useful results.

• Monitoring goes without saying. We built our own suite of LLM specific tests, this was also useful for benchmarking different models for price/quality comparisons.
• Consensus voting, having multiple smaller models run on the same problem and vote on the answer was very helpful. See https://arxiv.org/abs/2511.09030.
• Qualitative checks. More than in traditional systems, quantitative checks rarely tell the full story. For one example, our tests were showing that all contact enrichment was bringing back 3 or 4 contacts at record low token cost. Looking into it in more detail we noticed that every Garden Centre owner being returned in the contacts was called David. Our web search had been misconfigured in the update, and the LLM was just guessing feasible contacts for every location.

Selling to SMEs

It's always tempting to sell to small companies. They're generally more sympathetic to new ventures and less intimidating than enterprise sales. However they are also more conservative when it comes to investing in technology and we found it hard to move from happy user to paying user.

Entity Resolution

One of the most frequently arising problems for us was working out how to model companies and contacts in a domain where there isn't an established identifier. For example there are companies which don't have a domain but have an instagram presence, or only a facebook page, or maybe multiple facebook pages, one for each branch. We eventually settled on domains as the most practical identifier, but there were lots of edge cases remaining for non-email related contact.