What Stage Is Your Project? Takeaways For AI Engineering: How to Adjust Your Tactics to Fit Your Project's Maturity

Dear friends,

Moving forward on early stage, 0-to-1 projects and mature projects requires very different tactics. For those who aspire to be skilled at AI Engineering, I've found that selecting the right tactic based on stage of project is one of the hardest but most important things to learn.

In the 5 letters on the AI Engineering Skills Map, you may have noticed calibration to the project stage was a recurring theme. This letter explains why. For many AI engineering tasks, like building evals, choosing software architecture, or getting product feedback, the right option usually depends on the project stage.

Take the task of evaluating an AI system — say, an automated customer-service email system. In an early-stage project, you might manually examine a dozen examples and check how sensible they are. A later-stage project might have hundreds of test examples and a written rubric for judging the quality of AI-written emails. A mature product might have tens of thousands or more test examples, a more detailed rubric, and rigorous processes for evaluating not just the quality of an email but its downstream effects (such as whether it causes a customer to be more likely to return).

Calibrating to the right stage of a project is important for many other AI Engineering skills. It is not helpful to over-design at the early stages, or under-design a mature product. For an early stage project, if your primary goal is to quickly test a product idea, a casual software architecture design might be okay, with minimal thought given to efficiency, data schemas, cost of third party services, and so on. But as a project matures, giving careful thought to tradeoffs like latency, availability, consistency, reliability, maintainability, simplicity, and cost will give you a better outcome.

Similarly, getting product feedback can range from pulling aside 2-3 people and asking what they think, to running large-scale user studies, A/B tests, and analyzing product usage data.

One way to gain experience with different approaches is to work on different projects spanning early stage and mature ones. Someone working in a startup might learn the quick ways to do evals, and someone in a large company the best practices for slower, more rigorous approaches. I have seen engineers from large companies jump into startups and ask for overly slow/rigorous approaches. Similarly, engineers from startups may move to large companies but see their applications hit a performance ceiling until they learn to go past quick, but less accurate, ways of doing evals. Project experience is valuable, but if you don't want to have to take years to gain experience in both small and large companies to master this breadth of skills, DeepLearning.AI is here to help!

Every week, I am in discussions about products that serve 100M+ users as well as products that do not yet have any users. The operating cadence is very different for these types of projects! This is why corporate policies that mandate a one-size-fits-all approach, like requiring certain types of testing before anything can be shipped, can be counterproductive.

Given that even large companies should have small, innovative projects, it's worthwhile for everyone to learn the fast, efficient tactics that let small teams move quickly. At the same time, to avoid hitting a ceiling and being unable to develop your projects beyond a certain point, it is also worth knowing how to do things in a slower, more rigorous way. 

Keep building, and I hope some of your early stage projects turn into large, successful mature ones!

Andrew