I’ve been consulting at a startup for about 4 weeks now. Lets call it client 1. (Wait, wait, maybe I should call it client zero…..to reference 0 based counters and the fact that it is the origin of my consulting practice (as in patient zero)).
I’m helping them look for a CTO/VP Engineering as well as hire more developers and advise on how to improve the current development process. Its made me feel a bit like a glorified recruiter, but so far I cost less than the percentage they’d have to pay a recruiter, so it is a justified role.
In my search for a CTO I find myself asking a question that I think could be a real teller, if the candidate opens up and answers honestly….”What do you believe in? Which software development practices do you really fight for when push comes to shove?” In other words, if the CEO came over to your desk demanding that certain features be delivered in x days and you knew it would entail sacrifice, what would you be willing to sacrifice and what would you not!
Some options to toss out the window
- Unit Tests
- QA / Regression Testing
- Configuration (global vars!)
- Reusability (hard code it!)
- Flexible code/ Design for easy iterating and change
- Scalability (put it all in one class/method!)
- Security (do I really have to encrypt the users’ passwords?)
- Monitoring/Instrumentation (What? My MongoDB ran out of memory and has collapsed?)
- Metrics (analytics of user behavior)
Its a tough question I think. I believe that in a startup, you have to work with the business side of things to create a quality product in a timely matter. That means sometimes compromising your ideals for a delivery date while maintaining a high level of quality. The definition of quality is subjective given the list above. Some engineers would say that you cannot sacrifice any in that list (well, except for documentation, who really wants to write or read documentation?)
When I think about what may or may not be sacrificed, I think about site stability first. And I think about my past experiences. In my last position as CTO, I was using MongoDB for the first time. MongoDB loads your “working set” of data into memory to provide quick reads and writes. But once the size of your working set exceeds the amount of allocated memory, the performance can degrade quickly and without warning because of expensive disk I/Os. This is what happened to me. The size of the data did not increase suddenly or significantly, it simply crossed an important line in the sand one day. That line was the amount of allocated memory. Looking back through the Mongo logs, we were able to determine that the amount of wait time per query had been increasing steadily as the amount of data increased (and the size of the indexes increased). If we had the correct instrumentation in place, we would have been alerted soon enough to increase our DB memory.
You can bet, the next time I use MongoDB in a production environment, I will have the right instrumentation and alerts in place such as this. Something to dig in my heels and fight for.