Experiences working with an Outsourced Dev Shop

Many years ago, a younger and more naive version of myself decided to try bootstrapping a startup together with my co-founder, while still juggling our full-time jobs. She had the idea, product vision, and contacts for initial users, while I was in charge of tech development. Our plan for the MVP was to launch using an iOS and android app.

My primary experience is in backend development, and I had never built a mobile app before. Rather than having me attempt to learn it from scratch, we decided to instead hire external developers to build the mobile apps, while I managed all the server-side development, P/SaaS integrations, and infrastructure.

Context

This wasn’t a completely new experience, which in retrospect, gave me undue confidence. For a previous bootstrapped startup, I had hired a young part-time freelancer in another country. He was personally recommended by someone I knew, did a great job, and charged less than $10/hour. 

I didn’t realize it at the time, but given how talented and conscientious he was, his rate was a steal. He is now making six figures in San Francisco.

Unfortunately, he was no longer available for a similar gig at this point. My co-founder also wanted a more reputed organization managing our front-end development. So we decided to look for dev shops, as opposed to individual freelancers.

We certainly did our due diligence. We looked at independent 3rd party reviews, asked for references, talked to their previous clients, and shopped around for multiple shops before finally settling on one that gave us the most confidence. They charged about $25/hour – significantly more than comparable freelancers. But we figured we were hiring an experienced and professional organization, not just a random individual, and that would be best for us.

My co-founder was a lawyer, and we certainly attempted to be as thorough as possible in our contract with them. Knowing that software projects are extremely prone to estimate overruns, we negotiated for a fixed-price contract, with a “warranty” for any and all bugs. After a lengthy period of nailing down the contract details, and detailing every single feature that they were supposed to build for us, it finally came time for us to make the first payment and begin the project.

And this is where we made our fatal mistake.

As per our contract agreement, the project was broken up into 3 segments. We were to pay 40% upfront, before any work was done at all. And then another 30% at the end of each of the first two segments… before we receive the deliverables from the just-completed segment. Given the fact that we were bootstrapping this startup, and making a 5-figure payment upfront, with no deliverables provided until after we’ve made the next payment… we were effectively locked in for the duration of the entire project.

We knew this going in, but figured it was fine. They had good reviews from independent sources, great customer references, and no red flags. Besides, we knew that it’s never easy to ramp up a new developer on a project built by someone – so we were planning to stick with them for the entire project anyway. 

In hindsight, that was the single worst technical decision we made, and dealt our startup a body blow.

Technical Challenges

One of the key features we wanted in our app was real-time chat. During our contract negotiations with them, they gave a couple of SaaS recommendations to simplify building the real-time chat functionality – one of which was Twilio Chat. After researching their various different recommendations, Twilio seemed like the best option, and we all agreed to use that for our chat functionality.

Unfortunately, once it came time to actually build it, they hit a wall. They couldn’t figure out how to make Twilio Chat work with React Native… even though they were the ones who had recommended using both Twilio Chat and React Native in the first place.

Worse, instead of coming clean and telling us that they were stuck… they simply told us that “Twilio Chat does not work on React Native.” And they now wanted us to switch over to an entirely different chat service provider (by a company that we had never heard of), start over again, and pay an additional fee for this mulligan (even though it was supposed to be a fixed-price project).

Worst of all, what they said wasn’t even true to begin with. Twilio Chat did work perfectly with React Native – they just didn’t know how to do it. Ultimately I, a developer with zero React Native experience, spent many hours digging for solutions, and taught them how to make this work. And even after I demonstrated this to them, they still needed me to spoon-feed them the links to documentations, and explain to them how to use the Twilio APIs.

If I hadn’t been around, or hadn’t figured out how to do their job for them, we would have gone ahead with their recommendation.  We would have ditched Twilio entirely and switched to a completely different and sub-par service. A decision that would have set the project back by many months and a big chunk of money.

Lax Security

I wish I could say that was the end of that Twilio debacle, but there was more to come. All twilio chat messages are part of a channel, and channels can be marked as either “private” or “public”. As implied by the name, private channels are private to the specific users in the channel, whereas public channels can be “seen and … joined by non-members. Additionally, the public channel, along with its members and messages, is visible to every client endpoint in a given service.”

For obvious reasons then, all non-public messaging should have been implemented using private channels. But astoundingly, they had implemented it all using public channels – something I was able to catch while browsing the Twilio console. If we had gone live with their implementation, anyone with even a modicum of development experience would have been able to eavesdrop on the private conversations of every single user in the app. And if I hadn’t spotted this myself, the development company most certainly wasn’t paying for any pen-testers to catch these sorts of security problems.

This was an outrageous mistake on their part. Even more shocking, instead of apologizing for their grievous oversight, they pushed back on this change. Apparently using public channels makes it easier to implement the chat functionality, and so, they preferred to keep it that way. It was only after a strong complaint from us, that they finally agreed to change the implementation.

Bugs Everywhere

One of the reasons we were excited to hire a development shop, as opposed to individual freelancers, was because of all the other support they promised us. Especially a QA team that would exhaustively test the app before showing it to us. 

It’s inevitable that any software project will run into bugs, so we knew they can’t make any promises. But we took them at their word, when they said that we should only expect to find a few corner-case bugs.

As we later found out, this was complete nonsense. Any and all deliverables we received from them, were absolutely filled with bugs. Even the most basic functionality was not working – I even suspected that they had never tested the app using actual phones… if they had even tested it at all. My co-founder and I had to spend multiple hours every day, for an entire week, painstakingly testing and documenting all the bugs that popped up everywhere.

Minimum Viable Programming

One example of a bug we found – if a user had more than 50 people she connected with, only the first 50 results were shown in the app. All the other connections could not be accessed at all. Turns out that one of our SaaS integrations was paginated, and the developers had only implemented code to fetch the first page of results.

Because this bug couldn’t be triggered until you had 51 connections on a single user, and we were still in private testing, it took us a while to run into the bug. Once we did, we reported it to them, and they promptly fixed it. We tested their fix, and it seemed to be working fine. 

When I reviewed their code change though, I realized how hacky their fix was. Instead of using a while loop to fetch all pages, they had simply added an if condition, to specifically fetch only the 2nd page. Once a user exceeded 100 connections, she would have run into this exact same bug once again.

I can excuse the first bug as an unintended error. But the second bug was willful negligence. They must have figured that it took us a very long time to trigger the 2nd page of results, and it would take us even longer to trigger the 3rd page. They knew exactly what they were doing, the limitations of their “fix”, and they did it anyway. And if someone hadn’t carefully examined their code, this bug would have leaked into production as well.

No Version History

As a developer myself, I knew first hand how immensely useful version-control history is. It helps future developers learn why certain design decisions were made, how specific functionalities were built, and a template for how to build other similar features.

For this reason, during contract negotiations, I specifically insisted that the final deliverable should be a git repository. They readily agreed to this, and said that they have generally used git internally as well. 

Unfortunately, when it came time for them to deliver the source code, they sent us a single zip file, containing a mix of all source code and generated files.

I reminded them that they had contractually agreed to give us a git repository. In fact, in the zip file they had sent, I even saw a “.git” directory – indicating that they had indeed been using git for their development.

The next day, they promptly sent us a git repository… containing a single commit… comprised of the exact same zip file they had sent us the previous day.

I contained my frustration, and told them that we wanted the entire version history, not just a single commit containing the same zip file. They replied that they have some “sensitive information” in their git repository, not intended for external audiences. Therefore they cannot share it with us. “The contract only says to deliver a git repo. It didn’t say that the repo should contain all development commits and history”

Moving Goalposts

During negotiations, we mentioned multiple times that the server-side APIs were not yet fully implemented, and that we want to do the back-end development concurrently with the front-end development. That at the start of the project, I would provide them with all the API endpoints, of which a few would be fully implemented. This way, they could start work immediately on the simpler features using those few endpoints. And by the time they finished those, the APIs would be ready to support the next batch of features. 

Our goal was to avoid delays and work on both concurrently, so as to launch sooner. This was something we stated upfront, clearly and repeatedly. And we were always told that it would be fine.

Unfortunately, as soon as we paid them the money, we started hearing something completely different. They flat out refused to start any work at all, until the back end development was 100% done and finalized, for every single feature in the entire project.

Luckily, because of how long we spent on contract negotiations and design work, I was almost done with the backend development by that point. So this didn’t end up being a problem. But it was both shocking and saddening that they weren’t honoring the promises their sales people had earlier made.

My Way or the Highway

When we were talking to them as potential clients, they rolled out the red carpet for us. But once we got down to brass tacks, they dug in and insisted on everything being done exactly their way.

For example, after doing some research on various options, I used swagger to document all of the API endpoints, their inputs, schema, descriptions and behavior. This way, the documentation would be embedded in the code itself, auto-generated, and kept up-to-date. The swagger GUI also provides a very pleasant way to browse all the API documentations, and even make API calls for testing purposes directly from their GUI.

Unfortunately, this wasn’t the way they did things. And therefore, they refused to accept swagger as a source of documentation. Instead, they insisted that we email them a word document, containing all the same content as found in swagger, but populated using their own specific formatting requirements.

We had to spend multiple days debating this, before they finally backed down. But the same attitude continued throughout the general development process. We had hired them in order to build mobile apps using our back-end APIs. But they were very demanding in wanting the APIs to work in very specific ways. Anytime we had a philosophical disagreement over API design, we had to endure multiple days of discussions and complaints.

It’s possible that this debate was driven by their passion for API best practices… but I suspect it was primarily driven by their wanting to make their own jobs as uninvolved as possible. And their difficulty in figuring out how to implement the required functionality using an existing API.

No Direct Communication

Another big surprise after the project started, was the lack of communication. In all my previous engineering projects, when collaborating across team lines, we would talk directly to their engineers to better understand and resolve issues that came up. To my surprise, this was something they explicitly banned. 

As per their policy, we would only have a single point-of-contact… with someone who was a non-technical project manager. Despite our requests, they refused to put us into contact with the developers who were actually working on the project. Furthermore, their project manager also refused to communicate in real-time over chat. They insisted on everything going via email.

Over time, this led to huge communication problems. Whenever their developers had a question, or couldn’t figure something out, they would send their question over to the project manager. She would then aggregate all the questions, and send me one big email at the end of the workday. And once I replied to their email, they wouldn’t see it until the start of their next workday. 

As a result, even a simple Q&A took 24 hours to round-trip. More complicated discussions took multiple days to resolve, instead of being discussed and settled in a 30-minute chat session. Thankfully, towards the later stages of the project, they finally realized how inefficient this process was and gave us some direct contact with their developer. Unfortunately by then, it was too little too late.

At the very start of the project, we knew that this would become a massive problem. They assured us that it wouldn’t be. But sure enough, our concerns became reality. Turns out it is very hard to be agile when you’re only allowed to communicate via one email per day.

Major Delays

Unfortunately, all of the above problems translated into real consequences for the project timeline. What was supposed to be a 2 month project, ended up taking 7 months instead. This was a major setback for us, as we missed out on many potential users who decided they couldn’t wait for our launch.

In retrospect these delays were not surprising at all, given their lack of technical expertise, insistence on a waterfall methodology, and refusal to communicate directly over chat or phone calls. But I suspect that this wasn’t the only issue either. 

I suspect that at various times, they had other projects that they considered more lucrative, and hence de-prioritized the work they were supposed to be doing for us. And that this was the reason they instituted a major personnel change in their developer team, midway through the project.

Scapegoats Galore

If there was one constant from all of their failures, it was their utter refusal to take responsibility for any of it. Before working on any task, they expressed 100% confidence in their ability to produce wonderful results. And when they failed to live up to their promises, there was always someone else to blame.

Can’t figure out how to use the twilio SDK?
“Twilio chat doesn’t work with React Native”
(It does)

Chat implementation would have publicly exposed all private conversations?
“The alternative is too complicated”
(That’s why we hired you)

Can’t figure out why a particular screen is taking 30 seconds to load? 
“We have to make 5 API calls and that slows it down”
(Those 5 API calls combined take less than 1 second to finish)

Project took 3x longer than promised?
“The server API was very bad and buggy”
(Yes, I’m sure it has nothing to do with your lack of prioritization, competency, and communication methods)

The frustrating thing about it all… is that this approach works! If I wasn’t a developer myself, I would have believed everything they told me. There’s a reason why they block direct communication with their developers – their project managers are experts at schmoozing, projecting confidence, and deflecting blame. God help the poor soul who hires them but doesn’t have the technical chops to call them out.

Lessons Learnt

When looking at all the things that went wrong above, it would be very tempting to simply say “offshore developers suck.” But such a conclusion is both small-minded and overly limiting. As someone who has worked with excellent engineers from other countries, it is pure folly to believe that good developers exist only in America. 

It is also tempting to declare that you should never outsource your development work. If you’re a mature company like Google, or a VC-funded startup, it makes sense to build everything in-house, using developers on six figure salaries. But for a bootstrapped startup with a small founding team, it certainly makes sense to use a few cheaper mercenaries to help get your MVP out the door. This is an approach that we were able to use successfully on other occasions.

It is also very tempting to see everything that went wrong above, and to guard against them by negotiating for specific contract clauses. Such an approach is doomed to fail. There are far too many unknowns, and too much subjectivity, to ever encapsulate everything into a legal document. Not to mention that legally enforcing contracts through a lawsuit, is a massive undertaking in itself.

Ultimately, there’s only one thing that matters when you hire a freelancer or dev shop. The ability to, and the threat of, walking away if they don’t do a good job. Almost every problem we ran into, stemmed from our lack of leverage. Because we had paid so much money upfront, we had no ability to walk away and hire someone else, even when things went very badly.

A Better Way

As soon as our contract with them came to an end, we cut ties with them and breathed an immense sigh of relief. I literally felt a huge burden fall off my shoulders. Going forward, we radically changed the way we worked with external developers:

1. Draw up a sequential list of features that we want to have built
2. Find a small handful of developers. Preferably independent freelancers, but a development shop would be fine too if they agree to the below process
3. For each developer, pick the top feature in the list, discuss with them the feature requirements, estimates, and cost
4. Have them implement that feature and test it
5. Have someone in-house review their pull-request, test the updated app, and flag anything problematic
6. Once satisfied, merge and deploy the feature, so that all founders/users can continuously review the app and give feedback or pivot as needed
7. If we’re happy with the job they are doing, pick the next feature we want them to work on, and repeat this process all over again
8. If we’re not satisfied with their work, cut them loose, and look for a replacement

It was fascinating to see how much smoother and more pleasant the development process was, once we got rid of huge upfront contracts, and replaced it with the above more incremental approach. Our developers were more pleasant to work with, showed more flexibility, communicated with us more reliably, and produced better work-output in less time. 

Best of all, not being handcuffed to a single shop for extended periods of time, derisked us and gave us immense peace of mind. If ever we weren’t happy with the way things were going, we knew we could walk away a week later. This ended up saving our butts on a few occasions, and took a huge load off our back. 

Conversely, I’m sure the developers appreciated this flexibility as well. Our continued collaboration was something we mutually agreed to every week, not something they felt forced into by a prior contract.

Is it possible to succeed with a “big waterfall project” if you avoid our mistake and hire the right dev shop? Sure, but how confident are you that you won’t run into the same problems. This entire debacle has given me renewed appreciation for Agile, and why it was developed. 

• Customer collaboration over contract negotiation
• Individuals and interactions over processes
• Working software over comprehensive documentation
• Responding to change over following a plan

Turns out many development shops refuse to work this way, and insist on using Waterfall paired with a large upfront contract. But on all future endeavors, this is something I have now learnt to insist on.

---

Related links:

Reddit discussion thread