Why bother writing functional/technical specs when developing software?

Maybe it’s just my personal experience, but I have only ever worked for one company that actively committed to writing functional and/or technical specs when developing software. Chances are that you’ve probably had the same experience in your career and perhaps you might even be anti documentation altogether. Personally I don’t think there’s anything wrong with coding without a spec if you’re working on your own pet project in your spare time, but on the other hand I’m a strong believer in writing a spec for formal projects. So before I begin making my case, let’s just quickly define what a formal project is vs just a hobby or pet project.

  • Formal project: A formal project differs from a pet project in the following ways.
    • Money: You’re getting paid to develop the software.
    • People: You’re not working by yourself, but in a team with other developers, managers, project owner and customer/s.

I’m going to use an analogy to make my point: developing software without a spec is like travelling from Cape Town to Cairo (10 134.8 km)  without a map. Instead of spending some time looking at the map and planning out your trip before leaving, many people are under the impression that the time spent planning is better spent travelling. This theory would work well if the road between Cape Town and Cairo is a straight line, but unfortunately it is not. Along the way there will be thousands of decisions that will need to be made. Without a plan you will take many wrong turns costing you a countless number of hours. If you had planned out your trip, you would know exactly which turn to take before you even arrive at an intersection.

  • Benefits of writing a spec: Here are the main reasons for why you should be writing a spec:
    • Managing budget and timelines: If you’re winging it without a plan you will never be able answer questions in terms of how far you are from the destination or how much more petrol money you’ll need. In the software world this translates to not being able to answer the customer when they ask how far you are and how much it will cost to deliver the software solution.
    • Avoiding scope creep: managing the scope of a project us really nothing more than managing the customer or project owner’s expectations. The spec serves as the scope definition and without a scope you will experience scope creep where the customer gets more than what they’ve paid for, while you and your team will suffer financial losses or at the very least negative cash flow. Aside from the financial implications, not managing the customers expectations and scope of a project, also results in chaos and stress, which obviously nobody enjoys.

Imagining people reading this, I can just hear the millions of excuses for not wanting to write a spec or any sort documentation. So below is a list of common excuses I’ve heard for not wanting to write a spec, and a counterargument for each.

  • “I don’t have anything against writing specs, but I see it as a luxury not a must-have. Unfortunately, I don’t have the time to write a spec because we’re running against a tight deadline.”
    • Counterargument: Continuing with our Cape Town to Cairo analogy, in the software world, the spec is your map which you can use as a reference on your trip to software excellence. For this reason, an hour spent planning your trip will spare you taking hundreds of wrong turns, and therefore of will save you days on your travels.
  • “Writing a spec is only useful when your developers are monkeys i.e. where you need to explain every little thing to them in detail. My developers and I are all very intelligent. Hence, we’ll just figure things out as we go along because we don’t need anyone else telling us what to do”.
    • Counter argument: This argument is essentially saying that we’ll just wing it. Continuing with the above analogy on travelling from Cape Town to Cairo, this false argument is equivalent to saying that you don’t see the point of planning your trip because Cairo is North of Cape Town and therefore you just need to head North and you’ll figure out the rest as you go along. Again the problem with this strategy (or lack thereof), is that it doesn’t matter how smart you are since you still don’t know what you don’t know i.e. even if you’re heading North, you won’t know what to do when you hit a T-junction. Instead you’ll rely on luck going either left or right and just hope that there will be another road somewhere that will get you back on track towards the North.
  • “A spec is just a piece of paper/document, it doesn’t actually do anything. Aside from using it as toilet paper, it has no other uses. Therefore it adds no value to the project.”
    • Counterargument: A spec is not just a piece of paper/document, it is a contract between you and the people you’re developing the software for. The objective of the contract is to manage expectations in terms of what the customer is paying for vs what is being delivered. You cannot meet expectations if you don’t know what those expectations are and failing to meet expectations leads to failed projects, negative cash flows and perhaps even your company closing down.
  • “I’m a creative type and therefore don’t like being constrained by a spec that forces me to develop the software in a specific way. Instead I like to think out of the box and prefer having the freedom to come up with more innovative ways of doing things on the fly.”
    • Counterargument: this one really comes down to your level of technical skills and knowledge i.e. whether you’re a junior or senior developer.
      • Junior Developer: If you’re a junior developer then chances are that you not quite read to make design level decisions on software that falls in line with the customer’s requirements. In such cases it would be better for a senior developer/analyst to write up the spec and for you to follow it religiously. If you feel that you can contribute with alternative solutions, you should make the suggestions to the senior that wrote the spec. At this point, the merits of your alternative approach should be discussed in an open minded fashion where you are taken seriously. If you can prove based on facts that your solution is a better one, then your solution should in fact be implemented. Otherwise the senior developer should provide you with a solid fact based explanation as to why your solution would not work in the given context.
      • Senior Developer: If you’re a senior developer and feel that you’re constrained by the spec given to you, then you should bring your suggestions to the analyst/developer that wrote the spec. Better yet, if you feel that you are capable of it, then you should probably be the one writing the spec. Reason being that having a single person designing and developing a system is a lot more productive that splitting the tasks between multiple people. This avoids having too many cooks in the kitchen. Either way, being the creative and agile type is not an excuse for not designing the software and thereby writing a spec before coding.
  • “The person writing the spec is only a systems/business analyst with limited technical knowledge of how software is actually developed. That means that their spec will be inaccurate and the software that I develop won’t match the spec anyway. So what’s the point of having a spec written anyway?”
    • Counterargument: This is a very valid point and I have in fact seen this scenario play out several times throughout my career. The solution to this problem can be solved in one of several ways:
      • Teach the analyst: Spend some time explaining what can and cannot be done and just generally bringing them up to speed on the process of software development. This simply comes down to managing expectations, not only between you and the analyst but between the analyst and the customer/end user. Educating the analyst is key for managing this scenario. Of course there are many cases where certain types of people are not open to learning or are simply too arrogant to accept help or advice.
      • Find another analyst: If you have no luck in teaching the analyst, you probably have no voice but to find another more competent analyst. Due to politics and also not wanting to hurt people’s feelings, this could end up being easier said than done. However, provided that you have tried everything in your power to remedy the situation, you will be left with no other alternative but to find a replacement.
      • Do it yourself: In many cases, simply replacing the analyst is not always possible, such as in cases where they are your boss or perhaps due to politics. In such cases, it’s best that you take over the design work by writing the spec yourself. Don’t bother asking for permission, just get it done as it’s easier to ask for forgiveness than to ask for permission. If you  delivery the solution successfully, you probably won’t have to ask for forgiveness anyway. The bottom line here is that whichever way you end up doing it, you still don’t have an excuse for not working with a spec.
  • “Documentation is for business people. I’m a technical person and I just don’t enjoy reading or writing documentation. For me the code is the documentation. If anybody wants to understand what the software does and how it does it, they just need to read through my code.”
    • Counterargument: It’s all good and great to say that you understand what the software does by reading the code, but what about the business people that cannot read code, how will they understand what the software does or is supposed to do? Furthermore, think about how long it will take another developer to reverse engineer the business logic out of the code. The point being that you have to think about more than just yourself, and instead keep the whole team in mind. This is a prime example as to why people still pay for software instead of using open-source because most open-source project lack documentation i.e. it’s all good and great to get free open-source software, but it turns out to be pretty useless if you don’t know what it does, how to use it and/or troubleshoot and fix it when necessary.
  • “It’s better to write unit tests for defining and validating requirements. Writing unit tests serves the same purpose as writing a spec by forcing you to think abut the requirements. Furthermore, it has the additional benefit of enabling you to also test your code, which is something that a spec cannot do.”
    • Counterargument: This is a comparison between apples and oranges. Unit tests and documentation are two separate things which serve a different purpose. This is the exact same false argument as the one above which implies that unit tests, which are in fact code, serve as documentation. Once again the rebuke on this argument is that code is not documentation since it’s incredibly difficult to reverse engineer business rules out of code.
  • Nobody ever reads specs anyway.
    • Counterargument: This is a people problem not a technical problem and therefore the people need to be dealt with. The easiest way to deal with people that don’t read the spec is to simply reply to their questions by saying “Check the spec, on page X” or even just “RTFM”. Sooner or later those people will take the hint. The moral of the story is that you need to teach people how to treat you which is to respect the time and effort you’ve put into writing the spec.
  • “Even if specs get written, nobody ever updates them, which renders them useless in the long run.”
    • Counterargument: Firstly, who’s fault is it that the spec doesn’t get updated? This argument is equivalent to saying that you won’t take a shower every day because you’re just going to be dirty the next day anyway. That is obviously nonsense and once again this is a people problem not a technical problem. The simplest solution is to become disciplined by getting into the habit of updating documentation on a regular basic.
  • “Working with a spec would make us too rigid, and requirements change anyway. Hence we prefer using XP and/or Scrum to manage requirements which allows us to be more agile.”
    • Counterargument: Yes, it’s true that requirements change, but regardless of whether they change or not you will still need to define the the scope of the project. Changes can still be handled in an agile way by using Change Requests which can be filed when new requirements are introduced. What’s important to remember, regardless of what methodology you chose to use, is that no customer or company that you work for will ever give you a blank cheque to spend countless number of hours developing software with no end in sight. Therefore wanting to be agile is no excuse for a lack of documentation and design.
  • “The software we’re developing is really large, will take a really long time to develop and as such it is not possible for us to design and document every little feature months or years ahead of time, due to the fact that things will change in the future.”
    • Counterargument: again is the same argument as the one above in terms of looking for flexibility and agility. Indeed it is a valid argument that one cannot predict what will happen months or years in advance. However, you should be able to predict what will and should happen a month or two in advance. I’m not saying that you should write a ten thousand page spec for the development of a two year long project/product. There’s nothing stopping you from breaking down the software into features that should be delivered in the next month or two and then further breaking it down into weekly XP sprints. However, you should write a spec for each software component/feature that you plan on delivering. In other words instead of writing a spec for a complete product that will take two years to be developed, you should write a spec for the first few features that will take two months to be developed. Thereafter the documentation can be updated with the next set of features that are planned for the next two months, thus enabling the documentation to grow and adapt organically as the code does. The point being that the documentation can adapt to changes just as the code does, but the spec and design for each feature should still be completed before any code is written.

In summary, I’m going to throw the age old proverb at you by saying that “if you fail to plan, you plan to fail“. When all is said and done, it really is that simple and there really are no excuses for not writing a spec when working on a formal project or product.

What to look for when hiring a software developer

Like most people that have been in the industry for a number of years, I’ve been on both ends of the hiring process; both interviewing candidates and being interviewed. Conducting an interview and deciding which candidates to hire is incredibly difficult because there are a multitude of red and green flags to look out for. Hiring a software developer is even more difficult mainly because so much of the value that a developer provides is intangible i.e. their talent and personality are difficult to measure.

There have been cases where I’ve sat on interview panels, where we all thought the candidate was fantastic, but proved to be a nightmare to work with once hired. Obviously a similar situation arises when choosing which company to work for, but that’s a subject for another chapter. Here I’ll cover a few mistakes that I’ve noticed being made during interviews, as well as a few things you should be looking for when hiring.

  • Mistakes in interview process: here are a few mistakes I’ve seen companies make in the interview process.
    • Technical assessment before culture fit/personality interview: as far as I’m concerned, testing someone’s technical abilities before getting to know them is like having sex with someone before even talking to them. Some people might be fine with that but I believe that it might set off a red flag to some candidates and as a result you might lose a few talented ones that could have otherwise been great for the job. As a candidate, this approach would tell me that the company doesn’t care whether they’re hiring Mother Teresa or Adolf Hitler as long as they can get the job done. Chances are that a company like that will already have a few Adolf Hitlers roaming around, and that’s not an environment I would personally work in. Some companies might defend their position by saying that they will still have the culture fit interview after the technical assessment, and they’re choosing this approach in order prevent wasting time on people that are not technically competent. That’s fair enough, but at the same time it clearly exhibits the company’s priorities and as a result some candidates will feel that in a company like that they will be treated as disposable “resources”.
    • Theory based technical assessments: I’ve seen software developers (and people in general) that are exceptional at learning theory but are useless at putting it into practice. People that are great at memorising large amounts of theory might excel in some professions like law and maybe even medicine, but software development is not one of those professions. For example, I could take a programming book on C# for instance and give it someone asking them to read through the entire book, perform all the exercises and learn the C# syntax. I could thereafter give the same person a test asking them to write an if statement, while loop, create a class, name all the different accessibility modifiers, tell me the difference between an interface and an abstract class etc. Having read and memorised the book, the person will pass the test with flying colours even though they’ve never developed a complete app in their entire life. When hiring a software developer you shouldn’t be looking for someone that can talk about software, but rather  someone that can actually develop software. Moreover, software development is more than just hacking a few hundred lines of code together, instead it’s about developing complete applications that could require thousands of lines of code.  Hiring someone simply because they have the knowledge means nothing if they can’t actually put it into practice.
    • Skills based technical assessments: the company may be missing out on some really talented candidates if they’re filtering out candidates simply based on a specific skillset. For example, you could have a candidate that’s a genius but they’ve never worked with a specific technology used in the technical assessment. As a result the candidate will be filtered out without being given the opportunity to impress you with their other abilities. For example you could have a software developer that is brilliant at C++ but the interview is for a Java development position. Obviously hiring this candidate will involve a learning curve, but if they’re talented they will learn the Java development stack and become productive relatively quickly. On the other hand if you have a candidate that is an average or poor developer and only knows Java, he/she may be hit the ground running almost immediately but will never learn anything new and the Java code he/she writes will be terrible.
  • What to look for in a personality interview: knowledge and skills in a specific domain are obviously important and useful for a candidate to hit the ground running. I would definitely look for skills when hiring a contractor for a certain timeframe on a given project. However, when looking to hire someone permanently you are effectively making an investment in this person and expecting to get a return on that investment. If you’re only looking for a specific a skillset the person may deliver on a given project but may not necessarily deliver on the next project if a different skillset is required. For this reason I find it futile to hire people for specific skills and knowledge. More importantly knowledge and skills can be learnt by pointing them in the right direction, giving them a book to read, sending them for training, or even sitting down and teaching them personally. However, there are a few things that you cannot teach or change about a person and so these are some of the attributes I like to look for before I even think about doing a technical assessment:
    • Personality: when given the choice between working with a highly talented asshole versus working with a nice guy that needs a bit of training, I would chose the nice guy over the asshole every time. The reason being that you can teach skills but you can’t change an asshole. It’s that simple.
      • Test: This is obviously something that is very difficult to test, but through experience in dealing with people you can get a pretty good idea of the kind of person you’re dealing with by simply having a casual chat and asking about specific cases in which they dealt with conflicts in the workplace and how they handled it. If budget allows, you could even ask them to complete an aptitude test with a qualified psychologist.
    • Analytical: a person can have all the knowledge in the world but if they are not able to logically analyse a problem, troubleshoot and come up with solutions, they will never make it as a software developer. Being analytical is something you’re born with; you either have it or you don’t.
      • Test: The first thing I look at are their exam results from school and/or university, specifically on subjects that required logic reasoning. I wouldn’t care too much about their test results on theory based subjects such as Geography, History or even certain IT related subjects like Databases, Operating Systems, Networks etc. Instead I would specifically look at their results for Maths, Physics, Programming and other subjects that required solving equations like Electrical Engineering, Electronics, Digital Systems etc. If I had to chose between a candidate that had below average school results with two years experience in C# versus a guy that has never written code in his life, but was an A student in Electrical Engineering, it would be a no brainer i.e. I would hire the Electrical Engineer to work as a Software Developer. The reason being that Electrical Engineering is an incredibly tough course and completing it with As tells me beyond a shadow of a doubt that the he/she could learn to code relatively quickly and will deliver excellent results.
    • Inquisitive: in the world of software and technology things move very fast, with new technologies popping up every day. It’s obviously impossible for a software developer to know everything, but what you should be looking for is a candidate that is inquisitive and enjoys learning new things all the time i.e. as opposed to someone that sees learning as a burden/work.
      • Test: if you do a bit of research you will find that there are various ways of assessing whether someone is curious or not. The easiest way is to ask them at the end of the interview whether they have any questions for you. People that are inquisitive will ask a lot of questions, while others might only ask one or two questions just to show interest.
    • Quality of work: there are certain jobs that require people to complete a set of tasks to be productive. Some examples include a warehouse worker who moves boxes around the warehouse, a driver who has to do deliveries, a cashier in a store, an accountant who needs to process salaries, or even a businessman/salesman who needs to make X number of calls per day, close deals and reach a target by the end of the quarter. What all of these jobs have in common is that it doesn’t matter how the job gets done as long as it gets done. Software development and engineering in general is not one of these jobs. As an engineer you are not just performing a list of tasks, but creating products. The products created will be used by endusers who expect and are willing to pay for a certain level of quality. Therefore how you do the job and the quality of your work is not just important but critical. Obviously there are times when engineers work under financial constraints which limit the time required to achieve excellence. Nevertheless, you should look for candidates have some of the following attributes, which I believe makes them produce quality work:
      • Pride: nothing irritates me more than an engineer that has an “it’ll do” or “it’s good enough” attitude. When you’re creating products “it’ll do” is just not good enough. There is no substitute for people that put pride in their work and that’s not something you can teach i.e. you’re either born with it or you’re not.
        • Test:
          • Ownership: a simple telltale sign that someone puts pride in their work is to ask the candidate how they feel about code sharing i.e. someone taking over their code and/or modifying it. People who put pride in their work take ownership and accountability for their work, and generally have an emotional attachment to what they produce i.e. it’s like their baby. Having someone come in and take over and/or “damage” their “work of art” infuriates people that take pride in their work. People that do not take pride in their work are quite happy to let someone else deal with their mess. I know that many managers actively encourage code sharing in their teams mainly because they see the developers as just “resources” which they allocate to projects and codebases. I think that’s a terrible mistake on the manager’s part because it discourages ownership, accountability and above all reduces the quality of the code.
          • Neat & tidy: I’ve seen software developers in interviews that are incredibly bright, knowledgeable, confident, well spoken etc. But when you look at their code it looks worse than a dog’s breakfast. They simply don’t care how the code looks, they only care whether it works or not. If these people leave their job and someone else needs to takeover their codebase it will be a nightmare for the next developer who needs to deal with the mess. The only way to test for this attribute is to look through some of their existing code.
        • Eye for detail: a software developer that doesn’t have an eye for detail is like a blind sniper i.e. they will create a ton of bugs and produce shockingly bad quality software. You cannot teach someone to have an eye for detail so don’t bother hiring such a candidate into a developer position. You could instead put them in a support role.
          • Test:
            • Present a vague problem: the easiest way to test for this attribute is to verbally present the candidate with a feature request or a problem to be solved. They need to provide you with a solution of how they would solve the problem. Your description of the feature/problem needs to be quite vague with limited context. Ensure that you provide at least some sort of context. A candidate that has an “eye for detail” or detailed oriented will begin by drilling down into the problem asking you a million questions for clarification. You should obviously be prepared to provide those details. Pay careful attention to the level of detail they provide when proposing the solution. Give them the opportunity to write down the proposal since most people cannot present a long but structured solution verbally. A candidate that doesn’t ask for a lot of details will not able to provide details in their proposal, and will obviously only reply with vague solutions to your problem.
            • Email exchange: if working through a recruitment agent, you may not always have the opportunity to exchange emails with candidates. If however you do have a few email exchanges, pay careful attention to the length of their emails and the level of detail they provide in their replies. In my experience, people that write long winded but coherent emails are generally very detail oriented. Obviously it depends on the subject of your emails, therefore try and ask some detailed questions, which would required detailed responses. People that consistently reply with a single sentence to your emails are not detail oriented and probably never will be.
  • Technical assessment: once the candidate has passed a personality interview and you’re happy to take it further, you can then conduct a technical assessment. As mentioned previously I’m not a fan of theory and/or specific skills based interviews. Here are some of the best approaches I’ve seen in technical assessments.
    • Develop an app: a great option is to give the candidate a set of requirements to develop a full application. It doesn’t need to be anything complicated or incredibly time consuming, but it should require them to spend a few or more hours writing enough code that you can see their coding habits. The requirements must be such that it will require writing more than just a single function or algorithm. Also do not force them into writing in a specific programming language or technology stack, since you want to them to use whatever they feel most comfortable with, which will enable them to showcase their skills as opposed to have to learn a new language just to complete the assessment. The requirements for the app should also not be too constraining since you’ll want to give the candidate the opportunity to demonstrate their creativity. Since this exercise might take more than an hour or two, you can let the person develop the app in their own time and submit it when they’re done, which could be even a week later. To ensure that they did in fact write the code themselves without help from others, you could ask them to come back for code review.
    • Code review: the easiest way to ensure someone can actually code is to look at their code. One option is ask them to develop an application as mentioned above and then do a code review on that. However that can prove to be very time consuming for both you and the candidate, especially if you’re interviewing multiple candidates at the same time. An easier method could be to ask the candidate to take you through one of their existing code bases. Some candidates will say that their code is owned by their current or previous employers and cannot be shared with you. If that’s the case, don’t bother hiring the person because any developer worth his salt will have their own code that they work on in their spare time. Once they have agreed to show you their code, you could of course just look through it by yourself, but to ensure that it was written by the candidate and not someone else, it would be wise to have a code review either face-to-face or via a conference call. Ask them as many questions as you can, paying careful attention to how they’ve structured their code, whether it’s object oriented or procedural, the design patterns used (if any), whether it’s neat and tidy, easy or difficult to read, and ask them specifically about which parts of the code they struggled with the most.
    • Pair Programming: there’s a quote attributed to Plato where he said that “You can discover more about a person in an hour of play than a year of conversation.” I wholeheartedly believe in this. The quickest way to get to know a person is by working with them as opposed to just talking to them. The surest way of seeing the how someone works is to work with them even for just an hour or two. Give them a problem to solve or a feature to develop and then sit with them working through the problem together. You will immediately see their personality coming to light as well as their many other attributes such as troubleshooting and communication skills, work ethic, perseverance and ability to come up with new ideas. In the process, the candidate will also have the opportunity to learn something about you and what it would be like to have you as a colleague and/or manager. In the end if you extend the job offer and they accept, you’ll know that the decision was based truth and not merely on perceptions you have of one another.

In summary, you should always aim to hire software developers for talent and train for skills as well as paying careful attention to personalities.

Do project managers actually add value to software projects?

Throughout my career I’ve worked with various types of PMs (Project Managers) and I too have managed many projects over the years. I’ve often pondered about whether PMs do indeed add value to projects or not. The simple answer is that I think PMs can indeed indeed add value to a project, but in my experience 90% of the time they just simply don’t. If anything they probably make the project go slower and bring about unnecessary complexity. Having said that, I don’t believe it’s any fault of their own. Personally I think this is caused by upper management assigning the project managers for the wrong reasons or there are other corporate politics in play that prevent PMs from adding value to projects.

So let’s start with a list of very wrong but very common reasons why a PM gets assigned to a project and how they end up adding little to no value:

  • Promoting the weakest link: Suppose there’s a guy on a software team called John and suppose that John has limited technical skills. This could be because he doesn’t have any talent for coding, or he’s lazy, or he isn’t analytical enough, or he writes horrible code, or maybe he simply doesn’t care enough about the company or project. When it comes to employees like John, upper management will have tried to place him in various roles in the hope that there’s something that he’s good. Unfortunately, John cannot code to save his life, and when given the opportunity to code, he creates more problems than solutions. They’ve tried putting John into an analyst role to gather and document requirements, but because of his limited technical skills he cannot translate functional requirements into technical requirements. Worse yet he doesn’t have any eye for detail, and misses a lot of important information when relaying information between the customer and the development team, which in turn can cause havoc on a project. They’ve tried putting him in a support role, but he ends up escalating almost all the issues to the developers without ever learning anything in the process. As ugly as it may sound the bottom line is that John is pretty much useless. So, what do you do with a guy like that when you’ve tried everything to increase his technical skills and the only thing that he seems to be half decent at is talking to people. Well, what often happens is that upper management will assign him as a PM on the next project. Their reasoning is that in their minds, project management is only about managing the customer, so John will definitely not be able to screw it up. Even if John does screw it up, project management can cause the least amount of collateral damage i.e. the developers will continue with their development and the project can still be a success.
  • Maximising billable hours: In software consulting companies it’s all about resources (people) and time. The more hours the company bills the more money it makes. Suppose you have two developers on a project and they communicate with the customers directly: any time spent attending meetings, phone calls and emails is often ignored i.e. it’s a loss to the company. Even if that time is actually billed for, you’ll only have those two people billing time. Therefore what many software companies do is to add as many people on the project as possible i.e. instead of having only having two resources on a project let’s add a PM into the mix so that he too can bill hours for attending meetings and drawing Gantt charts. This is the oldest trick in the book by any company that makes money through selling services. Lawyers often apply the same tactics: instead of getting one lawyer that can do everything, they will tell you that one lawyer specialises in one side of the law while another will specialises in something else, and you’ll need another to take notes, another to do research, and another to supervise the whole circus. In the software industry, it often happens that a PM gets assigned purely to increase the amount of billable hours instead of to actually manage the project.
  • Specialised project managers: many software companies hire PMs that have some sort of formal project management course under their belt and they assign these people purely into PM roles without any technical responsibility i.e. they’ll never do any analysis, design, development, or support work etc. The problem with this approach is that these PMs have no technical skills or engineering experience whatsoever. So if they ask developers for time estimates, they will take whatever the developer tells them because they don’t have any hands on experience to tell right from wrong. Worse yet, they might not even ask for estimates from the developers, and instead simply thumb suck the schedules and timelines. When deadlines are not met being met, the PM passes the buck by blaming the developers. Furthermore, when the developers try to explain a technical issue they’re struggling with, the PM cannot articulate those issues to the customer.
  • Glorified secretaries as project managers: many software companies are run by non-technical business people. For this reason, these business people often do not understand the work that the developers do, nor can they measure their developers’ progress. They’re often also not interested in the technical details regarding issues the developers are facing. What the business people do understand though are numbers. So what they often do is assign a PM for the simple purpose of capturing, keeping track and reporting back on costs and timelines throughout the project. At the same time, the PM is given limited authority. So when it comes to the technical staff not performing, missing deadlines and dealing with unhappy customers, this PM has virtually no power to do anything about it i.e. they might have the word “manager” in their job title but none of the technical staff report to them. In cases like this the PM is in fact nothing more than a glorified secretary that schedules weekly meetings to ask for progress updates, accepting anything the developers say without questioning anything. Thereafter the PM will update the Gantt chart and spreadsheets and forward them to the real manager who is not even involved in the project. This is a recipe for disaster because these kind of PMs end up being nothing more than shock absorbers for the developers thereby fostering  laziness i.e. the developers never have to face a screaming customer because their glorified secretary of a PM will take the hit, while at the same time the PM is not able reprimand them for non-performance.

For all the above reasons I believe that PMs add little to no value in most cases. However, I do believe that PMs can indeed add an incredible amount value under different circumstances.

Here are a few very good reasons for assigning a PM on a project and some strategies for maximising the value that they can add:

  • Reduce communication channels: the most important reason for assigning a PM to a project is reduce the number of communication channels between stakeholders. This really comes down to Fred Brooks’ famous quote, “What one programmer can do in one month, two programmers can do in two months.” The simple logic behind this statement is that the more people you add on a project the more these people spend talking instead of working. If you only have a single person on a project, then this person has no one else to talk to and therefore spends all their time working. If you have two people, they will spend a considerable amount of time talking to each other and consequently establish a single communication channel amongst themselves. Having three people means you have three communication channels between them. Having four people results in six communication channels. Hence, increasing the amount of people on a project increases the communication channels exponentially, which in turn reduces productivity. This is rule is known as Combinatorial Explosion and the formula for determining the number of communication channels is as follows: l = n(n -1) / 2 where n is the number of people on the project and l is the number of communication channels. Using the formula we can calculate the number of communication channels on a team of 10 people to be 45 i.e. 10(10 -1) / 2 = (10 x 9) / 2. As you might expect, in such a team, the 10 people will spend all their time talking instead of working. This being the reason why so many corporate environments have endless meetings without ever getting anything done. Smaller companies on the other hand tend to be a lot quicker and more productive. It’s in cases like these that PMs can be added to reduce the communication channels. By adding a PM in the centre of a 10 people team it will reduce the communication channels from 45 to 10 i.e. instead of each team member talking to every other team member, the communication becomes centralised by them talking only to a single PM.
  • Assign a PM with strong technical skills: adding a PM to reduce the communication channels is all good and great, but it will prove to be futile if the PM has insufficient technical skills to hold a discussion with the developers. For this reason it’s important to ensure that a PM has a strong technical background as well as people skills to communicate not only with the developers but also to manage the customers and/or business people. The result of having a weak project manager is that the developers will simply resort to sidestepping the PM by talking directly to the customer as well as manage the project amongst themselves. Evidently this defeats the object of assigning a PM in the first place.
  • Increase the PM’s authority: If a PM is going be assigned to a project, the PM then needs to be given absolute authority over the project. Even if some team members do not directly report to the PM within the company, within the context of the project, they should be reporting to the PM. Higher level management should explicitly make it clear to the team that they report to the PM and failure to do so comes with consequences.
  • Do one project at a time: Even if absolute authority is given to the PM on a given project, the developers for example may be working on multiple projects at a time reporting to multiple PMs simultaneously. The problem with this scenario is that the developers can use one project as an excuse for non-performance on another project. For example, a developer could say to PM 1 that they haven’t made any progress on project A because they’ve been busy on project B. They could then turn around and tell PM 2 that they haven’t made any progress on project B because they’ve been busy on project A. Meanwhile back at ranch, the developer hasn’t made any progress on either of the two projects, but the two PMs will be none the wiser. Amongst many other pitfalls of running concurrent projects is that it undermines the PMs’ authority thereby enabling the developers to run amok.
  • Global authority: In many companies it’s simply not possible to run a single project at a time. This could be for financial reasons where customers cannot be held off for several months in waiting for resource availability. In such cases, it’s best to assign a single PM to manage all concurrent projects, thereby granting the PM complete authority across all projects. The goal being that a single developer should report to more than one project manager at any given time. By doing so the developers will never be able to deceive the PM by using one project as an excuse for non-performance on another project.

Managing Costs of a Software Project

Developing software is an extremely time consuming process, requiring long hours of uninterrupted focus on the technical task at hand. That coupled with constantly learning new technologies, analysing requirements, designing, documenting,  coding, debugging, testing, installing etc. makes software development a stressful and difficult job. But no matter how difficult the job, for certain types of people the process of imagining something and bringing it to life from nothing proves to be work that is incredibly rewarding and self satisfying. Most successful software developers that I’ve met over the years love coding and most cannot imagine doing anything else with their lives.

Before I started working as a software developer professionally, I  remember spending hours sitting at my computer for hours on end writing code just for fun, without any interruptions or any other worries. When I was done my code had no bugs whatsoever and by that measure the code was perfect. If I wanted to implement another feature, I would just start coding it right away. The only limits were my own imagination. However, after I started working as a software consultant for a few years I noticed a few things that would kill the joy of coding. It’s for these reasons that many developers end up changing careers or at the very best keep switching between jobs in the hope of finding a light at the end of tunnel i.e. bringing joy back into coding.

The truth is that if you want to enjoy your career in the real world, you cannot continue sticking your head in the sand and pretend that you’re just a coding geek, and ignore the business aspects. You may not enjoy getting involved in the business aspects of software, but as the saying goes, “if you can’t beat them, join them” and believe me when I tell you that you definitely cannot beat the business people for the simply reason that they are pragmatic realists while most coding geeks are idealists with their heads in the cloud … or sand or whatever else geeky developers put their heads in. As George Carlin once said “Inside every cynical person is a disappointed idealist.” Continuing on a path of avoiding business meetings will turn you into a disappointed and grumpy pessimist, ending up complaining to your grandkids over Christmas dinner about how unfair the world is. So you may as well join the business people and turn yourself into a pragmatic realist that somehow always seems to find a solution where others see only problems.

Here’s some advise for getting involved in the business side of things. It all starts with managing ongoing costs of projects. Firstly, every software project has three components that need to be managed and there isn’t a single business meeting in which these components are not brought up:

  • Money: in the real world, people need to put food on the table, bills need to be paid and companies need to make a profit otherwise they go out of business. For this reason every little thing needs to have a monetary value assigned to it. So if you have an idea for a feature you’d like to develop, you cannot just start coding it because someone needs to pay for your time and effort. Your customer/boss will only pay for you to do it if it provides an ROI (Return On Investment) to the company i.e. a financial benefit the company gets from having this feature/application developed. Therefore before you start coding, you first have to create a proposal, estimate costs, prepare an ROI analysis, present it to the customer and wait for approval. Depending on the costs, complexity of the project and other factors this process could take a week, a month, a year, or it might never get approved (probably because your proposal sucked).
  • Time: regardless of whether you’re billing by the hour or you’re getting paid a flat salary, your time is being paid for. Therefore any feature or app you are developing must have a time constraint assigned to it because time equals money. To make matters worse, there are deadlines imposed by the customer because they need it by a certain date. If that’s not bad enough, you’re competing with other software companies, and so in order to win a contract, you’ll then have your salesman promising the customer (on your behalf) that you will in fact delivery the finished software in half the time for a fraction of the money that other companies are offering it for. While you’re still standing there dealing with the shock and horror of the situation, the salesman will then turn around and spew out motivational quotes to you which he heard at some sales seminar that he went to last week … or maybe he’ll just quote Barack Obama by saying “Yes, we can!”.
  • People: the more people are involved in a project the more politics you can expect. That’s just a fact of life. The reason being that different types of people have different agendas with different incentives in their job. So once again, before you get to write any code, people need to agree on what this code needs to do and how it needs to do it, how long it’s going to take, what budget is going to be allocated to the project and who is going to handle which work. Going forward you can expect millions of emails, phone calls, conference calls and endless recurring meetings that will make want to pull your hair out … assuming you have any hair left to pull out. Throughout this process both functional and technical specifications need to be written up which essentially serve as contracts for what will be delivered as the finished software to the customer.

You would think that people would be aware of all the above, but most people, especially inexperienced or non-technical stakeholders, somehow still believe that software projects are purely about just writing code. They either fail to foresee all the above mentioned factors or they simply choose to stick their heads in the sand and not learn from past mistakes. So they typically end up quoting purely for the time required to write the code, thereby setting up unrealistic expectations with the customers. On the customers’ end, it goes without saying that most customers are not software people and therefore never expect to pay for anything other than the time required to write code.

When these naive people inevitably run into these problems they throw their toys out the cot, arguments ensue which lead to meltdowns and they just generally become stressed out and unhappy with life. Many of the software companies then end up going out of business because of failed projects or loss of money due to time that was never billed for.

In order to to survive in this industry and avoid projects going pear shaped, you have to start managing expectations. Expectations translate into managing money, time and people and it all starts with the first quote to the customer. A million questions will arise when doing the first quote on a project. Here are some examples:

  • “Should we quote for emails, phone calls, meetings and other project management related work?”
  • “What if the customer refuses to pay for anything other than purely coding time?”
  • “Should we then just act like a charity and give that work away for free?”
  • “What percentage of the project do we spend doing project management versus time spent developing software?”
  • “What if unexpected things go wrong and the costs go up?”
  • “What if the customer keeps changing their requirements resulting in scope creep?”
  • “Should we then quote for requirements analysis and documentation work?”
  • “Will our competitors be cheaper than us because they’re not quoting for all the above?”
  • “What if our competitors undercut us by deliberately underestimating the time and effort required and oversimplifying the customers’ requirements?” This happens on a regular basis by the way. Many sales people do this because they know that once they get started on a project the customer will not be able to simply turn back if the project ends up costing more than what was originally quoted for. This approach is like the drug dealer sales approach i.e. the first one’s free, after that we’ll nail you.
  • “If that happens, will the customer be experienced enough to realise that they’re being played by our competitors and that in the long term it will eventually cost them more than they bargained for?”
  • “Should we then preempt the competitors’ tactics and deliberately under quote just so we can get the deal i.e. should we stoop to our competitors’ level?”

These are all difficult and valid questions to ask yourself before quoting on a project. What I can tell you though, beyond a shadow of a doubt, is that if you’re only charging for purely coding time, your company will inevitable go out of business. The reason being that in my experience at least 50% of the work required on most projects excludes coding time.

So what is the best way to quote and bill then?

  • Wrong sales approach: Under quote and over bill i.e. the drug dealer sales approach. As mentioned above, most inexperienced sales people will under quote and over bill in this manner and most inexperienced customers will accept these quotes, which unfortunately perpetuates these bad practices i.e. people learning from bad examples. This approach is equivalent to over promising and under delivering in terms of costing. There are all sorts of other problems which this approach can create:
    • By quoting for purely coding work and/or cutting the budget, you are cutting the time on the project, thereby putting unnecessary pressure on the developers who will end up taking shortcuts in their code, resulting in massive amounts of technical dept being accrued. In the long run, the project will end up costing way more anyway. Hence I say that you would be putting yourself and/or other developers under unnecessary pressure.
    • It goes without saying that taking shortcuts leads to lower quality code being produced and the software being filled with bugs.
    • Due to the low quality code and over-budget costs, your reputation will end up being tarnished in process. So even if you end up finishing the project successfully, that very same customer will most likely never call you again for the next project.
    • In the end it really comes down to the customer getting what they pay for. That being the reason for many internal and enterprise products only being as good as they need to be i.e. sales people undercutting competitors, underestimating requirements and customers being cheap and shortsighted.
  • Right sales approach: Over quote and under bill: it’s always difficult to estimate the costs and timelines of a project. For this reason and all the reasons mentioned above, I have found the best strategy to be as follows:
    • Sales meetings: During your first sales meeting/s with the customer, get a sense of the their requirements. You don’t need to get into too many details about every custom single business rule and how exactly each feature will be developed. The reason being that the customer is not paying for these sales meetings. Hence you don’t want to spend too much time doing the analysis and design work before any payments have been made or approvals have been granted i.e. you do not want to work for free. Furthermore, it can often happen that you’ll go through a long and rigorous analysis phase, designing and fully documenting a proposed solution only later to find out that the customer took your design to a competitor and ended up paying them for only their coding time. You may or may not be surprised to know that this in fact happens quite often. If you do this enough times your company will end up going out of business. The purpose of the sales meetings should only be to:
      • Sales Pitch: Pitch yours and the company’s services to the customer. The main aim of your first sales meeting should simply be to prove your competence and skills to the customer.
      • Requirements Overview: get an overview of their requirements for the software they’re looking to develop. Don’t get bogged down into specifics. The specifics should be left up to the analysis phase when the customer is actually paying for your time.
      • Proposal: put together a proposal for the solution you plan on implementing. This proposal should not be a functional and especially not a technical specification. There should be no mention of specific features or technical details of how the software will be developed.
        • Technical Information: from a technical perspective it should only provide an overview of the technologies (hardware, frameworks, SDKs and programming languages) you plan on utilising to provide the solution. You can  provide them with generic architecture diagrams, but don’t spend too much time creating custom diagrams that are specific to their business.
        • Solution Benefits: you should mention the benefits of implementing such a solution. This could be an increase in productivity, accuracy, automation etc.
        • Costing: this is the tough part that most people get wrong. Only through experience will it become easier to prepare costing estimates.
          • Development Costs: firstly, you need to think of the time required to develop the software by breaking down the tasks. If you’re using XP and Agile, these could be your stories or whatever you like to call them. If you’ve done similar projects in the past it will be easy to make an estimate. However for features that you’ve never implemented before, I would advise you to always be conservative i.e. think about the time required for research, testing debugging etc. If you’re not the one developing the software, this process will be become even more difficult in which case you will need to speak to the developers and get estimates from them.
            • Developer Confidence: Keep in mind that you cannot just take the developers’ estimates at face value. The reason being that some developers provide optimistic estimates, while others provide pessimistic estimates and this is based on their confidence, which may or may not necessarily have anything to do with their actual skill level. Confident developers typically tend to under estimate the time and effort required for coding, while under confident developers will overestimate time and effort.
            • Code Sloppiness: Some developers can be incredible quick to write code but they code can be sloppy. For such developers, you need to add buffer time for them to go back to refactor and debug etc. Others are perfectionists and take really long to code, but when their work is done it’s actually done. Therefore, you need to take all of this into account on the overall costing.
            • Buffer Time: Once you’ve come up with a fair estimate, it is best to always add a further buffer by multiplying the estimated hours with a factor. This factor is something of gut feeling you get through experience. For example, suppose a developer gives you an estimate of 10 days of development, you could then multiply that by 1.5, thereby your estimate to the customer being 15 days. The reason for this buffer is to take into account unexpected issues that can and do always pop up. Some may say that allocating extra time encourages procrastination, but I think that’s non-sense because it is the project manager’s job to check on up progress and ensure that people are meeting their timelines.
            • Do not quote on details: when quoting for development work, do not make the mistake of separating and displaying costs for every little coding task e.g. do not quote for each and every single button that they want on a form. Instead group the costs together and have only one or two development line items on the quote. The reasoning behind this is because you do not want the customer interrogating you about every single line item e.g. “Why would it take 8 hours to develop a single button.” The purpose of not quoting on details is not to be dishonest about the costs, but rather to avoid having technical debates with a customer that isn’t technical i.e. the customer has no idea how much work goes behind implementing the logic of that specific button and therefore to the customer it just looks like a simple button, while there could be 2000 lines of code behind that button for it to do what it’s supposed to do.
          • Project Management Costs: think about all the time that will be spent on the phone, writing emails, in meetings and driving around to those meetings. You must also take into account the kind of people the customers are when making this estimate. If your customers are professionals that know exactly what they want and they themselves plan ahead, then the project management costs can be reduced. However, you will have customers that know very little about software projects, are unprofessional, will constantly change their minds about what they want and the goal posts and timelines will constantly change. For unprofessional customers, I’d advise you to increase the project management costs substantially because you’re going to be doing a lot of baby sitting.
          • Analysis Costs: this is the time required for you to sit in workshops and meetings to discuss and finalise requirements and write up documentation. This time needs to be paid for because at least a third of your project will involve analysis work. It is for this reason that you should never make the mistake of starting the analysis phase before presenting the proposal and getting the customer’s go-ahead on the project.
        • ROI Analysis: finally once you’ve come up with a final number for the costing of a project, you then need to sell it against a value you’re providing to the customer’s business i.e. you need to give your customer a financial reason to spend the money on doing the project. Simply listing reasons is not going to cut it. Your motivation for doing the project needs to be have financials attached to it and that is typically the ROI (Return On Investment) calculation. In most cases, customers will already know why they want the software implemented, that being the reason why they called you in to take on the project. However, that may not always be the case, especially in cases when you are the one approaching them. In such cases you need to show the customer that doing the project will be an investment in their business and like any investment it must have a return/payback in a certain amount of time. So for example, if the project costs will be $100 000, you need to show the customer that the software will reduce costs in other areas of their business thereby enabling them to save X amount of money every month.  Those savings will amount to the total investment in a certain amount of time and that will be your final ROI payback period. Going forward the customer will continue to save for as long as they continue using your solution. In other words, the purpose of an ROI calculation is to establish a win-win deal with the customer whereby you earn money from the project implementation and the customer wins even more in savings over time. Selling based on ROI is the most difficult kind of sale you can make because you will need to have a thorough understanding of the customer’s business in order to put together an ROI calculation. However this kind of sale is also the most rewarding because instead of selling based on billable hours, you’re selling based on the value your software solution provides. It always helps if you are already familiar with the specific industry that the customer is in. If not, then you will be required to do a lot of research into the industry. Either way you will need to ask the customer a multitude of questions regarding their business before you can understand how to solve their problems and cut their costs.
      • Project inception: once you’ve presented the proposal to the customer you can expect to have a few backwards and forwards rounds of meetings and discussions about the proposal. Do not make the mistake of starting a project without the go-ahead from the customer. If it is a new customer that you’ve never dealt with in the past, I would advise you get a formal approval in written form from the customer. This can take the form of a PO (Purchase Order) or otherwise some sort of a signed contract.
      • Analysis and Specification Documents: it is only at this stage once you’ve gotten the go-ahead from the customer that you can start working. However, it is still not time to start coding just yet.
        • Project Plan: First and foremost you need to put a project plan together. The project plan will consist of the timelines of when each project deliverable (feature/work) will be completed and which resources (people) will be working on those deliverables. Both you and the customer need to agree on the project plan and the timelines need to be in line/comparable to the project costing in your proposal.
        • Functional and/or Technical Specifications: included in the project plan should be time allocated for analysis work and writing up the functional and/or technical specification. This phase of the project will involve having workshops and meetings to discuss the requirements of the software. Throughout this process, you will document the requirements in a formal specification document.  Many developers will say that documentation is a waste of time, but in the business world these documents will essentially server two purposes:
          • Contract for work required: The specifications will serve as a contract for what the customer can expect of the software. No more and no less than what is specified in the document will be delivered to the customer for the agreed upon price. It is for this reason that many software companies only quote the customer after the analysis phase, because they will say that they cannot quote for the project without fully understanding all the project requirements in detail. However, remember that I mentioned that in the costing process you should always add a buffer i.e. allocate extra budget and time for unexpected costs. This buffer that you added in your costing should cover any unexpected work that is being requested during the analysis phase and it will also protect you from doing the analysis phase at the risk of not getting paid for this work.
          • Mitigating scope creep: without a specification in place, the customer can continue asking for changes and you will end up spending way more time on the project than you initially anticipated all the while the customer is getting all this additional work for free.
      • Billing:  Many software companies bill only at the end of the project with an agreement in place that the customer only pays once the project has been completed and the solution has been implemented and tested. This is a huge mistake. The reason being that the customer will be under the impression that the project costs you presented in the proposal are a flat fee and in such a scenario the customer will not take into account the accruing costs of changing requirements, calling you in for additional meetings etc. All of that can land you in a mountain of scope creep costing your company ridiculous amounts of money. Hence it is important to:
        • Bill actual hours not estimated hours:  It is crucial to make the customer understand that the costing you did in the proposal are only estimated costs and that they will be billed the actual hours worked instead of these estimated hours. This clearly tells the customer that it will not be a flat fee for the project, but instead can change based on changes in the requirements. If you added a buffer to the budget (over estimated costs) in your proposal, the customer will end up being pleasantly surprised when the costs of the project are less than the estimated costs. However, if the customer continuously asks for changes throughout the project even after the specification document has been agreed on, these changes will be handled as CCRs (Change Control Requests) which fall outside the scope of the initial specification. The CCRs must be paid for and given enough of these CCRs the total cost of the project will end up going over the initial costing in the proposal. Having these mechanisms in place will encourage the customer to think twice before asking for changes to the initial agreement and will protect you against scope creep.
        • Bill in cycles: Bill the customer for the hours worked in continuous cycles e.g. monthly or even every two weeks. The developers and other resources on the project will then have to keep track of their hours worked and ensure that the costs are not going over budget. If the costs do end up running over budget, either because your initial estimation in the proposal was wrong or because of CCRs, the customer must be alerted of it and approval must be gotten from the customer before continuing with the project.

Keeping all of the above in mind, here’s a summary of golden rules to follow in order to manage the costs of a project.

  1. Always over quote and under bill instead of under quote and over bill.
  2. Do not get into too many details in the sales cycle i.e. when drawing up the proposal and estimating the costs of the project.
  3. Bill actual hours worked instead of estimated hours: Explain to customers that the quote is purely an estimation and that they will be billed actual hours worked rather than estimated hours. Explain to them that if there are no change requests, they can most likely expect the total cost to be less than the estimated cost in the initial proposal.
  4. Never work for free: aside from the sales meetings, proposal and costing, no other work should be done for free without formal approval from the customer. Even after the project has started, any change requests must be paid for.

Why do large software companies open-source their projects that they’ve invested so much into?

This is a good question, because it seems like a bit of a contradiction: these companies are there to make money and they’re spending money to develop these products but they’re still “giving” their code away for “free”.

Obviously I don’t know exactly what the thought process is by the executives of these companies, but I can take a guess. My guess is that open-source is great if you want to:

  • Lower costs: get others in the open-source community to work for free in testing and spotting bugs in your software.
  • Lower risk: this goes along with lowering costs i.e. lowering the costs lowers you investment risk. Furthermore, getting feedback from the open-source community is a lot quicker than having to wait for the software to be released and only thereafter get feedback from your users, by which time they might have created a negative impression with the end-users (if the software is full of bugs).
  • Increase adoption rate: most of the products being open-sourced are SDKs and other development tools. This is very beneficial to developers using those development tools i.e. because it gives the developers more freedom and control. In the long term they end up with more developers/users hooked on their platforms at a quicker rate. As an example, I’m guessing it’s one of the reasons Microsoft opted to finally open-source .NET because it would get a lot more developers hooked on their platform, programming languages and development tools, which ultimately results in more apps being developed using Microsoft technology. Once they’ve increased the number of apps on the market developed using .NET, I’m pretty sure Microsoft have plans to capitalise on that in one way or another.

When to open-source and when not to open-source

If you’re a developer and have no intentions of every getting paid for writing code, then deciding between open-sourcing your software or not can be a simple decision. However, if you have any intentions to get compensated for your work, then the decision to open-source or not can be a complicated and long winded debate which has gone on for decades. On the one side you have the capitalists/conservatives that sell software for a price and their pricing models are relatively simple to understand especially in commercial and non-enterprize environments i.e. you get this product for this price. On the other hand, you have the open-source socialists/liberals/anarchists, advocating for making software “open”, transparent and most of the time offering it for free. However, for anybody that has ever delved into the open-source debate, they will quickly realise that open-source is in fact not that clear and transparent after all.

It’s all good and well to spew out socialist rhetoric about how we should all be more altruistic by volunteering our time and skills to open and transparent software communities, but once we’ve all had our turn at singing kumba-ya, the truth is that in reality there’s no such thing as free lunch. Meaning that although many open-source enthusiasts pitch the idea of so called free software, the users will still end up paying for it one way or another, be it in some sort of fee, time, energy and/or frustration. The reason being that in the real world, time equals money and money makes the world go round. Therefore, any developer working on a “free” and open-source project still needs to pay the bills, put food on the table and at the very least still have money left over to buy a laptop on which to work on. So how do open-source developers get compensated? Open-source projects typically start off as side-line hobby projects, then perhaps moving on to offering paid for versions and/or charging for services.

Over the years I’ve thought about this debate countless number of times and like most analytical people I’ve searched far and wide for an absolute answer i.e. is open-source good or bad. When I was younger I leaned more towards the open-source point of view and their anarchist/hippie rhetoric about how corporations are evil, money-grubbing establishments designed to create monopolies and control people’s lives. Now that I’ve gotten older, my view has become a bit more conservative. Maybe it’s got something to do with there being a grain of truth in Winston Churchill’s wise words where he said that “any man who is under 30, and is not a liberal, has no heart; and any man who is over 30, and is not a conservative, has no brains.”

So here’s my attempt at an absolute answer as to when to open-source and when not to:

  • How most open-source projects start off: as the saying goes, “necessity is the mother of all inventions”. By that token, most open-source projects start out from a need/problem that someone has. A developer may want to achieve something that they cannot do with commercial software that is available or they just don’t have the money to purchase commercial software that can perform the given task. While developing this new app for their own needs, it suddenly dawns on the developer that other people may have the same need/problem that their new app can solve. At this point the developer begins to think about how they could possibly get others to use this new app and possibly capitalise on it. After careful consideration they realise that there’s more to selling software than simply putting a price tag on it i.e. a company needs to be setup, marketing needs to be done, payments need to be captured, orders and invoices need to be created, taxes need to be paid and amongst many other things, all sorts of people need to be hired e.g. marketing, sales, finance, support etc. Starting and managing a company requires a lot of work and responsibility. Due to the fact that a descent amount of capital is required, it also involves a huge amount of risk i.e. it’s not always easy to predict whether or not a software product will be a hit or not. So as opposed to going through that whole nightmare, our friendly developer decides to rather just upload his code to a popular open-source repository that all his friends can access. If nothing else, it will look good on their resume.
  • If the software is a hit: if there proves to be a substantial demand for the software, the developer may begin thinking about capitalising on it. At this point the developer will probably be stuck between a rock and a hard place. If he closes down the open-source repository and begins to charge for the software he’s fear is that the users will just continue using the latest open-source version available and never purchase any new versions. Even if there are some people willing to pay for the software, the problem is that the developer will never be able to sell it to enterprise customers because no large company will agree to adopting a product that is being supported by a single individual i.e. it would just be too risky for the company. So it’s a bit of a chicken and egg scenario i.e. you need to generate revenue to hire more people, but customers are hesitant to commit to a one man show. On the other hand if he keeps the open-source repository available, then there is no way to make money on it. To get out of this dilemma, most developers in this situation either begin selling services or offering paid for upgraded versions with additional features.
  • Charging for services approach: most open-source developers will write the code for free to develop the software, but if users need any help with specific feature requests, installations, advice or any consulting services, they will be charged an hourly rate or are sold a 1/3 year Service Level Agreement. As a bit of a realist, it is at this point that I start questioning the incentives put in place for this type of revenue stream. I ask myself, what incentives do open-source developers have to produce quality software that is easy install and use when their primary revenue stream comes from selling services? If charging for services is your only revenue stream on an open-source project, then there is a clear conflict of interest in this approach and I would advise against it. Reason being that what users want is software that is easy to install, configure and use, while at the same time the developers are well aware that the easier the software is to configure and use, the less services they will sell. So what you end up with, is open-source software that is counterintuitive and users are constantly left with more questions than answers. In this kind game with these kind of incentives in place, there is not much that is open and transparent to the users i.e. the software and code is available and free for anybody to use, but nobody knows how to set it up and us it. You may be of the opinion that it doesn’t matter whether there’s a conflict of interest or not, as long as you’re making some money off of your efforts. However, my argument would be that users are not stupid, they always know when there is a conflict of interest and sooner or later they will start posing questions. This approach also creates confusion that results in users asking why you’re doing some work for free while charging for other work. This is where open-source can lack transparency. With the above said, there are some more negative side effects with this approach:
    • Minimal documentation: the whole point of writing documentation is to make it easier for users to understand the process of installing, configuring and using a given piece of software. Given the fact that many open-source software is offered for free, the developers have almost no incentives to write comprehensive documentation i.e. the more difficult it is for a user to get up and running and troubleshoot issues themselves, the more services the developers can sell. What you naturally end up with is open-source software that often has little to no documentation aside from a simple two page quick start guide. The counter argument from open-source developers is that “the code is the documentation, just look at the code”. But if we cross over from the delusional hippie world into the real world, we realise that most users are not coders and therefore having the code adds zero value to their lives. Even as a coder myself, if I’m interested in using someone else’s software I will neither have the time nor the interest in doing support on other people’s products. Personally I just want to be able to click install, use the software and move on with my life because I have enough of my own code to deal with, without having to deal with other people’s broken code.
    • Lower quality of open-source software: in terms of quality, the argument made by open-source purists is that their software is of much higher quality than that of proprietary software. According to them this is due to the fact that open-source communities are larger and there are more people testing, spotting bugs and fixing them as opposed to small teams that develop proprietary software. This is based in Linus Torvalds’ law that “given enough eyeballs, all bugs are shallow”. That does indeed sound plausible, but on the other hand it seems to me like a bit of a contradiction to the many open-source companies that rely on services as their primary revenue stream. If the quality of open-source software is so incredibly high, then how do they make their money on services i.e. if there are no bugs to fix, the apps are easy to use and everything just works, then what kind of services are these developers selling to pay the bills? Developers creating proprietary software on the other hand have every incentive to write high quality code and create intuitive user interfaces, because if they don’t, then nobody will buy their software.
  • Paid for upgraded versions approach: an alternative to only offering services is to develop additional features into the open-source software and offer those features at a premium i.e. offering a “free” open-source community version as well as a paid for version with additional features that are enticing enough that make users want to spend money. Open-source can be fantastic when you want to lower your risk while at the same time increase the user adoption rate of your software. By open-sourcing your software you can start a project without having to take on the risk of starting a company, investing capital etc. and if there proves to be a high enough demand for your software, you can then take it to the next level by offering additional features and charging for them. This is the only time I would see open-source as good idea i.e. assuming that you actually want be compensated for your work.

In summary, before open-sourcing your software you should consider the number of features that you have in mind and whether or not there will ultimately be a demand for those features on the market. There are many apps that can be incredibly useful, but beyond their basic features there are a limited number of additional features that can be developed and incorporated into future paid for versions. A great example would an SDK (Software Development Kit) to interface with third party application or hardware: there are only so many features you can add to such an SDK. This kind of software with limited useful features should either be open-sourced without any intention of ever making money or it should never be open-sourced, but rather be released and sold as proprietary software right from the start.

Productivity and creativity are like yin and yang

This may be a bit of a controversial topic which many may not agree with, but throughout my life I’ve come to the conclusion that productivity and creativity are a bit like yin and yang i.e. they are often indirectly proportional to each other.

As a kid, when I was in school, I was very good at maths and art. I was either terrible or average at almost all other subjects mostly because they didn’t interest me, but that’s besides the point. Like all students in South Africa, as I got further into high school I needed to chose which subjects to take/keep and which to drop. My father suggested I take a more technical path in my studies, which included maths, science, electrical theory and technical drawing. His reasoning was that a technical path would be more practical in the sense that it’s easier to make a living later in life by doing something technical rather than being a struggling artist, because generally speaking people are less willing or able to pay others to draw pictures. I loved drawing and being creative, but on the other hand, having immigrated to this country with only two suitcases and knowing what it’s like to be poor, I couldn’t dismiss my father’s advice i.e. making a good living weighed heavily on my mind. After much deliberation, I decided to drop art. When I went to varsity, I made the same decision when having to chose between studying Graphic Design and Computer Systems Engineering i.e. combination of software and hardware subjects (programming, electrical engineering, electronics, digital systems etc.). Throughout varsity I spent huge amount of my spare time producing electronic music. Although I loved coding, the feeling I got from making music was like no other feeling I’ve ever gotten in my life; it was like a drug that took over my life. After having started my first job, I once again made the same decision to drop the music in order to focus on coding.

I realized that in order to succeed at a particular thing in life, you often have to sacrifice other activities that you love doing. Therefore, aside from perhaps designing user interfaces and choosing the clothes I buy, I have done little to no exploring of my artistic/creative side ever since I started my career in software development. It is for this reason that not a day goes by where I don’t think about the decisions I have made. However, to this day I still believe that I made the right decisions and here’s my reasoning behind it:

All the way back in high school I realised that as much as I loved drawing I could not imagine being asked to draw/paint on demand for any teacher, lecturer, manager or customer. The same applied to making music; I couldn’t put a track together on demand. I could only do these things when I felt inspired. On the flip side, I could solve an equation, troubleshoot a technical issue or write some code on demand. Unfortunately, to make a living in this world people will expect you to produce on demand not just when you feel inspired. It is for this reason that I believe I made the right decisions in my life given my circumstances.

In the quest to make more money, I could have chosen to become a stock broker, a salesman or accepted a job writing software for banks. But instead I chose a career in software development, writing various kinds of software from mobile, to desktop, web and speech recognition apps for various types of industries. My salary will never come close to that of a stock broker, banker or insurance salesman, but to be quite honest there are limits to how much I’m willing to sell my happiness for.

In my view, software development serves as a perfect equilibrium for my personality: it is a combination of intellectually challenging tasks/features that need to be developed on demand while requiring a certain amount of creativity for designing elegant technical solutions and user interfaces.

This finally brings me to my point, that throughout my life I have often noticed that I am most productive when working on mundane repetitive tasks, while I am least productive on tasks requiring me to be creative; mainly because being creative requires extended periods of time thinking, day dreaming and searching for inspiration. As an example, one will probably find that asking a factory worker to drill holes all day will yield a large amount of productivity, which in turn will increase a company’s profits, but the person’s quality of life and happiness will diminish day-by-day. Inversely, you will find that asking even the most talented painter to create an original and world renowned painting every week/month will not prove to be a very productive endeavour.

In line with Joel Spolsky’s thinking, you can either be a world renowned chef like the Naked Chef or you can be McDonalds: Big Macs vs. The Naked Chef. Although I don’t agree with Joel’s bias towards the Naked Chef, I do believe there are some lessons to be learnt from his analogy. McDonalds’ main appeal to the public isthe speed (productivity) in which they prepare burgers and meals. What is not mentioned in Joel’s post is that McDonalds will always make more money globally than the Naked Chef because it requires less creativity and more productivity. A chef’s success on the other hand is fully dependant on talent and creativity, which cannot be scaled nor can new and original recipes be created on demand. The only caveat with aiming to be a world renowned artist/chef is that very few make it to that level, while most end up living off of coupons and hanging on to their pipe dreams.

So how does this apply to software development teams/companies? Here’s an example of two companies I worked for in the past, both consulting companies. I won’t mention any names, but it’s essentially a comparison of the Naked Chef vs McDonalds.

  • Cowboy Company: a small startup where I was the first employee. It was founded and managed by one of the most hard working, talented and creative software developers I’ve ever met in my life.
    • Work environment:
      • We were all encouraged to have a can-do attitude i.e. nothing is impossible, anything can be done and we can do it over night.
      • There were no boundaries between us and our customers in terms of budgets, time constraints etc. If the customer wanted a product developed that would typically take 6-12 months to develop, we told the customer we could get it done in two weeks at obviously a fraction of the cost that any other company quoted at.
      • Experimentation and hacking was encouraged i.e. software tools, utilities, programming languages, repositories, APIs, SDKs etc. were swapped and changed on a regular basis.
      • Little to no training was offered internally. If you were working on something new and involved a learning curve, you were regarded as an idiot if you asked questions.
      • The day-to-day work environment had zero structure, rules or processes. Thus it was the most chaotic work environment I’ve ever been in. For the first time in my life at the age of 24 my hair started falling out to the point that I was wiping my hair off the keyboard every hour on the hour.
    • Outcome:
      • Company outcome: the company was in existence for only about 18 months before we were told that we’re closing shop.
      • Reasons for the company outcome:
        • If creative people make for the worst employees to manage, then the same creative people make for even the worse kind of managers for the day-to-day running of a company. Here’s why: the process of being creative requires a huge amount of experimentation and trial and error. This experimentation process is unfortunately very counter productive because you’re not walking in a straight line, but rather zigzagging your way to nowhere in the hope that one day you’ll find the pot of gold.
        • Creative people make the worst kind of managers for managing people and budgets. Ensuring that money always comes in and that people get paid their salaries requires structure and discipline. Managing people’s productivity requires rules and methodologies to be followed in order to set expectations because as we all know the most common cause of unhappiness is unmet expectations.
      • What happened to the manager: this highly creative, talented and unstructured man ended up becoming the CTO of the world’s second largest gay social network, which he developed from scratch. Within a few years, they acquired the largest gay social network in the world. Needless to say that he’s become very successful in the end.
  • Structured Company: a large multi-national goliath of a tech company, with very rigid rules and regulations.
    • Work environment:
      • Highly structured and regulated work environment. Everybody had specialised roles knowing exactly what needs to be done with clearly defined goals and expectations.
      • A strong emphasis was put on training and documentation. Everybody was taught everything they had to know in order to get their job done. Asking questions was encouraged and the managers were always happy and enthusiastic about sharing their knowledge.
      • The management of people was pretty good; people were treated with respect, expectations were set and met etc.
      • The software consulting department I reported to was run like a well oiled machine, with projects being delivered on time and within budget. No major surprises ever came about (within the department), and every change introduced was carefully planned out.
      • Everybody was encouraged and directed to follow a straight line to success, which was great in many ways for your own mental health.
      • On the other hand, new ideas, strategies or ways of thinking were strongly discouraged. Internally, the typical company slogan was “this is how we do things around here” or “this is not how we do things around here”. In other words, there was little to no room for creativity or original thoughts … unless if course you were considered someone of importance i.e. you had a VP (Vice President) somewhere in your job title. Putting up your hand in QBRs (Quarterly Business Reviews) and mentioning my concerns and ideas often resulted in being laughed at or told that “you clearly don’t know how this company works”. Coming up with ideas for new apps or ways of generating more money resulted in being told that “it’s not your job to be thinking about that, don’t look to the left or to the right, just look straight ahead and do the job you’ve been hired to do”.
    • Outcome: over the course of a decade the revenues of this goliath of a tech company started dropping year-by-year. Eventually it ended up being cut up into pieces and sold off.
      • Reasons for the company outcome: innovation stagnates in an environment where creativity is choked in favour of productivity. The company stopped innovating and instead only released incremental improvements on their existing products. The people in the highest levels of management believed that the strategies that worked two decades ago will continue to work today. I personally witnessed such managers looking at spreadsheets in QBRs and scratching their heads as to why the revenues were plummeting. All the while blaming the sales people for not pushing more sales, without ever realising that you cannot sell yesterday’s technology for tomorrow’s prices.
      • What happened to the management: the company was acquired by another stagnant company, and they are still conducting their business the same way they always have, all the while still living in fear of their revenues dropping which continue to do so. To this day they still refuse to invest in R&D on certain technology stacks and prefer to purchase, rebrand and resell their partners’ products. Certain components in their hardware products are even purchased from their competitors. Due to these people being focused primarily on productivity and profits, their only concern are the earnings for the next quarter, and then the one after that. Short sightedness and puddle thinking is embedded into the cultures these types of older corporations.

The moral of the story is that companies which purely focus on productivity will always outperform creative companies in the short term, but they will never hit the jackpot. Creative companies on the other hand will always struggle in the short term and many will fall by the wayside, while a small minority of them will hit the jackpot in the long term. In the end it all comes down to high risk high rewards for creative companies and low risk low rewards for productivity focused companies.

On a more personal note, depending on your level in an organisation and your perspective, here’s what you can take away from the story:

  • If you’re an employee: part of growing up and maturing is the ability to come to terms with your own limitations. So you firstly need to know your own strengths and weaknesses and ask yourself whether you’re the creative cowboy type or the structured book smart type of person. Thereafter strive to work in environments where you are surrounded with like minded people. Personally I would advice you to work for a mid-sized company which is still in the process of growing, thereby offering opportunities to be both productive and creative.
  • If you’re a manager: as a manager you don’t (or should not) have the luxury of being biased towards either types of employees. Realise that you may have productive and creative types working for you and an ideal tech company requires both kinds of people i.e. you need the productive people to milk the cows, while you need creative people to invent new cows to be milked. Incentivise people accordingly. For productive types give them tangible goals and deadlines and you’ll probably learn that you’ll have to micro manage them. For creative types, give them a bit of freedom, a generous budget and time to play and experiment.

So summarise: productivity and creativity are indirectly proportional, but ideally they both are required for a tech company to not only thrive in the short term but also stand the test of time. Hence the I believe productivity and creativity are like yin and yang; they’re opposing forces but you still need both.