Background Banner Video

Lorem ipsum dolor sit amet, consectetur adipiscing elit

Background

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

How to Build Enterprise Solutions With a Small Team

Written By: David Berman, Vice President of Technology, Infront X

Today’s tools enable a small team (2–5 people) to build enterprise class solutions that scale rapidly. This article will show you how to outperform large teams and get your product to market faster too.

First I’m going to summarize the most impactful things you can do to enable a small team to crank out value fast. Then I’m going to get into some specific things they need to do so that you have a solution that scales, is reliable, and maintainable. Bugs are going to surface in production, so you want to build things in a way that you know when they are there before your customers do, and have the tools to fix those things very rapidly.

 

Minimize communication

  • Adding people to a team increases communication exponentially. Going from 2 people to 3 people on your team takes you from a simple drag and drop of a ticket in Trello to needing meetings.

  • Keep meetings to a minimum and as small as possible. Don’t default to pulling in everyone. Two people can hop on a Zoom or Slack huddle at any time to bang something out, but to get 3 people in your meeting now you need to schedule it for next Thursday to find a time that works for everyone.

  • Status can be shared daily with a couple of sentences in a slack channel, or you can hold daily standup, depending on the seniority of the team. An advanced team is comfortable proactively raising impediments and communicating if tasks need more time than originally estimated, or asking team mates for help. If your team isn’t on that level yet, have the daily standup to make sure people are working on the right things, and that someone on the team can be working to remove impediments. Make sure it’s clearly understood during stand up if there is an impediment to deal with; sometimes people are reluctant to ask for help.  After standup, leave developers to concentrate uninterrupted for the rest of the day.

  • If a developer is having trouble with something, share that with your team and get help from the team mates. Encourage asking for support and going to pair programming workshops. It solves problems much faster. Avoid working in a bubble, invite feedback. This will ensure team mates understand what each other’s code is doing, so they can more easily integrate with it or support it.

Write the least amount of code possible

  • Limit scope. Make people fight for features. Take the term “MVP” seriously. The biggest waste in software development is building things you don’t need.

  • Avoid implementing reports, especially for new products. It’s very difficult to build reports well. When you start, you don’t usually know the right questions to ask. The team can sink valuable time writing code to capture user behavior events, more time to query that data to produce reports, only to realize that the wrong data is being collected. Save reporting for when projects are further along and better understood, and plan on iterating several times on reports to refine them.  In general, it’s usually better to use a business intelligence tool like PowerBI or Excel on top of a SQL database like AWS Redshift then it is to build your own reporting system. A light weight analytics package can be helpful, but you need to make sure someone on your team really understands how to use it well and has time to study the data, otherwise postpone that until later.

  • Plan your code before you write it. For anything that isn’t trivial, take some time to think through what you are building in detail and write out some notes for yourself. Its highly recommended to discuss the planned approach to a complex problem with your team during planning poker, or in a 1 on 1, it will save a lot of time refactoring later.  When you sit down to write a function that’s going to be more than 2 steps, list the steps the function is going to perform in the comments before you start. By planning things out you will avoid a lot of rework and get things done much faster.

  • Finish what you start. If you work a little on something, and then switch to something else before you are done, you have the cost of the work but no return on that investment. Coming back to it later means reloading all the context, so it will take much longer to get the job done if you ever return to it. Keep the scope of tasks as small as possible so it's easy to finish things that add value to the product increment (the next version you plan to release).

Apply these best practices

  • Establish success criteria up front. For example, a latency goal like web pages must load in under 1 second, API requests must respond in 200ms or less, or operating cost must be under $10,000 per month. If your team doesn’t know what they are aiming at, they can’t hit the target.

  • Use automated testing. Automated testing is the only way your small team is going to program a lot of functionality at low cost. Here’s why: Each time you build a piece of functionality, you have to test that you didn’t break the functionality already in your product. That means that each release becomes more expensive than the previous release and takes longer!  With automated testing, you incur some additional overhead during development to write those tests, but the cost per release to do regression testing (and time taken) stays flat. If you skip this you’ll see your costs keep going up, it takes longer and longer to release, and in production things will break that used to work and your clients will get very angry very fast and not trust you any more. There are lots of types of testing. If you are under severe time or resource constraints, end-to-end testing is often a good fit. If you make an API, apply functional testing to validate invalid or missing inputs are handled and valid inputs give you the expedited result.

  • Comment before you code. Before programming your function, outline what it will do in the comments. Then fill in the code. This is going to save you time because you planned out what you are going to do instead of refactoring your code several times. It also enables other people to understand your code. This is especially important with JavaScript code that has anonymous and asynchronous functions written with concise syntax. Maintainability means others can work on your code.

  • Validate API input parameters. The first thing you do when writing an API method is validate the input parameters and send a response back with a 400 code if there is something missing or invalid, and a message explaining the issue. Don’t waste computing power processing invalid requests. Do send back enough information to enable the developer consuming your service to understand what is wrong. This saves lots of time and eliminates communication between them and the dev writing the API.

  • Integrate detailed logging from the beginning. If you don’t have logging, you are completely blind. When things break in production you will have to make your team work nights and weekends to guess what might be wrong and lob “Hail Mary”code patches at your production server. Instead, integrate detailed logging from the beginning and pair this with structured exception handling (try/catch blocks). Make sure that if anything doesn’t work, it’s written to the log, with the data that failed. In production your code will encounter inputs that you didn’t expect. Make sure to log where in your code the issue occurred, what data couldn’t be processed, and then throw a 5xx series error. Use log levels so you can see very detailed logs in lower environments but log only warning and errors in production where logging gets very expensive very fast. For example: logger.error(“[orderSandwhich] Error processing request for cheese sandwhich. Out of cheese.”, request);

  • Utilize Application Performance Monitoring (APM). With APM, a library is integrated with your code that tracks each request coming into your website or API service and tracks where that method spends its time. It will show you what your latency is over time with mean (p50), p90 and p99 probability so you know what your slowest user experience is. It will detect when new errors occur so you know what build the bug was introduced in. It pairs each request with the logs of the request. It will show you which requests are slow, what database queries they ran and other insights. This will enable you to discover problems in the field and prioritize them based on impact and frequency of occurrence. When you do it right, APM will show you how often errors occur and where in your code they occur, paired with your logs with the data your app couldn’t process. This will enable you to solve production problems very fast and enables your team to deliver with confidence.

  • Load test and stress test. Before deploying a service you must run a load test and a stress test. The load test will validate that your service can operate at the expected peak level of demand, and enables you to measure your latency and throughput. For most enterprise solutions, throughput (requests per second) and latency (time spent waiting for a response most of the time) are the most important factors. You should ensure that the response time of your API / service meets the target (for example, under 300ms per request) at the peak expected load. You need to measure how many requests per second each worker process can add so you can plan your operating cost and server capacity. Plan for the future, take a step back and think about what bottlenecks you might encounter as the solution scales. The load test will help you identify where those bottlenecks might surface first.

  • Use caching effectively, starting with a CDN. Put your solution behind a content delivery network and web application firewall.  This is easy to do and gives you an immediate boost in performance. A content delivery network shields your origin servers when secured properly. Make sure to document for each API method what the cache TTL policy is and include that in your automated tests. Don’t allow traffic from the Internet to go directly to your origin server. A web application firewall gives you a lot of protection without having to spend a lot of time on it. If someone else breaks your service YOU will be blamed. Once CDN caching is in place, measure the performance of your application and evaluate other ways where caching can be used. By caching slow queries, your application will scale greatly. If you introduce a Redis cache, you can often see significant gains and scale down your database infrastructure, washing out most of the additional operating cost. Redis is very easy to work with and is interesting for the development team. Make sure to configure the cache key eviction policy, as the default policy could cause you to run out of memory if you don’t put expiration policies on every cached key.

  • Automate your deployments from the beginning. Choose a platform for scaling from the beginning, such as cloud functions, containers, or auto scale groups. Automate the deployment process from the beginning. Running automated tests before deployment is part of this.

Examples of tools:

  • Load testing: Loader.io, used for affordable load and stress testing.

  • APM and Logging: Stackify or DataDog. NewRelic APM is good but logging is weak. Be careful to filter by log level when logging in production at high volumes as you can get a nasty surprise log related bill.

  • Automated testing: PostMan is a great tool for documenting your api and adding automated testing, but for a few complex scenarios, a JavaScript is better. Make sure you can run your test suite against any environment including production. This enables you to validate a deployment was successful.

Interested in learning more about how you can leverage efficient teams to build large-scale enterprise solutions?  Feel free to reach out to us through our contact form.