Tuesday, 7 August 2018

Fail Early, Fail Fast and why you should throw exceptions

Debugging is a necessary part of software engineering. All software has bugs, and so debugging goes with the territory. Some are simple to fix. Some are only show up under certain (sometimes unusual) conditions, maybe after an operation has been running for some time or if that operation occurs at a certain time of day. Sometimes applications catch bugs and produce a helpful message or  a stack trace in logs that can help us lead us to where the bug occurred. And sometimes they don't because exceptions are trapped and that information discarded.

It's rare, but most developers have experienced bugs during their career that take them away from feature development for a day or even a week at a time.

Such bugs often present them in a form such as:
  • "The application locks up at around 10am every morning and we don't know why."
  • "The user clicks the button but nothing happens, and nothing shows up in the console."
  • "The customer says they placed the order but we can't see it in the back end."
  • "The wait message shows indefinitely"
There is a temptation for programmers to attempt to build robust applications by trapping exceptions and trying to 'handle' them. This maybe the correct action if the error condition is known to be frequent and there is a clear way of working around the error. For example when making calls to an SMTP server that is known to sometimes reject requests, or uses a less than robust network connection, the programmer might want to build in a retry mechanism. Of course it is useful for users or administrators of a system to have to deal with as few failures as possible.

All too often however, programmers catch exceptions without fully 'handling' the error. It might be that a program continues to run without a required value because the exception thrown due to a failed database connection was trapped and the method allowed to return a null value. Such a program might not fall over completely, but it may behave strangely, or not complete subsequent operations in the way that was intended.

The presumption has to be that every line of code in your application does something important. If not, why is it there? Allowing it to work with incorrect or missing data should be expected to be problematic. The resulting types of bugs, where something is 'wrong', rather than the application having visibly failed are usually harder to track down than the equivalent situation where the application has been allowed to fail.

Fail Fast, Fail Early

The solution to the problem is to 'fail fast, fail early'. Exceptions are called exceptions because they occur in exceptional circumstances. The programmer is not expecting the user to experience them regularly. Clear and visible failure prompts a fix, and makes the cause of the exception easier to find. In the long term, this is what results in robust software.

However, Don't (Always) Crash the Application

Making failure visible does not mean crashing the application. Presenting the user with technical looking error messages is rarely appropriate. In some types of application, such as software that runs on the desktop or as a service, allowing a crash is likely not acceptable, and for mission critical software applications potentially unsafe.

Making Failure Visible

So if an application crash is not acceptable, then some sort of error handling is necessary. This should take the form of a global exception handler. Where programs run multiple threads, trap exceptions at the highest possible level, either within the thread or, if your language / framework allows, by assigning an exception handler to the thread.

Ultimately, the operation that the application was attempting should stop, even if the application itself does not fully. The error should then be logged for the attention of a developer.

Where the program has a user interface and the operation was intended to result in visible feedback to the user, a user friendly error view should be presented to the user. This may prompt the user to report the issue. If that is likely, then it might be useful to print the date and time on the screen, and ask the user to take a note of what they were doing at the time when the error occurred.

Prioritising Errors 

Some errors are expected frequently, such as 404 errors when you run a public facing web application. Content that no longer exists or where bots or users attempt to guess URLs that never existed result in 404s. It might be useful to review these 404s intermittently, but it's unlikely that you want to know about every one. Application errors, and fatal errors that do crash a process or sub process likely require more attention. Some development teams write or copy logs to a centralised location and make failures visible on a UI that everyone can see. Smaller applications might email out errors to system administrators. There are services like Splunk that help to process log data, useful where large volumes of log information are created.

Final Note

So if you're going to fail, do it fast, visibly, and make sure you collect the information you need to direct you to the source of the problem. While this may result in more visible errors in early life for an application, it will result in more robust, easy maintain software in the future.

Further reading on this topic can be found below.



Wednesday, 19 March 2014

Using Amazon Web Services (AWS) Elastic Load Balancers (ELB) to Server Multiple SSL Secured Applications From a Single EC2 Instance

Standard EC2 instances have the limitation that only a single IP address can be associated with each. Amazon VPC allows you to allocate more IP addresses and network interfaces, the amount depending on the instance type.

The number of additional IP's that you can add does not necessarily equal the number of SSL certificates that you can install per instance, as you will need a network interface per SSL certificate. I use VPC for one of my smaller application servers that runs a couple of low traffic websites in this way, and it works well. 

What if you want more applications per instance though, or you don't want to incur the cost of a larger instance that has the required number of network interfaces?

One option is to set up multiple Elastic Load Balancers (ELB) that each direct traffic to the same EC2 instance. The ELB can listen on multiple ports, and each of these ports can be configured to forward traffic to different port on the target instance. In addition, HTTPS can terminate at the load balancer, and subsequent communication with the EC2 instance can go over HTTP. The user of your application still has their requests handled and responded to via SSL. This feature is what enables us to run multiple applications requiring an SSL certificate on the same EC2 instance.

There are cost implications to this technique, since VPC is free to use, and ELB's incur an hourly useage charge. However as the number of SSL applications per machine increases, and the instance sizes increase, the relative cost of each additional ELB goes down in relation to the comparative cost of running multiple instances because you need the additional network interfaces.

Based on 2 SSL applications per EC2 instance:

ELB Hours = $0.025 x 24 hours = $0.60 x 30 days = $36.00

So the cost of sharing the instance for additional SSL certificates is $36.00. However each additional SSL secured application can be achieved with a single additional load balancer at $18.00, which is where this method becomes more cost effective.

In addition there may be minor application code modifications required to use your web applications in this scenario. At least one application on the instance should provide a url at which the load balancer can call to check the health of the instance (the instance will be taken offline otherwise). In my case, one application is setup to receive all traffic not routed to a specific application, and respond with a 200 OK. Also, because comms between the load balancer and the instance is always over HTTP port 80, any code that checks for https or a secure connection may need to be altered to check for a X-Forwarded-Proto HTTP header, which the load balancer will insert into the request headers indicating the original protocol used for the request (.NET C# example follows later).

In this instance, the target EC2 instance is running two ASP.NET MVC web applications running on a Windows Server 2008 R2 EC2 instance.

Step 1 - Obtain SSL Certificates for Each of The Web Applications 

In this case follow the AWS instructions AWS instructions for generating a private key and CSR. Follow the instructions up to and including "Submit the CSR to a Certificate Authority" to the point where you have received your certificate file back from your SSL certificate provider. I used openssl via Cygwin on Windows to create the private key and CSR files.

Note that we will be installing each certificate into the ELB, not IIS, so we will not be creating our CSR using IIS as you might for a normal web application running under Windows Server. If you do subsequently need to import the certificate into IIS for some reason, you can convert the certificate and private key file into a .pfx file using openssl that can be imported into IIS.

Step 2 - Create 2 ELB's (Elastic Load Balancers)

Create each of the two load balancers as below. Note that the load balancer listens on port 443, but forwards comms over plain HTTP on port 80.

Since the load balancers both exist to direct traffic to the same EC2 instance, the instance for each of the load balancers will be the single EC2 instance that is runnin both of your web applications.

On configuring the HTTPS listener, you will be prompted to upload or select an existing SSL certificate. Choose the appropriate key and certificate files so that you can paste the content into the AWS UI when prompted.

Step 3 - Check Firewalls

You are unlikely to need to do anything here, since the load balancer communicates with the web server over port 80, so port 443, 8443 etc need not be open.

Step 4 - Set Up Bindings (IIS)

To distinguish web traffic targeted at each web application, each web application should have bindings configured to handle traffic for each appropriate host. These are only special in that there is not an HTTPs binding, and no server certificate uploaded.

Step 5 - Modify Application if Required ASP.NET MVC Context

In short, application code relying on 
Request.IsSecureConnection || string.Equals(Request.Headers["X-Forwarded-Proto"], "https", StringComparison.InvariantCultureIgnoreCase);

Basically, we are saying if the standard web framework says the connection is secure, then treat as secure, but also treat as secure if the X-Forwarded-Proto header indicates an HTTPS connection.
For the health check url that is required by the Elastic Load balancer, this can be as simple as the following:
public ActionResult Ping()
            return View();

Useful links

Another blog explaining how to set up SSL in EC2 Elastic Load Balancers


Discussion of the X-Forwarded-Proto HTTP header in ELB context


Stack Overflow Disction of MVC Require HTTPS and X-Forwarded-Proto HTTP header


Converting private keys and CSRs to .PFX format for IIS import (not required for ELB SSL)

Ping / Xport forwarding