AWS Lambda functions become more flexible, but use caution
AWS Lambda added custom runtimes and longer timeouts. Review the benefits, such as flexibility and easier migration, along with the drawbacks, which include potentially higher costs.
AWS Lambda functions became more flexible last year with additional runtimes and longer execution times, but these updates need to be used carefully.
AWS tripled the maximum execution time, or timeout, for Lambda functions -- from five to 15 minutes. You can continue to use the service to power fast APIs and other lightweight tasks, but these extended timeouts now enable Lambda to run heavier tasks, such as data processing and analysis, as well as batch compute jobs.
One of AWS Lambda's appeals is that it manages the hardware for you. And with this added leeway on timeouts, it's even easier to quickly develop longer-running application components and deploy them to the cloud. You don't need to launch EC2 instances, so your software development cycle becomes more agile. And since most applications, sooner or later, need to handle long-running tasks, this will be a welcome upgrade for many Lambda users.
Long-running asynchronous processes often consist of multiple tasks, which need to be orchestrated and also require state management, error handling and retry mechanisms. Developers that want to take advantage of these longer timeouts should consider AWS Step Functions. While Step Functions can integrate with any HTTPS endpoint, it easily integrates with AWS Lambda. All you have to do is configure your Step Functions tasks with a reference to an existing Lambda function. This will enable you to operate a 100% serverless workflow.
Migrate existing workloads to Lambda
In addition to longer Lambda timeouts, AWS also added Lambda custom runtimes. You're no longer limited by the default Lambda runtimes, such as Python, NodeJS, Go, Java, Ruby and .NET. If you manage long-running tasks in your data center or on an EC2 instance, you can now use AWS Lambda to migrate those workloads to a serverless architecture, regardless of the workload's programming language.
Custom runtimes also make Lambda more flexible by significantly expanding the number of applications that can run on Lambda. You can now run application components that are written in any language supported by Amazon Linux, binary executables compiled in Amazon Linux -- e.g., PHP, R, Rust, etc. -- or even shell scripts.
Longer timeouts don't mean more processing power
Since longer timeouts tend to attract heavier jobs, it's important to know AWS Lambda service limits, especially those that can't be increased. AWS Lambda jobs still can't exceed 3,008 MB of memory, 512 MB of local storage, 1,024 file descriptors and 1,024 processes or threads.
In some cases, you can design and optimize your implementation to avoid those limits. For example, you can reduce the scope of a particular function or use tools such as AWS X-Ray to find application bottlenecks. However, there will still be cases when Lambda won't be able to accommodate your application's needs. For example, applications that process large media files or a high number of concurrent files might exceed Lambda limits. Identify those limits as early as possible to avoid issues in production.
Lambda lacks hardware optimization
Longer jobs tend to attract workloads with specific hardware requirements. For example, memory- and compute-intensive jobs require high disk I/O or high network throughput.
With Lambda, you only choose the memory allocation. This is typically fine, but it becomes a problem if you run heavy jobs that need some hardware optimization. On the other hand, EC2 offers more than 20 different instance types, each one designed for specific resource consumption requirements.
The ability to choose the optimal hardware is why there are many use cases where EC2 is still a better option.
AWS Lambda just got more expensive -- if used incorrectly
AWS Lambda is not cheap at scale. The free tier and low-volume scenarios can be useful, but once your execution volume goes up, cost starts to become significant.
There are scenarios where Lambda is cheaper than EC2. But, as you can see from the example above, 10 m5.large instances could process 100 long-running executions in an hour cheaper than Lambda functions that utilize these new longer timeouts. And EC2 deployments get even cheaper when you go with Reserved instead of On-Demand instances.
The final pricing depends on the nature and volume of your workload. However, the higher your Lambda usage for long-running tasks is, the more likely EC2 will be a more cost-effective option. If organizations use these longer AWS Lambda timeouts correctly, they'll have more flexibility to build and launch applications. But even with longer timeouts and custom runtimes, AWS Lambda hasn't completely replaced EC2 and traditional architecture.
