At Unai, we’re always on the lookout for better developer tools – and we work to keep an open mind about what’s best. Just as when we’re reviewing each other’s work, the whole team gets involved.
We have a particular love of open-source tools. They’re generally enjoyable to work with, they’re built on a knowledge-sharing ethos, and our developers can take everything they learn about them and go on to use that expertise elsewhere.
There are two open-source tools that we’ve become particularly adept at using: Terraform and Apache Airflow. Both make development faster, easier and more robust for our clients, and mean that our developers can focus their expertise where it matters most.
Terraform – building out complex architecture at the click of a button
Terraform has a name that sounds like it’s straight from outer space and in many ways this open-source tool is like a mothership – a central hub that coordinates the configuration of provisioned cloud services.
Terraform not only manages infrastructure across multiple cloud providers, it can also codify infrastructure and make it reproducible. This makes building complex architecture much, much easier.
For example, with Terraform we can write code that determines how the website should work, how the database should be created, how they talk to one another, and how information should be deployed. Once we’ve got the core code in place, we can click a button and Terraform will communicate with the cloud provider and create all of the required services needed to run the website.
That’s quite labour intensive if you just want to create this once and never again. But when you have complex architecture - say a hub that communicates with multiple accounts - writing code into Terraform means you can create multiple spokes on the wheel almost instantly. Crucially, those automatically created spokes will also be secure and compliant, and we can strengthen this even further through CICD integration.
How Terraform made a difference for us
One of our clients who deals with patient information has a very nuanced architecture. Their users access a central hub account, which in turn needs to talk to many different AWS accounts.
Each account needs to be secure, accessible only to selected organisations, and it needs to expose anonymous sets of data, based on patient information, without allowing organisations access to the actual patient information.
To do this, all the accounts needed a very similar infrastructure. They all needed APIs to send down the information, and a pipeline to turn the patient information into standardised, anonymous data.
Configuring this manually would take far too long – and we’d be more prone to making mistakes as we tried to recreate each pipeline from memory, but using Terraform, as soon as a new account comes on board the existing infrastructure can be copied over; and because of the way the code is structured, our client’s hub can automatically talk to the new account.
Because so much of the process is automated, it’s faster to get a new engineer working on the infrastructure, and it’s much faster for us to onboard new accounts for our client. Plus we lose none of the watertight security and compliance measures we have in place.
Apache Airflow – problem solving with ease
Apache Airflow is an ETL (extract, transform, load) platform that enables us to coordinate tasks and programmatically manage workflows.
It’s an excellent solution when there’s a complex series of tasks that need to happen in a particular order, or each step is convoluted, or when data needs to fan out to several destinations.
There are a number of ETL tools that work in a similar way but we’ve found that Airflow goes further than the others. It’s resilient and, when a task fails, it will automatically retry until it’s either succeeded or until it’s failed enough times to tell you exactly what’s gone wrong.
Airflow enables us to see where a problem is in the ETL process and pinpoint the code that’s causing the issue. Normally we’d be forced to figure it all out ourselves but Airflow bakes its own problem solving into the software.
This is a huge upgrade for us and our clients who need to move data into a database on a regular basis.
How Apache Airflow made a difference for us
For one of our financial services clients, this has proven to be highly effective. On an hourly basis, we move data from a third party API into our client’s data warehouse, supplementing the data with credit scoring and other attributes. Because the transfer is so regular, there are many opportunities for bugs and problems to arise.
With most ETL tools, this would result in an ever-throbbing headache. Every time things we didn’t recognise came down the pipeline, or every time data wasn’t in the right format, our developers would be feeling the strain.
Airflow, meanwhile, shakes out the tablecloth and lets the problems tumble directly into our developers hands. They can fix the issues quickly and simply, which keeps both our developers and our clients in a good mood.
Finding the next tool for Unai
The sharing of expertise is a big deal at Unai and we have a lot of respect for the work that other developer communities put into creating open-source software like Terraform and Apache Airflow for everyone to use.
Right now, we have a technology radar full of the research that our developers have undertaken, analysing the reams of software out there for the next addition to the Unai toolkit. Anything that makes us work smarter is a welcome aboard.
That’s not to say technology is the centre of everything at Unai - In fact, we believe relationships should be at the heart of data science. Collaboration, creativity, and a desire to keep innovating internally keeps us at the forefront of our field.
But software tools make our work easier and more efficient - and since our developers’ time and expertise is precious, we want them to be putting their ingenuity where it counts.
To learn more about how Unai works, pull back the curtain to see the extraordinary results of the first Unai hackathon, or drop us a line.