Check out the episode on:

We therefore ran a controlled experiment, pitting a team of AI-assisted software engineers and quality analysts against a ‘human-powered’ group who were only allowed to use their own brains.

Here’s what we did and what we discovered…

How we ran the experiment

In all, a total of 52 of our people took part in our experiment, over a series of two hackathons.

We split them into two groups:

• One assisted by AI tools, with 30 engineers and 8 automation quality analysts (QAs)
• A control group that was purely human-powered, with 12 engineers and 2 automation QAs – allowing us to baseline the potential gains of using AI

We gave each group three hours to perform the same task.

For the engineers, the task was:

• Develop a .NET Web API to dynamically process mathematical expressions
• Implement a custom PEMDAS-based algorithm for expression evaluation, without using any third-party libraries
• Write unit tests to validate the functionality of the solution
• Test the implementation using five provided edge cases

And for the automation QAs:

• Write three automated functional test scripts for specified scenarios
• Test these functions using the provided test web shop application

The tools used by the AI-assisted team were:

• GitHub Copilot
• Cursor IDE (using ChatGPT)
• Qodo (was Codium)
• Tabnine

The results

We expected the results of this experiment to be positive in favour of the AI team, but we were still amazed by the difference the AI tools made in terms of enhancing speed and quality.

Here are the overall outcomes we recorded… 

On average, when compared to our human-powered team, our AI-assisted engineers were able to:

• Complete the coding nearly 2x (44%) faster

• Conduct the unit tests just over 2x (51%) faster

• Cover nearly twice as many (83%) more edge cases

And when we compared our fastest human-powered engineer with our fastest AI-assisted engineer, the results were even more impressive: the AI-assisted engineer was nearly 5x (78%) faster.

We also compared a human-powered engineer with one who already had experience using AI tools (in this case, GitHub Copilot). We found that:

• For the coding, the AI-powered engineer was 4x (75%) faster

• For the unit tests, the AI-powered engineer was 6x (83%) faster

This demonstrated to us that, as our team of engineers become more experienced with AI tools, our productivity gains will increase even further.

For the QAs, we also saw a significant improvement in the times it took the AI-assisted analysts: the average overall time was just over 2x (54%) faster with AI.

And, as with the engineers, we compared the two fastest times, and found that the first AI-assisted QA to complete the task was 9x (89%) faster.

A comparison of AI tools

We also aimed to make some comparisons between the four different AI tools that we used in the experiment, in particular with regard to user experience, productivity gains, and security and IP protection.

For user experience: GitHub Copilot came out on top. Our developers rated it as a robust and mature tool, suited for .NET application development. It offered consistent suggestions and responses as well as strong context management. Cursor and Codium came in joint second place.

For productivity gains: Cursor came out on top, allowing our team to be 3.2x (69%) faster than human-only developers when it came to completing the full task. GitHub Copilot was in second place, making the team 2.7x (62%) faster.

For security and IP protection , we found the following:

• GitHub Copilot transmits code snippets from the integrated development environment (IDE) to GitHub in real time to generate relevant suggestions. Once the suggestion is created, both the prompt and the suggested code snippet are immediately discarded—but note that this is only the case for the Business and Enterprise licence options.

• Cursor provides a Privacy Mode that can be activated during the onboarding process, ensuring that no code is stored on their servers or by their sub-processors.

• Qodo: Paid licence user data is not used to train its AI models. The data is deleted from their storage after 48 hours. Also,
they provide an option for a zero-retention policy, where data is removed immediately if users specifically request.

All three tools are certified for SOC2 compliance.

(Note that we didn’t assess Tabnine as we felt the model wasn’t mature enough and its users struggled to complete the task.)

Conclusion and our next steps

Our experiment made it clear that AI could offer us some huge benefits in productivity and quality. In every aspect of the tests we conducted, from coding to unit tests to automated test script production, there was a clear time saving – in most cases very significant. It will also enable us to improve the quality of our outputs, as AI-generated code was able to give us broad edge case test coverage.

Furthermore, we expect our efficiency gains to improve further as it is clear that development speed increases with experience when it comes to using AI tools. We found that just a short amount of training significantly accelerates outputs.

As for the future… Our product owner colleagues also ran an experiment to understand how AI can accelerate and improve the product discovery process. We are now looking at how we can use their AI-generated requirements as prompts to build applications – ultimately with the possibility of using AI-assisted processes from an initial description of requirements right through to final outputs.

Meanwhile, right now, we’re already starting to reap the benefits of AI-assisted development with some of our clients, delivering even greater value for them.

If you’d like to find out more about how AI-assisted software development can benefit your business, get in touch now.

Aleksandar Karavasilev, CTO at Damilah

More than anything, companies should be considering how they can achieve a better balance in their workforce. UK-based businesses will always need to have employees in this country, especially those who are in customer-facing roles. However, when it comes to other positions, such as software engineers, it’s even more important to take a carefully considered view on where to base them.

Near-shoring – or, partner-shoring, where a client and near-shore partner work closely together as a single team to achieve common goals – is a solution that can not only help to keep costs down, but also reduce your risks and increase flexibility when it comes to resourcing.

It’s not just about the money…

One of the most obvious and immediate benefits of partner-shoring is financial, which can be a huge upside, especially when a lot of businesses are just beginning to emerge from some challenging years.

What’s more, partner-shoring can offer a greater degree of certainty, where your labour costs are fully known. Any company that has just hired a lot of UK-based employees will now be reeling from the shock of the Budget, as few anticipated the NI hike, or for it to be quite so extreme. And, of course, there’s no certainty that the costs to employers will stop there – there may well be further increases in taxation to come.

By partner-shoring, it’s possible to agree costs in advance and only pay for the work that is produced for you – so, for example, not having to cover people’s sick leave or parental leave which can often be an unknown quantity.

Having clarity on costs in this way makes it a lot easier for companies to plan ahead, particularly in these uncertain times when a wide range of external factors beyond mere NI rises – such as volatile market conditions – can create significant challenges for businesses.

Added to this is the enormous benefit of flexibility. The ability to scale up and down relatively painlessly is a valuable attribute for many businesses, and having a near-shore partner allows you to do this seamlessly, while the responsibility of employee protections remain the concern of the business you’re partnering with.

Acting responsibly for maximum value

That said, it’s important to emphasise that outsourcing some of your company’s labour requirements shouldn’t mean acting irresponsibly or turning to an unethical supplier that exploits its workforce.

In the outsourcing sector, there are undoubtedly bad actors in relatively deregulated markets who will underpay their employees and offer them few, if any, benefits in order to keep costs as low as possible.

In my experience, there is no value in this for anyone involved. Companies that do not invest in their workforce will earn very little loyalty from their employees. This can lead to not only lower productivity and poor outputs, but also high attrition rates that result in large amounts of key knowledge being lost on a regular basis.

On the other hand, near-shore firms that invest in their people and constantly work hard to ensure they have good employee experiences will always deliver fewer risks and greater benefits – and therefore greater value – to their clients. And that’s exactly what we believe in, here at Damilah. We invest heavily in training to ensure we enable our people to reach their full potential, as well as offering an attractive benefits package. By doing this, our employees pay us back many, many times – and our clients also reap the rewards of this.

In particular, we are proud that our annual staff turnover rate is just 2%. This saves on the expense of recruitment and onboarding (thus helping to keep costs lower for our clients), ensures that essential knowledge and experience remains within the company and is there for our clients’ benefit, and increases the quality, consistency and speed of our outputs.

Look before you leap

One final point to make is that, although the severity of the measures regarding NI may have come as a shock to many, you should avoid making a knee-jerk reaction.

It may be tempting, for example, to immediately cut your UK workforce or instantly put a freeze on recruitment, and transfer as many roles as possible overseas. While this may ultimately be the wisest course of action for certain businesses, it’s important to take a considered approach to finding the optimum balance in your workforce, rather than immediately jumping to the first solution that springs to mind. It’s also worth taking the time to find the right partner to work with.

Here, the benefit of experience and expertise can really pay dividends and help you to deliver the value for money and flexibility your company requires.

At Damilah, we have decades of experience that we are happy to share, as well as a team of highly skilled and motivated near-shore employees who partner effectively with our clients to deliver outstanding outcomes.

To find out more about how you can minimise the impacts of the Budget on your business, get in touch now to discuss our range of services.

Iain Bishop, founder and CEO, Damilah

Sport gives us some of the clearest examples of this challenge. Assemble a team of individually brilliant players, in any sport, and you will not automatically get a brilliant team. History is full of squads with world-class talent that underperformed, and leaner, less decorated teams that consistently punched above their weight. The difference is rarely ability. It is how well the team is managed, motivated and united behind a shared goal.

The same is true in software development. You can’t produce great software without great people. But you also need to ensure you create the right conditions for those people to succeed.

We therefore believe it’s always worth the time, effort and expenditure of investing in all our people. By showing we care, and helping them to reach their potential, they will always go the extra mile and produce better quality work for our customers. So here are some of the ways we enable this at Damilah:

Maintaining a positive culture

We strive to create a culture that is open and supportive, where everyone is clear on what success looks like and how they can find the right path towards it.

Our line managers are ultimately responsible for an employee’s growth and development, but it’s not limited to them. For example, someone might need support for a particular skill that is not within the line manager’s competence – in which case, we’ll ensure they are connected to the right person in the business to help them develop in the desired way.

In other words, we’re constantly supporting each other and growing together.

What’s more, it’s important to us that our people are prepared to take risks in order to develop and innovate. If they are afraid to, they will usually struggle to advance. Of course, that doesn’t mean they’ll take silly or unnecessary risks – but, rather, in a measured way with the right support.

Software and technology are always changing and we constantly need to try new things to stay ahead of the game. We’ll usually do that in close consultation with clients, and often ‘spike’ technology – that is, test something in a controlled environment. If it works, then great: we can roll it out. And if it doesn’t, we’ll learn from this and move on.

We’re doing exactly that with one of our clients. This approach allows the client to be reassured that what we’re suggesting will work, while also allowing everyone to feel in control. What’s more, we all learn together in an environment built on mutual trust.

Investing in training and development

I’ve worked in businesses as a CTO where the first thing to go when budgets are tight is training. But that’s always a false economy that ultimately impacts on the quality of the work delivered and the success of the business. So we will always continue to invest in training, because we know we’ll get a return on it, tenfold or more.

Therefore, every employee has a development plan, and that can include both technical skills and the ‘softer’ skills, such as communication or presentation. Doing this, we believe, delivers the obvious short-term benefit of upskilling our team – but also the long-term benefit of an increasingly knowledgeable, committed workforce that chooses to stay with us.

Committing to innovation

With technology changing as fast as it does, it is important for us to stay close to new developments and make space for our people to explore them. We do this by providing opportunities to all our people to initiate or join specific innovation projects, which allow them to invest time in trying out new technologies that enhance our capabilities and ensure their skills remain at the forefront of the industry.

Celebrating success

There is no greater motivator than success – and it can be addictive. We therefore ensure we celebrate those successes at every opportunity, for example by having annual awards for both individuals and teams.

It’s important that these are not subjective. Instead, they’re based on performance and also the way in which our people uphold and emphasise the values of our business. We find that’s a really powerful way of encouraging further success, which of course will always lead to additional benefits for our clients.

Working as one team with our customers

We believe strongly in the value of teamwork, and our commitment to this extends to our customers. We aim to avoid the traditional customer-supplier relationship, opting for collaborative, transparent partnerships from the outset.

We encourage customers to feel as if we’re all part of the same organisation, working towards the same goals. Quite simply, that’s our culture – and the culture we encourage from everyone we work with, both internally and externally.

If you’d like to find out more about any of the points raised in this article, and how we can deliver great software for your business, get in touch now.

Iain Bishop, founder and CEO, Damilah

Delivering great software on time for our customers comes down to effectively managing four key elements: scope, risk, quality, and talented people. In this series of articles, which accompany our Tech Talks video, we take a closer look at each one individually.

Here, we tackle the essential topic of ensuring quality…

It’s easy to fall into a trap when it comes to managing the quality of software: an over-reliance on quality assurance (QA) at the end of the process.

While QA is vital, we strongly believe that quality is pervasive, and there needs to be a constant focus on this throughout a project. That also means always paying attention to a customer’s or user’s requirements. After all, there’s no point in building the best product if it doesn’t do what’s needed.

Because we always take this approach, we believe that quality is everyone’s responsibility all the time – from the developers writing the code, to the project managers, and even the customers we’re working with.

Doing this is not easy, and it requires a high level of commitment. But by operating in this way, as a single team, we can deliver outstanding outcomes together. Here are some of the most important considerations we always keep in mind when it comes to ensuring we create great software.

Understanding from the outset what ‘good’ looks like

Quality starts at the very beginning. That means the first discussion with the main stakeholders and really understanding their requirements. We then need to ensure the requirements are described in a way that’s testable, so we can validate whether the finished product meets the correct criteria.

We’ll then work with our development teams to check the written requirements are achievable and can be implemented with the right level of quality. Only once everyone is satisfied with this will we be ready to start work on a product.

Managing time pressures

It’s rare to have a project without some kind of time pressure – and of course that can compromise quality if not managed properly. So, as well as defining ‘Ready’ before we start a piece of work, we also create an agreed definition of ‘Done’. This will include the types of tests and reviews which must be completed before a feature is considered finished.

But, of course, unforeseen problems or scope changes do happen during a project, meaning that things can take longer than expected or need to be reprioritised. When this happens, it’s important to be open and transparent, and also to have courage – for example to explain to a key stakeholder that a change in the delivery schedule might be necessary.

I’ve seen organisations that have a very top-down way of working where people are too afraid to give bad news to their superiors, so they’ll do whatever is expedient to avoid it. Unfortunately, though, the upshot is usually that the management will eventually discover that the delivered product is not good enough – and they’ll wonder why. The answer, of course, is because of the culture in their business. Therefore, transparency and honesty are vital. After all, everyone is ultimately working to achieve the same goals.

Rigorous testing – after every small step

Although, as we’ve stressed, quality needs to be baked in right through the process, effective testing is still crucial. And it’s important to test early and regularly, otherwise a nasty shock may arise at the end – for example, realising an entire project is built on code that doesn’t work properly.

We use agile processes, breaking a project down into sprints that usually last two weeks. At the end of each sprint, we’ll carry out tests to ensure the software is fulfilling its requirements, working effectively, and conforms to the highest security standards.

We employ tooling to continuously conduct vulnerability testing and ensure software complies with OWASP Top 10 security standards. We put these tools into the build pipeline to ensure they are always running and developers know immediately when code is non-conformant.

We also use static code analysis tools to ensure the code we produce conforms to quality, reliability and maintenance standards.

And while our most experienced, senior developers are involved in these processes, again we also look to automate as much of it as possible. That way, a test can be rerun on a frequent basis – as software will always change during an agile development process – with no backlog building up.

Scalability and future-proofing

Scaling needs to be considered from the get-go – but there’s always a balance to be found. It’s easy to end up over-engineering a product so it can scale when that may not be needed for several years, if at all. But ignoring scalability can lead to its own problems if it turns out to be necessary.

It’s therefore essential for us to truly understand our customer’s needs so we can ensure the right level of scalability is built in. We’re also finding that public cloud gives us the option of scaling a product easily and rapidly, if required.

In addition, it’s crucial to consider the future maintainability of a solution – and this is often more important than the initial cycle of development. If quality is compromised during the build phase, maybe for cost reasons, it can quickly become a false economy as maintenance of poor software can be expensive and greatly increase the total expenditure.

In our opinion, automation is definitely the way to go when it comes to maintenance. But it’s incredibly hard to automate if the product hasn’t been built to high standards in the first instance – which further emphasises the need for stringent quality control during the initial build phase.

If you’d like to find out more about any of the points raised in this article, and how we can deliver great software for your business, get in touch now.

Iain Bishop, founder and CEO, Damilah