Testing and deployment automation injects efficiency, boosts morale for DJSB
The Federal Government is committed to digitally transforming services across the Commonwealth to make them simple, clear and fast.
Simple, clear and fast are also the watchwords for systems testing and deployment at the Department of Jobs and Small Business (Jobs).
While much focus is placed on developing new services for citizens, the challenge of getting those systems maintained and enhanced increases exponentially. But it is essential for any enterprise bent on continuous improvement and efficient operation.
The Department is responsible for national policies and programmes that help people find and keep jobs, and work in safe, fair and productive workplaces.
Most critically, the Department is also responsible for managing jobactive, the largest Government programme to get more Australians into work. It connects more than 700,000 job seekers with employers and is delivered by a network of jobactive providers in over 1,700 locations across Australia. It is a critically important role, but also exists in an area that does not stand still – nor can the information systems that support it.
Explicit in the Department’s vision for innovation is a commitment to ‘prototyping, trialling and learning quickly; learning and growing from mistakes; and, mitigating risks to progress ideas.’ Turning that commitment to reality falls in part to Vlado Cvetkoski, the Director of Release Environments and System Enablement.
Backed by the Department’s senior executive and working closely with Team Leader Joel Taplin, Cvetkoski’s efforts ensure that new software is properly tested, as well as rapidly and reliably deployed, and then continually enhanced and improved.
While supporting day to day operations of the Department, Cvetkoski also keeps a watch eye on the future and in the last 18 months has been working with Microsoft on an initiative to support continuous integration and continuous delivery. His team has also been exploring how chatbots and cognitive services could help the Department in its mission.
The initial phase of the programme involved the Department working with Microsoft to benchmark the Department, and create a roadmap to support and streamline innovation.
In terms of that roadmap’s impact, the transition to automated testing and deployment using Microsoft Team Foundation Server (TFS) is proving a game changer for the Department. Microsoft field engineering resources worked closely with the team to ensure a smooth transition.
Taplin explains that in the past complex manual regression testing would take 40 minutes. That has now been automated using TFS and takes just seven minutes. The time to perform simple tests has similarly been sliced from five to one minute, while moderate complexity testing takes two instead of 15 minutes.
Importantly says Cvetkoski, it is also a process that is repeatable and reusable.
His team is also required to manage systems health checks – generally undertaken over a weekend to ensure minimal impact to end users. Automation using TFS means the test now takes seven minutes, and no longer needs one of his team to give up a chunk of their weekend.
Streamlined deployment
Post-test the deployment process has also been streamlined. In the past, says Cvetkoski; “We would write a set of instructions for the shared services – a separate group – and they would do deployments on our behalf manually. It was quite a painful process, we would do all this documentation only to have something go wrong because of human error.”
Now, for .net code, deployment has been largely automated using Release Manager, with the process auditable, traceable and with reduced human error. There’s no need to wait on someone from another team to manage the release, which is reducing bottlenecks and empowering the team.
Prior to the automation of database deployments and release consolidation it could take two or three days to shepherd software from pre-production through production and onto release with several manual touchpoints. The automated process takes 30 minutes.
This increased velocity without compromise of quality is essential as the Department transitions to a more agile environment, which can result in several new code releases each day. The release function needs to keep pace.
Particular benefit has derived from automated regression testing which can be a stumbling block for agile, if it is not automated. No matter how effective a sprint – if the outcome of that sprint can’t easily be regression tested delay is inevitable.
According to Cvetkoski; “The regression testing had started taking up a lot of the two-week iteration – it was not hard to convince the teams of the need for automated testing.”
Jobs is also on a journey to automate more of its database changes which were previously entirely manual. Matthew Groves, Senior Database Administrator has steered much of the effort automating SQL into TFS.
Benefits are being measured in terms of time saved, errors avoided and a greater opportunity to innovate.
The journey to automation is also shifting the dial on the workplace experience for the team and will accelerate as it transitions off XAML.
Taplin explains that in the past release management involved creating 300 templates and manually pushing changes through each one. Now a script is written to automate the process.
“It took half a day and build a script. Previously we’d have to go into each of the 300 templates manually – that was mind numbingly painful and boring. Now it’s a complex but interesting task that can be done much faster”- Team Leader Joel Taplin
The application programming interface to TFS has ensured much greater flexibility, which is important to a team managing hundreds of workflows. Now says Taplin; “If we need to change it’s a quick ten-minute job to change the script.”
Jobs is also exploring Microsoft Azure based test and development opportunities, particularly as a way to test third party solutions in a secure and controlled environment through modules such as DevTest Labs.
Cvetkoski says that the increasing automation reduces his team’s reliance on other groups in the Department. “In the past we had to wait for a production DBA to be available – now the deployment happens when we press the button.
“The whole process is automated, repeatable, reusable. It executes in the same manner, it eliminates human error and when we do run a deployment there is the same solution in all environments.”
He explains that the automation has freed his 2-strong team from being largely reactive, handling often manual and repetitive work. Now the team is freed for more proactive value adding and rewarding work.
“Staff morale is going up – people feel they are contributing, innovating and they see the benefits day in day out.”
