Between the first and second technical support lines
How often have you met application admins who like to deal with incident resolution?
Moreover, a significant flow of incidents to the second line of support is the so-called business incidents, that is, incidents that are either associated with a violation of the logic of the business process in the service, or with incorrect actions by the user.
We were able to remove this functionality from the second line as much as possible, transferring it to a separate team assembled from the employees of the first technical support line.
We will tell you how we did this and what difficulties we encountered in this article.
Traditionally, the second line of technical support was engaged in solving incidents on systems, those that provide maintenance of the service itself, perform product configuration settings, test environments, system monitoring, install patches, participate in on-call shifts, etc. And the solution of incidents in these teams was often accompanied by a “residual principle” (unless this is of course an incident with critical priority), however, in most cases they fit into the assigned SLAs.
Realizing that many of the incidents (and on the front of the systems - behind each) could potentially be the problem of the Bank's Client, we decided to minimize the execution time of these requests, but did not start by increasing the number of support teams, but by analyzing each of the stages of this process.
The traditional support process at the Bank was built according to the classical scheme and looked as follows
Fig. 1 Scheme of the organization of the support process (was)
The request passed through the first, second and third lines of support, was distributed between them in the following proportions: 15:75:10.
The first line - Service Desk - receiving, processing, routing calls using the following communication channels:
The first line also deals with resolving incidents remotely at users' workstations, providing access to the Bank’s information systems. I wrote about the work of this unit in a separate article - link here
The second line - application software maintenance teams + infrastructure + DBA + ...,
The third line is development teams.
Since banking systems are integrated among themselves, the occurrence of an incident in one of the systems is often the result of an error that has occurred in another system. And errors on the “back systems” primarily appear on the “front”, having a direct impact on users.
Identification of the root cause of occurrence in this case requires the participation of several escort teams, and, as a result, a series of routings.
Taking a sample of incidents over several months and analyzing their life cycle, we found out that they spend most of their lives in line, waiting for an analysis by an employee of a second-line support team (sometimes up to 80%), periodically, with reassignment between teams and a long correspondence in the body of the request.
In the process of analyzing and analyzing these incidents, we found out that to solve integration errors, colleagues need information from adjacent integration systems (logs, influence analysis, etc.), for which they turn to each other, routing incidents.
In order to minimize incident resolution time, as a pilot project, we decided to create a cross-functional integrated team of first and second line employees for resolving incidents in the Bank’s front systems - Internet banking, mobile banking, message sending services (sms and push) + main front - Bank office system.
This command is intermediate between the first and second lines - a one and a half line of support, which we called “Frontline”.
Already at the stage of formation of this team, we encountered certain difficulties, including from the side of the second line managers, so the transfer of even one employee to this project from each of the systems meant a decrease in the current team’s capital. And the second-line employees themselves, who were supposed to participate in the pilot, were not eager to dive headlong into solving incidents.
Through iterative negotiations, we were able to agree that the main task of the second-line participants in this project is to train first-line employees, jointly create a common knowledge base, build internal interaction processes, provide the necessary accesses, and after that, gradually return back to their units.
The location of the Frontline location was the Bank's IT Center in Obninsk.
The first line-up of the one and a half support line was as follows:
The main focus of the team was focused on 3 indicators:
fig. 2 Frontline Incident Handling Process
The next difficulty that we encountered at the beginning of the pilot was the lack of a team in our initial understanding, if any.
Despite the fact that Frontline employees were nearby in the same room, there were no attempts to switch to cross-functionality, exchange of experience, significant interaction inside. Each participant, as before, was focused on resolving queries through his system, as a result - someone was “littered” with incidents, someone was clearly freer.
It was decided to “change systems” within the team, for example, so that the representative of the Internet banking support no longer resolves incidents on his system, but begins to process requests for the front office system, and the representative of the front-office system support began to solve incidents on notification services, etc. d.
What for?
Made the necessary accounts, provided access to the application and database logs, and more! :)
Initially, incident resolution was reduced. It is clear that when you do not know the intricacies of the system, and you also need to solve incidents on it, this is not an easy task. But the guys began to turn to each other for help, study and update the general knowledge base, and gradually things went.
Also, every day we began to hold small stand-ups at the beginning of each working day, near the sheets of paper that we glued to the walls and with daily indicators. We discussed and finished them with a marker, rejoicing in their fulfillment or discussing the reason for the failure, if we could not achieve them.
Subsequently, of course, we replaced these sheets of paper with online dashboards.
fig. 3 Frontline efficiency dashboard
Here it is necessary to say a special “thank you” to the head of the support team for the front-office system of the Bank, who took on the leadership role and cultivated the development of the team from the inside - Alex, thank you! :)
In the first two months, the team could not cope with the set targets, the training process was ongoing, the knowledge base was updated.
Starting from the third month of the pilot project, we began to fit into the targets, and after 6 months we even began to exceed them.
Fig. 4 Frontline pilot project, first indicators
It soon became clear that the pilot successfully “took off”, and it was advisable to scale the project.
Gradually, we began to add competencies in other systems to the team, withdraw second-line employees and add employees from Service Desk.
Gradually, we switched to “T - cross-functionality”, when each participant was assigned one main system and two adjacent ones.
fig. 4 Comparative statistics for 2018 on the time of solving requests between Frontline and second-line tracking teams
In 2018, the Frontline team closed more incidents than any other escort unit at the Bank. The guys significantly exceeded the established targets, once again showing that teamwork and cross-functional competencies can achieve significant results.
For the second line of support, “Frontline” today is a reliable “shield”, which significantly reduces the flow of calls coming to them.
Fig. 5 Number and proportion of incidents resolved in Frontline (Fig. - Team1) in relation to incidents resolved on the second line for all Bank systems
Today, all Frontline participants are Service Desk employees who solve incidents on the Bank’s main front systems while meeting specified targets.
Also, “Frontline” is the next step for Service Desk employees in our career ladder, before moving to the second support line.
Moreover, a significant flow of incidents to the second line of support is the so-called business incidents, that is, incidents that are either associated with a violation of the logic of the business process in the service, or with incorrect actions by the user.
We were able to remove this functionality from the second line as much as possible, transferring it to a separate team assembled from the employees of the first technical support line.
We will tell you how we did this and what difficulties we encountered in this article.
Traditionally, the second line of technical support was engaged in solving incidents on systems, those that provide maintenance of the service itself, perform product configuration settings, test environments, system monitoring, install patches, participate in on-call shifts, etc. And the solution of incidents in these teams was often accompanied by a “residual principle” (unless this is of course an incident with critical priority), however, in most cases they fit into the assigned SLAs.
Realizing that many of the incidents (and on the front of the systems - behind each) could potentially be the problem of the Bank's Client, we decided to minimize the execution time of these requests, but did not start by increasing the number of support teams, but by analyzing each of the stages of this process.
The traditional support process at the Bank was built according to the classical scheme and looked as follows
Fig. 1 Scheme of the organization of the support process (was)
The request passed through the first, second and third lines of support, was distributed between them in the following proportions: 15:75:10.
The first line - Service Desk - receiving, processing, routing calls using the following communication channels:
- phone - 8% of the total number of calls
self-service portal - 83% of the total number of calls
email - 4% of the total number of calls
chat bot in Viber and Telegram - 5% of the total
The first line also deals with resolving incidents remotely at users' workstations, providing access to the Bank’s information systems. I wrote about the work of this unit in a separate article - link here
The second line - application software maintenance teams + infrastructure + DBA + ...,
The third line is development teams.
Since banking systems are integrated among themselves, the occurrence of an incident in one of the systems is often the result of an error that has occurred in another system. And errors on the “back systems” primarily appear on the “front”, having a direct impact on users.
Identification of the root cause of occurrence in this case requires the participation of several escort teams, and, as a result, a series of routings.
Taking a sample of incidents over several months and analyzing their life cycle, we found out that they spend most of their lives in line, waiting for an analysis by an employee of a second-line support team (sometimes up to 80%), periodically, with reassignment between teams and a long correspondence in the body of the request.
In the process of analyzing and analyzing these incidents, we found out that to solve integration errors, colleagues need information from adjacent integration systems (logs, influence analysis, etc.), for which they turn to each other, routing incidents.
In order to minimize incident resolution time, as a pilot project, we decided to create a cross-functional integrated team of first and second line employees for resolving incidents in the Bank’s front systems - Internet banking, mobile banking, message sending services (sms and push) + main front - Bank office system.
This command is intermediate between the first and second lines - a one and a half line of support, which we called “Frontline”.
Already at the stage of formation of this team, we encountered certain difficulties, including from the side of the second line managers, so the transfer of even one employee to this project from each of the systems meant a decrease in the current team’s capital. And the second-line employees themselves, who were supposed to participate in the pilot, were not eager to dive headlong into solving incidents.
Through iterative negotiations, we were able to agree that the main task of the second-line participants in this project is to train first-line employees, jointly create a common knowledge base, build internal interaction processes, provide the necessary accesses, and after that, gradually return back to their units.
The location of the Frontline location was the Bank's IT Center in Obninsk.
The first line-up of the one and a half support line was as follows:
- two first line support staff
- two employees of the front office system support group
- two employees of the support group of Internet banking and mobile banking
- one employee of the customer information services support group (sms & push)
The main focus of the team was focused on 3 indicators:
- SPEED - 70% of requests received by the team must be resolved in no more than 8 working hours
- QUANTITY - the team can route no more than 20% of requests to the second support line
- QUALITY - the proportion of incidents rediscovered by users should not exceed 3%
fig. 2 Frontline Incident Handling Process
The next difficulty that we encountered at the beginning of the pilot was the lack of a team in our initial understanding, if any.
Despite the fact that Frontline employees were nearby in the same room, there were no attempts to switch to cross-functionality, exchange of experience, significant interaction inside. Each participant, as before, was focused on resolving queries through his system, as a result - someone was “littered” with incidents, someone was clearly freer.
It was decided to “change systems” within the team, for example, so that the representative of the Internet banking support no longer resolves incidents on his system, but begins to process requests for the front office system, and the representative of the front-office system support began to solve incidents on notification services, etc. d.
What for?
- Try to come to universality so that employees can switch between incidents of different systems, thereby distributing evenly the load on all participants;
- Provide children with sufficient knowledge of related systems that will help them quickly identify the cause of the integration error;
- Establish communication within the team;
Made the necessary accounts, provided access to the application and database logs, and more! :)
Initially, incident resolution was reduced. It is clear that when you do not know the intricacies of the system, and you also need to solve incidents on it, this is not an easy task. But the guys began to turn to each other for help, study and update the general knowledge base, and gradually things went.
Also, every day we began to hold small stand-ups at the beginning of each working day, near the sheets of paper that we glued to the walls and with daily indicators. We discussed and finished them with a marker, rejoicing in their fulfillment or discussing the reason for the failure, if we could not achieve them.
Subsequently, of course, we replaced these sheets of paper with online dashboards.
fig. 3 Frontline efficiency dashboard
Here it is necessary to say a special “thank you” to the head of the support team for the front-office system of the Bank, who took on the leadership role and cultivated the development of the team from the inside - Alex, thank you! :)
In the first two months, the team could not cope with the set targets, the training process was ongoing, the knowledge base was updated.
Starting from the third month of the pilot project, we began to fit into the targets, and after 6 months we even began to exceed them.
Fig. 4 Frontline pilot project, first indicators
It soon became clear that the pilot successfully “took off”, and it was advisable to scale the project.
Gradually, we began to add competencies in other systems to the team, withdraw second-line employees and add employees from Service Desk.
Gradually, we switched to “T - cross-functionality”, when each participant was assigned one main system and two adjacent ones.
fig. 4 Comparative statistics for 2018 on the time of solving requests between Frontline and second-line tracking teams
In 2018, the Frontline team closed more incidents than any other escort unit at the Bank. The guys significantly exceeded the established targets, once again showing that teamwork and cross-functional competencies can achieve significant results.
For the second line of support, “Frontline” today is a reliable “shield”, which significantly reduces the flow of calls coming to them.
Fig. 5 Number and proportion of incidents resolved in Frontline (Fig. - Team1) in relation to incidents resolved on the second line for all Bank systems
Today, all Frontline participants are Service Desk employees who solve incidents on the Bank’s main front systems while meeting specified targets.
Also, “Frontline” is the next step for Service Desk employees in our career ladder, before moving to the second support line.
All Articles