STEM Intensive Learning Approach
In the world of engineering education, there are many excellent courses, but often the curriculum built on them has one serious drawback - the lack of good connectivity between different topics. One can object: how so?
When a training program is being formed, prerequisites and a clear order in which disciplines should be studied are indicated for each course. For example, in order to assemble and program a primitive mobile robot, you need to know a little mechanics to create its physical structure; the basics of electricity at the level of Ohm / Kirchhoff laws, the representation of digital and analog signals; operations with vectors and matrices in order to describe the coordinate systems and movement of the robot in space; programming basics at the level of data presentation, simple algorithms and control transfer structures, etc. to describe the behavior.
Is it all in university courses? Of course have. However, to the Ohm / Kirchhoff laws we get thermodynamics and field theory; in addition to operations with matrices and vectors, you have to deal with Jordan forms; to study polymorphism in programming - topics that are not always needed to solve a simple practical problem.
University education is extensive - the student goes on a wide front and often does not see the meaning and practical significance of the knowledge that he receives. We decided to reverse the paradigm of university education STEM (from the words Science, Technology, Engineering, Math) and create a program that relies on the coherence of knowledge, allowing for the build-up of completeness in the future, that is, it implies intensive development of subjects.
The study of a new subject area can be compared with the study of a certain area. And there are two options: either we have before us a very detailed map with a huge amount of details that need to be studied (and this takes a lot of time) in order to understand where the main landmarks are and how they relate to each other; or you can use the primitive plan, which shows only the main points and their relative positions - such a map is enough to immediately begin to move in the right direction, clarifying details along the way.
We tested the STEM intensive learning approach at the winter school, which we conducted together with MIT students with the support of JetBrains Research.
The first part of the school program was a week of classes in the main areas, which included algebra, electrical circuits, computer architecture, Python programming and an introduction to ROS (Robot Operating System).
The directions were not chosen by chance: complementing each other, they were supposed to help students see the connection between seemingly different, at first glance, things - mathematics, electronics and programming.
Of course, the main goal was not to give many lectures, but to give students the opportunity to put the knowledge they just gained into practice themselves.
In the algebra section, students could be trained in operations with matrices and solving systems of equations that were useful in studying electrical circuits. Learning about the transistor device and the logical elements built on its basis, students could see their application in the processor device, and after learning the basics of the Python language, write a program for a real robot on it.
One of the tasks of the school was to minimize the work with simulators, where possible. Therefore, a large set of electronic circuits was prepared, which students had to assemble on a breadboard from real components and test them in practice, and Duckietown was chosen as the basis for the projects.
Duckietown is an open source project that includes small, autonomous robots called Duckiebot, and a network of roads along which they travel. Duckiebot is a wheeled platform equipped with a Raspberry Pi microcomputer and a single camera.
Based on it, we prepared a set of possible tasks, such as building a road map, finding objects and stopping next to them, and a number of others. Students could also propose their own problem and not only write a program to solve it, but also immediately launch it on a real robot.
During the lecture, teachers told the material using pre-prepared presentations. Some classes were recorded on video, so students had the opportunity to watch them at home. During the lectures, students used materials on their computers, asked questions, solved problems together and independently, sometimes at the blackboard. Based on the results of the work, the rating of each student separately in different subjects was calculated.
Consider conducting classes in each subject in more detail. The first subject was linear algebra. Students for one day studied vectors and matrices, systems of linear equations, etc. Practical tasks were built in an interactive mode: the proposed tasks were solved individually, and the teacher and other students gave comments and tips.
The second subject is electricity and simple circuits. Students studied the basics of electrodynamics: voltage, current, resistance, Ohm's law and Kirchhoff's laws. Practical tasks were partially done in the simulator or performed on the board, but more time was devoted to building real circuits, such as logic circuits, oscillatory circuits, etc.
The next topic is Computer Architecture - in a sense, a bridge connecting physics and programming. Students studied the fundamental basis, the significance of which is more theoretical than practical. As a practice, students independently designed arithmetic-logic circuits in the simulator, for points they received points.
The fourth day is the first day of programming. Python 2 was chosen as the programming language, as it is used in programming under ROS. This day was constructed as follows: teachers told the material, gave examples of solving problems, while students listened to them, sitting at computers, and repeated what the teacher wrote on the board or slide. Then the students solved similar problems on their own, subsequently the solutions were evaluated by teachers.
The fifth day was devoted to ROS: the guys got acquainted with programming robots. Throughout the school day, students sat at computers, running the program code that the teacher talked about. They were able to independently launch the basic units of ROS, and also got acquainted with the Duckietown project. At the end of this day, students were ready to begin the design part of the school - solving practical problems.
Students were offered to join in teams of three people and choose a project topic. As a result, the following projects were adopted:
1. Color calibration. Duckiebot needs to calibrate the camera when changing lighting conditions, so there is the task of automatic calibration. The problem is that color ranges are very sensitive to light. Participants implemented a utility that would highlight the desired colors on the frame (red, white and yellow) and build ranges for each of the colors in HSV format.
2. Duck Taxi. The idea of ​​this project is that Duckiebot could stop near some object, pick it up and follow a certain route. A bright yellow duck was chosen as an object.
3. Building a graph of roads. There is the task of building a graph of roads and intersections. The goal of this project is to build a road graph without providing a priori environmental data for Duckiebot, relying solely on camera data.
4. The patrol car. This project was invented by the students themselves. They offered to teach one Duckiebot, the "patrol", to persecute the other, the "intruder." For this purpose, a target recognition mechanism by the ArUco marker was used. As soon as recognition is completed, a signal of completion is sent to the “intruder”.
The aim of the Color Calibration project was to adjust the range of recognizable marking colors to the new lighting conditions. Without such adjustments, the recognition of stop lines, lane dividers and road boundaries became incorrect. Participants proposed a solution based on preprocessing layout color templates: red, yellow, and white.
Each of these colors has a predefined range of HSV or RGB values. Using this range, all areas containing suitable colors are found on the frame, and the largest is selected from them. This area is taken as the color to be remembered. Then statistical formulas are used, such as calculating the mean and standard deviation, in order to evaluate the new color range.
This range is written to the Duckiebot camera configuration files and can be used later. The described approach was applied to all three colors, eventually forming ranges for each of the markup colors.
Tests showed almost perfect recognition of the marking lines, except when a glossy tape was used as the marking materials, reflecting the light sources so strongly that at the camera’s viewing angle the marking appeared white regardless of its original color.
The Duck Taxi project involved the construction of a search algorithm for a passenger duck in the city, and then its transportation to the required point. The participants divided this task into two: detection and movement along the graph.
Students carried out the detection of the duck, making the assumption that the duck is any area on the frame that can be recognized as yellow, with a red triangle (beak) on it. As soon as such an area is detected in the next frame, the robot should approach it, and then stop for a few seconds, simulating a passenger landing.
Then, having a pre-stored road graph of the entire duckietown and the bot’s position, as well as receiving the destination as input, the participants build a path from the departure point to the arrival point using Dijkstra’s algorithm to search for paths in the graph. The output is presented as a set of commands - turns at each of the following intersections.
The goal of this project was to build a graph - a network of roads in Duckietown. The nodes of the resulting graph are intersections, and the arcs are roads. To do this, Duckiebot must explore the city and analyze its route.
During the work on the project, the idea was considered, but then the idea of ​​creating a weighted graph in which the cost of the edge is determined by the distance (time to overcome) between the intersections was rejected. The implementation of this idea turned out to be too laborious, and there would not be enough time within the framework of the school.
When Duckiebot gets to the next intersection, he chooses the road leaving the intersection on which he has not yet moved. When all roads at all intersections have been covered, a generated list of intersection adjacencies remains in the bot’s memory, which is converted into an image using the Graphviz library.
The algorithm proposed by the participants is not suitable for arbitrary Duckietown, but it worked well in a small city consisting of four intersections used within the school. The idea was to supplement each intersection with an ArUco marker containing the intersection identifier in order to track the intersection.
The working scheme of the algorithm developed by the participants is shown in the figure.
The goal of this project is to search for, prosecute, and detain an intruder bot in Duckietown. The bot patrol should move along the outer ring of the road in the city in search of a well-known intruder bot. After the intruder is detected, the patrol bot must follow the intruder and make him stop.
The work began with the search for the idea of ​​detecting a bot on the frame and recognizing the intruder in it. The team suggested equipping each bot in the city with a unique marker at the back - just like real cars have state registration numbers. For this, ArUco markers were chosen. They were previously used in duckietown, as it’s easy to work with them and they allow you to determine the orientation of the marker in space and the distance to it.
Next, it was necessary to make the patrol bot move strictly in the outer circle without stopping at intersections. By default, Duckiebot moves in a lane and stops at a stop line. Then, using road signs, determines the configuration of the intersection and makes a choice about the direction of the intersection. For each of the described steps, one of the states of the state machine of the robot is responsible. In order to get rid of stops at the intersection, the team changed the state machine so that when approaching the stop line, the bot immediately went straight to the state of passage of the intersection.
The next step was to solve the problem of stopping the intruder bot. The team made the assumption that the patrol bot can have SSH access to each of the bots in the city, that is, have some information about what authorization data and which id each bot has. Thus, after the intruder was detected, the patrol bot began to connect via SSH to the intruder bot and turned off its system.
After confirming that the shutdown command was completed, the patrol bot also stopped.
The algorithm of the patrol robot can be represented as the following scheme:
The work was organized in a format similar to Scrum: every morning students planned tasks for the current day, and in the evening reported on the work done.
On the first and final days, students prepared presentations describing the task and how to solve it. To help students follow their chosen plans, in the rooms where the projects were being worked on, there were constantly teachers from Russia and America who answered questions. Communication took place mainly in English.
Work on projects lasted one week, after which students presented their results. All prepared presentations in which they told what they had learned at this school, what the most important lessons were learned, what they liked or didn't like. After that, each team presented their project. All teams coped with the tasks.
The team that implements color calibration completed the project faster than others, so they managed to prepare documentation for their program. And the team involved in the graph of roads, even on the last day before the demonstration of the project, tried to finalize and fix their algorithms.
After school, we asked students to evaluate past activities and answer questions about how the school met their expectations, what skills they acquired, etc. All students noted that they learned how to work in a team, distribute tasks and plan their time.
Students were also asked to rate the usefulness and complexity of the courses. And here two groups of assessments formed: for some, the courses were not very difficult, others rated them as the most difficult.
This means that the school has taken the right position, remaining accessible to beginners in a particular area, but also providing materials for repetition and consolidation to experienced students. It should be noted that almost everyone noted a programming course (Python) as uncomplicated, but useful. The most difficult, according to students, was the course "Computer Architecture".
When students were asked about the shortcomings and advantages of the school, many responded that they liked the chosen teaching style, in which the teachers promptly and individually provided assistance and answered questions.
Students also noted that they liked working in the daily planning mode of their tasks and independently setting deadlines. Students noted the lack of provided knowledge, which was required when working with the bot: when connecting, understanding the basics and principles of its work as shortcomings.
Almost all students noted that the school exceeded their expectations, and this indicates the right direction of school organization. Thus, general provisions should be maintained when organizing the next school, taking into account and, if possible, eliminating the shortcomings noted by students and teachers, possibly changing the list of courses or the time of their teaching.
Authors: JetBrains Research Mobile Robot Algorithm Laboratory Team
PS Our corporate blog has a new name. Now it will be dedicated to JetBrains educational projects.
When a training program is being formed, prerequisites and a clear order in which disciplines should be studied are indicated for each course. For example, in order to assemble and program a primitive mobile robot, you need to know a little mechanics to create its physical structure; the basics of electricity at the level of Ohm / Kirchhoff laws, the representation of digital and analog signals; operations with vectors and matrices in order to describe the coordinate systems and movement of the robot in space; programming basics at the level of data presentation, simple algorithms and control transfer structures, etc. to describe the behavior.
Is it all in university courses? Of course have. However, to the Ohm / Kirchhoff laws we get thermodynamics and field theory; in addition to operations with matrices and vectors, you have to deal with Jordan forms; to study polymorphism in programming - topics that are not always needed to solve a simple practical problem.
University education is extensive - the student goes on a wide front and often does not see the meaning and practical significance of the knowledge that he receives. We decided to reverse the paradigm of university education STEM (from the words Science, Technology, Engineering, Math) and create a program that relies on the coherence of knowledge, allowing for the build-up of completeness in the future, that is, it implies intensive development of subjects.
The study of a new subject area can be compared with the study of a certain area. And there are two options: either we have before us a very detailed map with a huge amount of details that need to be studied (and this takes a lot of time) in order to understand where the main landmarks are and how they relate to each other; or you can use the primitive plan, which shows only the main points and their relative positions - such a map is enough to immediately begin to move in the right direction, clarifying details along the way.
We tested the STEM intensive learning approach at the winter school, which we conducted together with MIT students with the support of JetBrains Research.
Material preparation
The first part of the school program was a week of classes in the main areas, which included algebra, electrical circuits, computer architecture, Python programming and an introduction to ROS (Robot Operating System).
The directions were not chosen by chance: complementing each other, they were supposed to help students see the connection between seemingly different, at first glance, things - mathematics, electronics and programming.
Of course, the main goal was not to give many lectures, but to give students the opportunity to put the knowledge they just gained into practice themselves.
In the algebra section, students could be trained in operations with matrices and solving systems of equations that were useful in studying electrical circuits. Learning about the transistor device and the logical elements built on its basis, students could see their application in the processor device, and after learning the basics of the Python language, write a program for a real robot on it.
Duckietown
One of the tasks of the school was to minimize the work with simulators, where possible. Therefore, a large set of electronic circuits was prepared, which students had to assemble on a breadboard from real components and test them in practice, and Duckietown was chosen as the basis for the projects.
Duckietown is an open source project that includes small, autonomous robots called Duckiebot, and a network of roads along which they travel. Duckiebot is a wheeled platform equipped with a Raspberry Pi microcomputer and a single camera.
Based on it, we prepared a set of possible tasks, such as building a road map, finding objects and stopping next to them, and a number of others. Students could also propose their own problem and not only write a program to solve it, but also immediately launch it on a real robot.
Teaching
During the lecture, teachers told the material using pre-prepared presentations. Some classes were recorded on video, so students had the opportunity to watch them at home. During the lectures, students used materials on their computers, asked questions, solved problems together and independently, sometimes at the blackboard. Based on the results of the work, the rating of each student separately in different subjects was calculated.
Consider conducting classes in each subject in more detail. The first subject was linear algebra. Students for one day studied vectors and matrices, systems of linear equations, etc. Practical tasks were built in an interactive mode: the proposed tasks were solved individually, and the teacher and other students gave comments and tips.
The second subject is electricity and simple circuits. Students studied the basics of electrodynamics: voltage, current, resistance, Ohm's law and Kirchhoff's laws. Practical tasks were partially done in the simulator or performed on the board, but more time was devoted to building real circuits, such as logic circuits, oscillatory circuits, etc.
The next topic is Computer Architecture - in a sense, a bridge connecting physics and programming. Students studied the fundamental basis, the significance of which is more theoretical than practical. As a practice, students independently designed arithmetic-logic circuits in the simulator, for points they received points.
The fourth day is the first day of programming. Python 2 was chosen as the programming language, as it is used in programming under ROS. This day was constructed as follows: teachers told the material, gave examples of solving problems, while students listened to them, sitting at computers, and repeated what the teacher wrote on the board or slide. Then the students solved similar problems on their own, subsequently the solutions were evaluated by teachers.
The fifth day was devoted to ROS: the guys got acquainted with programming robots. Throughout the school day, students sat at computers, running the program code that the teacher talked about. They were able to independently launch the basic units of ROS, and also got acquainted with the Duckietown project. At the end of this day, students were ready to begin the design part of the school - solving practical problems.
Description of selected projects
Students were offered to join in teams of three people and choose a project topic. As a result, the following projects were adopted:
1. Color calibration. Duckiebot needs to calibrate the camera when changing lighting conditions, so there is the task of automatic calibration. The problem is that color ranges are very sensitive to light. Participants implemented a utility that would highlight the desired colors on the frame (red, white and yellow) and build ranges for each of the colors in HSV format.
2. Duck Taxi. The idea of ​​this project is that Duckiebot could stop near some object, pick it up and follow a certain route. A bright yellow duck was chosen as an object.
3. Building a graph of roads. There is the task of building a graph of roads and intersections. The goal of this project is to build a road graph without providing a priori environmental data for Duckiebot, relying solely on camera data.
4. The patrol car. This project was invented by the students themselves. They offered to teach one Duckiebot, the "patrol", to persecute the other, the "intruder." For this purpose, a target recognition mechanism by the ArUco marker was used. As soon as recognition is completed, a signal of completion is sent to the “intruder”.
Color calibration
The aim of the Color Calibration project was to adjust the range of recognizable marking colors to the new lighting conditions. Without such adjustments, the recognition of stop lines, lane dividers and road boundaries became incorrect. Participants proposed a solution based on preprocessing layout color templates: red, yellow, and white.
Each of these colors has a predefined range of HSV or RGB values. Using this range, all areas containing suitable colors are found on the frame, and the largest is selected from them. This area is taken as the color to be remembered. Then statistical formulas are used, such as calculating the mean and standard deviation, in order to evaluate the new color range.
This range is written to the Duckiebot camera configuration files and can be used later. The described approach was applied to all three colors, eventually forming ranges for each of the markup colors.
Tests showed almost perfect recognition of the marking lines, except when a glossy tape was used as the marking materials, reflecting the light sources so strongly that at the camera’s viewing angle the marking appeared white regardless of its original color.
Duck taxi
The Duck Taxi project involved the construction of a search algorithm for a passenger duck in the city, and then its transportation to the required point. The participants divided this task into two: detection and movement along the graph.
Students carried out the detection of the duck, making the assumption that the duck is any area on the frame that can be recognized as yellow, with a red triangle (beak) on it. As soon as such an area is detected in the next frame, the robot should approach it, and then stop for a few seconds, simulating a passenger landing.
Then, having a pre-stored road graph of the entire duckietown and the bot’s position, as well as receiving the destination as input, the participants build a path from the departure point to the arrival point using Dijkstra’s algorithm to search for paths in the graph. The output is presented as a set of commands - turns at each of the following intersections.
Graph of roads
The goal of this project was to build a graph - a network of roads in Duckietown. The nodes of the resulting graph are intersections, and the arcs are roads. To do this, Duckiebot must explore the city and analyze its route.
During the work on the project, the idea was considered, but then the idea of ​​creating a weighted graph in which the cost of the edge is determined by the distance (time to overcome) between the intersections was rejected. The implementation of this idea turned out to be too laborious, and there would not be enough time within the framework of the school.
When Duckiebot gets to the next intersection, he chooses the road leaving the intersection on which he has not yet moved. When all roads at all intersections have been covered, a generated list of intersection adjacencies remains in the bot’s memory, which is converted into an image using the Graphviz library.
The algorithm proposed by the participants is not suitable for arbitrary Duckietown, but it worked well in a small city consisting of four intersections used within the school. The idea was to supplement each intersection with an ArUco marker containing the intersection identifier in order to track the intersection.
The working scheme of the algorithm developed by the participants is shown in the figure.
Patrol car
The goal of this project is to search for, prosecute, and detain an intruder bot in Duckietown. The bot patrol should move along the outer ring of the road in the city in search of a well-known intruder bot. After the intruder is detected, the patrol bot must follow the intruder and make him stop.
The work began with the search for the idea of ​​detecting a bot on the frame and recognizing the intruder in it. The team suggested equipping each bot in the city with a unique marker at the back - just like real cars have state registration numbers. For this, ArUco markers were chosen. They were previously used in duckietown, as it’s easy to work with them and they allow you to determine the orientation of the marker in space and the distance to it.
Next, it was necessary to make the patrol bot move strictly in the outer circle without stopping at intersections. By default, Duckiebot moves in a lane and stops at a stop line. Then, using road signs, determines the configuration of the intersection and makes a choice about the direction of the intersection. For each of the described steps, one of the states of the state machine of the robot is responsible. In order to get rid of stops at the intersection, the team changed the state machine so that when approaching the stop line, the bot immediately went straight to the state of passage of the intersection.
The next step was to solve the problem of stopping the intruder bot. The team made the assumption that the patrol bot can have SSH access to each of the bots in the city, that is, have some information about what authorization data and which id each bot has. Thus, after the intruder was detected, the patrol bot began to connect via SSH to the intruder bot and turned off its system.
After confirming that the shutdown command was completed, the patrol bot also stopped.
The algorithm of the patrol robot can be represented as the following scheme:
Project work
The work was organized in a format similar to Scrum: every morning students planned tasks for the current day, and in the evening reported on the work done.
On the first and final days, students prepared presentations describing the task and how to solve it. To help students follow their chosen plans, in the rooms where the projects were being worked on, there were constantly teachers from Russia and America who answered questions. Communication took place mainly in English.
Results and their demonstration
Work on projects lasted one week, after which students presented their results. All prepared presentations in which they told what they had learned at this school, what the most important lessons were learned, what they liked or didn't like. After that, each team presented their project. All teams coped with the tasks.
The team that implements color calibration completed the project faster than others, so they managed to prepare documentation for their program. And the team involved in the graph of roads, even on the last day before the demonstration of the project, tried to finalize and fix their algorithms.
Conclusion
After school, we asked students to evaluate past activities and answer questions about how the school met their expectations, what skills they acquired, etc. All students noted that they learned how to work in a team, distribute tasks and plan their time.
Students were also asked to rate the usefulness and complexity of the courses. And here two groups of assessments formed: for some, the courses were not very difficult, others rated them as the most difficult.
This means that the school has taken the right position, remaining accessible to beginners in a particular area, but also providing materials for repetition and consolidation to experienced students. It should be noted that almost everyone noted a programming course (Python) as uncomplicated, but useful. The most difficult, according to students, was the course "Computer Architecture".
When students were asked about the shortcomings and advantages of the school, many responded that they liked the chosen teaching style, in which the teachers promptly and individually provided assistance and answered questions.
Students also noted that they liked working in the daily planning mode of their tasks and independently setting deadlines. Students noted the lack of provided knowledge, which was required when working with the bot: when connecting, understanding the basics and principles of its work as shortcomings.
Almost all students noted that the school exceeded their expectations, and this indicates the right direction of school organization. Thus, general provisions should be maintained when organizing the next school, taking into account and, if possible, eliminating the shortcomings noted by students and teachers, possibly changing the list of courses or the time of their teaching.
Authors: JetBrains Research Mobile Robot Algorithm Laboratory Team
PS Our corporate blog has a new name. Now it will be dedicated to JetBrains educational projects.
All Articles