Convolutional texture

Self-updating textures

When it is possible to parallelize simulations or rendering tasks, it is usually best to run them in the GPU. In this article, I will explain a technique that uses this fact to create impressive visual tricks with low performance overhead. All the effects that I will demonstrate are implemented using textures, which, when updated, " render themselves "; the texture is updated when a new frame is rendered, and the next texture state is completely dependent on the previous state. On these textures, you can draw, causing certain changes, and the texture itself, directly or indirectly, can be used to render interesting animations. I call them convolutional textures .









Figure 1: convolution double buffering



Before moving on, we need to solve one problem: the texture cannot be read and written at the same time, such graphic APIs as OpenGL and DirectX do not allow this. Since the next state of the texture depends on the previous one, we need to somehow get around this limitation. I need to read from a different texture, not from the one in which I am writing.



The solution is double buffering . Figure 1 shows how it works: in fact, instead of one texture, there are two, but one is written to and one is read from the other. The texture that is being written to is called the back buffer , and the rendered texture is called the front buffer . Since the convolutional test is "written to itself," the secondary buffer in each frame writes to the primary buffer, and then the primary is rendered or used for rendering. In the next frame, the roles change and the previous primary buffer is used as the source for the next primary buffer.



By rendering the previous state to a new convolution texture using the fragment shader (or pixel shader ) provides interesting effects and animations. The shader determines how the state changes. The source code for all examples from the article (as well as others) can be found in the repository on GitHub .

Simple application examples

To demonstrate this technique, I chose a well-known simulation in which, when updating, the state completely depends on the previous state: the Conway game “Life” . This simulation is performed in a grid of squares, each cell of which is alive or dead. The rules for the following cell state are simple:

• If a living cell has less than two neighbors, but it becomes dead.
• If a living cell has two or three living neighbors, it remains alive.
• If a living cell has more than three living neighbors, then it becomes dead.
• If a dead cell has three living neighbors, it becomes alive.

To implement this game as a convolutional texture, I interpret the texture as the grid of the game, and the shader renders based on the above rules. A transparent pixel is a dead cell, and a white opaque pixel is a living cell. An interactive implementation is shown below. To access the GPU, I use myr.js , which requires WebGL 2 . Most modern browsers (for example, Chrome and Firefox) can work with it, but if the demo does not work, then most likely the browser does not support it. Use the mouse (or touch screen) [in the original article] to draw live cells on the texture.





The fragment shader code (in GLSL, because I use WebGL for rendering) is shown below. First, I implement the get



function, which allows me to read a pixel from a specific offset from the current one. The pixelSize



variable is a pre-created 2D vector containing the UV offset of each pixel, and the get



function uses it to read the neighboring cell. Then the main



function determines the new color of the cell based on the current state ( live



) and the number of living neighbors.

 uniform sampler2D source; uniform lowp vec2 pixelSize; in mediump vec2 uv; layout (location = 0) out lowp vec4 color; int get(int dx, int dy) { return int(texture(source, uv + pixelSize * vec2(dx, dy)).r); } void main() { int live = get(0, 0); int neighbors = get(-1, -1) + get(0, -1) + get(1, -1) + get(-1, 0) + get(1, 0) + get(-1, 1) + get(0, 1) + get(1, 1); if (live == 1 && neighbors < 2) color = vec4(0); else if (live == 1 && (neighbors == 2 || neighbors == 3)) color = vec4(1); else if (live == 1 && neighbors == 3) color = vec4(0); else if (live == 0 && neighbors == 3) color = vec4(1); else color = vec4(0); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Another simple convolutional texture is a game with falling sand , in which the user can throw colorful sand at the scene, which falls down and forms mountains. Although its implementation is a little more complicated, the rules are simpler:

• If there is no sand under a grain of sand, then it falls one pixel down.
• If there is sand beneath a grain of sand, but it can slide down 45 degrees to the left or right, then it will do so.

The controls in this example are the same as in the game "Life". Since under such rules, sand can fall at a speed of only one pixel per frame in order to slightly speed up the process, the texture per frame is updated three times. The source code of the application is here .

One step forward

Figure 2: Pixel Waves.



The above examples use convolutional texture directly; its contents are rendered onto the screen as it is. If you interpret images only as pixels, then the limits of use of this technique are very limited, but thanks to modern equipment they can be expanded. Instead of counting pixels as colors, I will interpret them a little differently, which can be used to create animations of yet another texture or 3D model.



First, I will interpret the convolutional texture as a height map. The texture will simulate waves and vibrations in the water plane, and the results will be used to render reflections and shaded waves. We are no longer required to read the texture as an image, so we can use its pixels to store any information. In the case of a water shader, I will store the wave height in the red channel, and the wave pulse in the green channel, as shown in Figure 2. The blue and alpha channels are not used yet. Waves are created by drawing red spots on a convolutional texture.



I will not consider the methodology for updating the height map, which I borrowed from the website of Hugo Elias , which seems to have disappeared from the Internet. He also learned about this algorithm from an unknown author and implemented it in C for execution in the CPU. The source code for the application below is here .





Here I used a height map only to offset the texture and add shading, but in the third dimension, much more interesting applications can be implemented. When a convolutional texture is interpreted by a vertex shader, a flat subdivided plane can be distorted to create three-dimensional waves. You can apply the usual shading and lighting to the resulting shape.



It is worth noting that the pixels in the convolutional texture of the example shown above sometimes store very small values ​​that should not disappear due to rounding errors. Therefore, the color channels of this texture should have a higher resolution, and not the standard 8 bits. In this example, I increased the size of each color channel to 16 bits, which gave fairly accurate results. If you are not storing pixels, you often need to increase the accuracy of the texture. Fortunately, modern graphics APIs support this feature.

We use all channels

Figure 3: Pixel grass.



In the water example, only the red and green channels are used, but in the next example, we will apply all four. A field with grass (or trees) is simulated, which can be moved using the cursor. Figure 3 shows what data is stored in a pixel. Offset is stored in the red and green channels, and speed is stored in the blue and alpha channels. This speed is updated to shift towards the resting position with a gradually fading wave motion.



In the example with water, creating waves is quite simple: spots can be drawn on the texture, and alpha blending provides smooth shapes. You can easily create multiple overlapping spots. In this example, everything is trickier because the alpha channel is already in use. We cannot draw a spot with an alpha value of 1 in the center and 0 from the edge, because this will give the grass an unnecessary impulse (since the vertical impulse is stored in the alpha channel). In this case, a separate shader was written to draw the effect on the convolutional texture. This shader ensures that alpha blending does not produce unexpected effects.



The source code of the application can be found here .





Grass is created in 2D, but the effect will work in 3D environments. Instead of pixel displacement, the vertices are shifted, which is also faster. Also, with the help of peaks, another effect can be realized: different strength of branches - the grass bends easily with the slightest wind, and strong trees fluctuate only during storms.



Although there are many algorithms and shaders for creating the effects of wind and displacement of vegetation, this approach has a serious advantage: drawing effects on a convolutional texture is a very low-cost process. If the effect is applied in a game, then the movement of the vegetation can be determined by hundreds of different influences. Not only the main character, but also all objects, animals and movements can influence the world at the expense of insignificant costs.

Other use cases and flaws

You can come up with many other applications of technology, for example:

• Using a convolutional texture, you can simulate wind speed. On the texture, you can draw obstacles that make the air go around them. Particles (rain, snow and leaves) can use this texture to fly around obstacles.
• You can simulate the spread of smoke or fire.
• The texture can encode the thickness of the layer of snow or sand. Traces and other interactions with the layer can create dents and prints on the layer.

When using this method, there are difficulties and limitations:

• It’s hard to adjust animations to changing frame rates. For example, in an application with falling sand, grains of sand fall at a constant speed - one pixel per update. A possible solution may be to update convolutional textures with a constant frequency, similar to how most physical engines work; the physics engine runs at a constant frequency, and its results are interpolated.
• Transferring data to the GPU is a quick and easy process, however, getting data back is not so easy. This means that most of the effects generated by this technique are unidirectional; they are transferred to the GPU, and the GPU does its job without further intervention and feedback. If I wanted to embed the wavelength from the water example in physical calculations (for example, so that the ships would oscillate along with the waves), then I would need values ​​from the convolutional texture. Retrieving texture data from the GPU is an awfully slow process that does not need to be performed in real time. The solution to this problem can be the implementation of two simulations: one with a high resolution for water graphics as a convolutional texture, the other with a low resolution in the CPU for water physics. If the algorithms are the same, then the discrepancies may be quite acceptable.

The demos in this article can be further optimized. In the grass example, you can use a texture with much lower resolution without noticeable defects; this will help a lot in big scenes. Another optimization: you can use a lower refresh rate, for example, in every fourth frame, or a quarter per frame (since this technique does not cause problems with segmented updates). To maintain a smooth frame rate, the previous and current state of the convolutional texture can be interpolated.



Since convolutional textures use internal double buffering, you can use both textures at the same time for rendering. The primary buffer is the current state, and the secondary is the previous one. This can be useful for interpolating the texture over time or for computing derivatives for texture values.

Output

GPUs, especially in 2D programs, are often idle. Although it seems that it can only be used in rendering complex 3D scenes, the technique demonstrated in this article shows at least one other way of using the power of the GPU. Using the capabilities for which the GPU was developed, you can implement interesting effects and animations that are usually too costly for the CPU.