Faking Ambient Occlusion in SFML tile maps

While working on a tile map based project in SFML I came up with this effect for creating soft, pre-rendered shadows which looks a little bit like ambient occlusion. If you don't know what ambient occlusion is then a quick search on <insert favourite search engine> will tell you that it is an effect employed by most modern 3D games to create subtle self-shadowing of models which provides added realism. As the effect is based on 3D data, however, it's not something readily available in 2D games such as one may make with a library like SFML. It is entirely possible, of course, to render your 2D graphic elements in a 3D package first or to even create a set of shadow tiles which can be placed in a map to create the effect - although these 2 approaches were not really applicable to my project at hand.

   A little background: I often use tiled maps in my games as it is simple and relatively quick for me to use Tiled a 2D tile map editor coupled with this class to load the map file into an SFML based project. Most often I use it to create a large map, either for a top-down view or platform game, where only the visible tiles of the map are ever drawn. In these scenarios it is usually quite viable, if a little time consuming, to carefully draw shadowing tiles and place them around the edges of walls and platforms in a map. My current project, however, is a little different. The premise is to have a maze, containing marbles or similar, which the user rotates and tilts using the mouse or keyboard in order to get the marbles to a specific destination. I used Tiled to create the maze because it was quick and easy to lay out and because the aforemetioned map-loader class made it easy to get in to my project.

    The big difference is that the entire map is seen on screen at one time so rather than loop over the array of tiles drawing each one at ~60fps it made sense to pre-render the entire map to a single render texture at load time, cutting the number of draw calls drastically as well as having the added bonus of creating one single sprite which can be manipulated / rotated by the user. This got me thinking about the other possibilities of using off-screen render textures particularly when coupled with some shader effects. As the pre-rendering sequence is only ever done once at start up it actually afforded me some render time to create effects which may have been less viable to do when rendering in real time. My initial flow was something like this:

Load the map file and tilemap texture

Create a render texture the size of the maze/map

Render all the tiles to the render texture

Create a sprite from this texture

Render this sprite in real time, rotating via feedback from user input.

When I looked at the render texture stage I realised that I needn't do much more than create another render texture to act as a shadow buffer. I could then add another step which rendered all the tiles as black and blurred the output via a convolution shader using a gaussian blur matrix. If you're not sure about convolution matrices then this is a good reference. Ultimately this gave me a blurred dark shadowy output which I could render underneath the tiles to create a subtle shadow effect. The flow now looked like this:

Load the map file and tilemap texture

Create two render textures the size of the maze/map

Render each tile to the first render texture setting their colour to black

Create a sprite from this render texture and render it to the second texture via the convolution shader to blur it

Render all the tiles to the second render texture over the top so that the shadow would 'bleed' out around the edges from underneath

Create a sprite from this texture

Render this sprite in real time, rotating via feedback from user input

This was a good start, however I wasn't entirely pleased with the blur effect. I'd used a convolution filter originally as it was a cheap effect processing-wise and only required a single pass (and has the added bonus of being able to create different effects by altering its matrix). Seeing as this was an effect that only ever got rendered once during pre-rendering I decided I could afford myself a little extra processing and replaced the convolution blur shader with a pair of gaussian blur shaders, one of which blurred pixels horizontally and one which operated on the pixels vertically. I also passed the shadow render through each of the blur shaders multiple times to really draw out the shadows and soften up the effect. The last part this required me to create another render texture to 'ping-pong' the blur passes before finally rendering the output.

The final flow:

Load the map file and tilemap texture

Create three render textures the size of the maze/map

Render each tile to the first render texture setting their colour to black

Create a sprite from this render texture and render it to the second texture via the horizontal blur shader

Use a loop to render the image back to the first texture via the vertical blur and the second texture again via the horizontal blur n times, where n is the number of blur passes (the 'ping-pong')

Render the blurred image out to the third render texture

Render all the tiles to the third render texture over the top so that the shadow would 'bleed' out around the edges from underneath

Create a sprite from this texture

Render this sprite in real time, rotating via feedback from user input

Which in SFML specific code goes:

    m_vBuffer = new sf::RenderTexture;
    m_vBuffer->create(640, 640);
    m_vBuffer->setSmooth(true);
    m_vBufferSprite = sf::Sprite(m_vBuffer->getTexture());

    m_hBuffer = new sf::RenderTexture;
    m_hBuffer->create(640, 640); //TODO kill magic numbers!
    m_hBuffer->setSmooth(true);
    m_hBufferSprite = sf::Sprite(m_hBuffer->getTexture());

    m_vBlurShader = new sf::Shader;
    m_vBlurShader->loadFromMemory(vBlur, sf::Shader::Fragment);

    m_hBlurShader = new sf::Shader;
    m_hBlurShader->loadFromMemory(hBlur, sf::Shader::Fragment);

    renderTexture.clear(sf::Color(68u, 58u, 7u));
    m_hBuffer->clear(sf::Color::Transparent);
    
    //render the tiles black for shadow
    for (unsigned layer = 0; layer < m_layers.size(); layer++)
    {
        for (unsigned tile = 0; tile < m_layers[layer].tiles.size(); tile++)
        {
            if (m_drawingBounds.contains(m_layers[layer].tiles[tile].sprite->getPosition().x, m_layers[layer].tiles[tile].sprite->getPosition().y))
            {
                m_layers[layer].tiles[tile].sprite->setColor(sf::Color::Black);
                m_hBuffer->draw(*m_layers[layer].tiles[tile].sprite);
                m_layers[layer].tiles[tile].sprite->setColor(sf::Color::White);
             }
        }
    }

    m_hBuffer->display();

    //ping pong blur
    for(int i = 0; i < 5; i ++)
    {
        m_vBuffer->clear(sf::Color::Transparent);
        m_vBuffer->draw(m_hBufferSprite, m_vBlurShader);
        m_vBuffer->display();

        m_hBuffer->clear(sf::Color::Transparent);
        m_hBuffer->draw(m_vBufferSprite, m_hBlurShader);
        m_hBuffer->display();
    }

    renderTexture.draw(m_hBufferSprite);

    //draw tiles normally
    for (unsigned layer = 0; layer < m_layers.size(); layer++)
    {
        for (unsigned tile = 0; tile < m_layers[layer].tiles.size(); tile++)
        {
            if (m_drawingBounds.contains(m_layers[layer].tiles[tile].sprite->getPosition().x, m_layers[layer].tiles[tile].sprite->getPosition().y))
            {
                renderTexture.draw(*m_layers[layer].tiles[tile].sprite);
            }
        }
    }

    renderTexture.display();
    mazeSprite = sf::sprite(renderTexture.getTexture());

This code won't work as a direct copy / paste, of course, but should be enough to get anyone interested in doing the same thing started. As always I'm open to suggestions for improvement, but I'm pretty pleased with the results.