Two-phase occlusion culling (part 2) - instance aware culling

I described the basics about the two-phase occlusion culling technique in the first part post, but there is one thing, that doesn't work well with this simple setup: Instancing or sometimes viewed as batching.


The problem is very simple, the solution however is not. Only a few engines managed to implement the technique, but it is unavoidable if one needs to render large batches or large amounts of instances. Quick recap, under the assumption that we compromise flexibility in favor of speed and use OpenGL 4.5 level graphics APIs:

Avoid buffer changes one, two different vertex layouts (for example for static and animated meshed)

Avoid program changes well defined firstpass program (deferred rendering) or subroutines in a global attribute buffer

Minimize drawcalls Usage of instanced and indirect rendering


The usage of draw commands and indirect rendering is nice enough. The amount of drawcommands is reduced by using instanced rendering for entities that have different uniform attributes but the same vertices and indices. This results in the command and entitiy buffers of the following form:

Entities
uint entityId	678
mat4 transform	000...
uint materialIndex	18
uint entityId	1566
mat4 transform	000...
uint materialIndex	2

Commands
vertexOffset	0
indexOffset	0
indexCount	99
instanceCount	2
vertexOffset	1200
indexOffset	99
indexCount	33
instanceCount	3

Firing indirect rendering call is now done with a count of two, because we have two commands. 5 objects will be drawn. The shaders are invoked several times and the entity data can be accessed with an index and fetched from the entities buffer. The resulting indices in the shader will look like

Shader Invocations
DrawId	InstanceId	Entity buffer index
0	0	0
0	1	1
1	0	???
1	1	???
1	2	???

As mentioned in the previous blog post, there's no chance for us to know the entity index when the second command is executed, because the current offset is based on the amount of instances the previous commands draw. The solution is to use an additional offset buffer with size of the draw command buffer. For every command, we set the offset to the appropriate value when creating the commands on the cpu side. With instance based culling, this problem intensifies, because the cpu doesn't know the offset anymore. The calculation has to be done on the GPU now. My solution is still based on vertex shader kernels, but I will tell you later why this is probably problematic. First how it is done, because conceptionally, it will be the same for compute shaders. The layout now looks like this:

Shader Invocations
DrawId	InstanceId	Entity buffer index (command offset + InstanceId)	Offset of the command
0	0	0	0
0	1	1	0
1	0	2	2
1	1	3	2
1	2	4	2

The first step is to determine the visibility for every single instance. Vertex shader kernels have an advantage here, because they can be arbitrarily large (compute shader groups are limited to 1024 or so). A single draw call can determine visibility for dozens of thousands of instances or entities. Combined with instancing, we can use DrawId and InstanceId to index into the command buffer and offset buffer (DrawId) and into the entity buffer (offset + InstanceId). Since the same kernel sizes are applied to every command, invocations are wasted if few commands have many more instances as others. So you might want to launch draw calls without instancing, one per command, which could be faster here.
The visibility is again stored into the visibility buffer, which has to be as large as the entity buffer now. With non-instanced culling, size of the command buffer was sufficient. One important thing has to be done now: Every visible entity thread has to increase a counter that is associated with the corresponding command - since this is done per invocation, atomic operations are needed. So we need another int buffer (just like the offset buffer), that holds the visible instances count per command.

The reason why this is so important is, that this is the bit of information that is used in a second step to append entity data and draw commands in parallel. This is done by an algorithm similar/equal to parallel prefix sum - you can google it. Tldr: Each command has to know how many visible instances all previous commands produce in total, so that it can append its own instances there.

I call the seconds step appending step, while the first one is the visibility computation step, that actually calculates the visibility of an instance. For a massive amount of instanced commands, one would probably want to launch a two-dimensional kernel of the size n*m where n is the amount of commands and m is max(maxInstanceCount, someArbitraryValue) or something. Again, with vastly different instanceCounts per command, multiple shader calls could give a benefit.

So now we need a target buffer that holds the entity data, a target buffer for the commands and a target buffer for the entity data offsets. Additionally, we need an atomic counter for the target command index and then an atomic counter per command, that is the current index of the entities per command. Each shader invocation has its own command index in the DrawID built-in and the instance index in the InstanceID  built-in. So we can calculate all information that is needed on the fly:

struct oldCommand = read from the bound command buffer with DrawID
uint  oldCommandEntityOffset = read from the oldOffsetBuffer with DrawID
uint oldEntityIndex = oldCommandEntityOffset + InstanceID
uint visibilityBufferIndex = oldEntityIndex
uint targetCommandIndex = atomicAdd(commandConuter, 1)
uint targetInstanceIndex = atomicAdd(commandCounters[targetCommandIndex], 1)
uint targetEntityIndex =  sumOfVisibleInstancesOfAllCommandsBeforeCurrentCommand

The current entity data can then be written to the targetEntityDataBuffer, as well as the offset for the instance/command. The command can be written by the first active thread (InstanceID == 0). The resulting buffers contain only the visible instances, offsets of the visible instances and drawcommands that can be used to only draw exactly those instances.

 Here's a video demonstrating occlusion and frustum culling with this technique:

Bonus: For the visibility computation step, it's nice to launch a vertex shader kernel, since the kernel can be arbitrarily large in two dimensions at max. Since there's no shared memory involved, this will probably perform better than the compute shader equivalent, maybe at the same speed, but it shouldn't be slower. The appending step needs an atomic operation per invocation, because every invocation increments the counter if the corresponding instance is visible (which the other invocations couldn't know). Instead of "global" shared memory, one could use shared memory of a compute shader group. Shared memory in a group is much faster than atomic operations between a gazillion vertex shader invocations, so I will implement this at some time and may be able to show a performance comparison.

Multithreaded game engines and rendering with OpenGL

There is this one question since beginning of multicore processors: How to multithread your game engine? I can't answer this question in general, but I can answer the question How to multithread the rendering part of my game engine?

Problem

The most simple case is a single-threaded game engine that does this:

 while(true) {  
   simulateWorld();  
   renderWorld();  
 }  

The update step produces a consistent state of the world. After each update, the rendering is done. After the rendering is done, the next cycle is processed. No synchonization needed, everything works. The bad thing: simulateWorld is a cpu-heavy task, renderWorld is a gpu-heavy task. So while the world is updated, the gpu idles, while the gpu renders, the cpu idles (more or less). And the worst thing: Your framerate is limited by both the cpu and the gpu. Your frametime is timeOf(simulateWorld) + timeOf(renderWorld). Our target frametime is timeOf(renderWorld) and to keep this time as low as possible. More precisely the target is timeOf(renderWorld).

Foundations

Okay I lied: The above mentioned problem is only the overall-globally-problem we solve. There's a shitload of other problems, that mostly depend on the language platform, graphics api, hardware, your skillset and so on. Since our lifetime is limited, and I want to give practical, real-world-relevant help, I make some assumptions. One of these is, that you use OpenGL and a pretty new version of it. Let's say 4.3. Multithreaded rendering could probably be done better and faster with DX 12 or Vulkan, but let's stay cross-platform and simpler with OpenGL.

I assume that you already have at least a little bit knowledge about how game engines work, timesteps, scene representations and so on.

Multiple gpu contexts

In order to be able to issue commands to OpenGL, you have to use a thread that has an OpenGL context bound to it. Architecture-wise, there is the CPU, the OpenGL driver and the GPU. The driver holds a queue, where the commands from the SPU are buffered somehow (implementation dependend). Then again, there is a queue for the GPU - that again buffers commands that came from the driver (implementation dependend). First, all commands the CPU issue are asynchronous, which means there's no actual work done on the GPU yet, but the command is queued by the driver. The order is preserved, that's what you can count on. The implementation can decide how many commands are queued up, until the cpu thread will block on command issuing. And the implementation can decide how many commands are buffered on the GPU side of things.

Now, having that said, let's think about multithreading. What we can control is the CPU side (unless you write your own driver, which I doubt you want to do). Multiple cpu threads would require multiple gpu contexts - one for each thread. While this is theoretically possible with OpenGL, it is a very dumb idea in 19 out of 20 cases. As with traditional multithreading, context switching comes with a very high overhead - so you have to ensure that context switching overhead doesn't eat up the performance you gained from using multiple threads. And here's what's wrong: It seems that all OpenGL drivers (where you issue your comamnds at) except the one for iOS, are implemented singlethreaded. That means you can issue commands from multiple cpu threads, but on the driver thread, those commands are processed sequentially, with a context switch in between. You can find a more detailled explanation here, so that I don't have to lose many further words, but: Don't use multiple contexts for anything else than asynchronous resource streaming.

I assume we only use one OpenGL context. That means, all OpenGL calls must happen on a single thread. The construct used for this is the command pattern. A few hints: You should issue as few commands to OpenGL as possible. You should issue as lightweight commands as possible. Obviously, using only one thread for command execution limits the amount of work that can be done by this (cpu) thread. Nonetheless, you want a single, inifintely running worker thread that owns your OpenGL context and takes commands out of a non-blocking command queue. For convenience, you can seperate commands that should return a result (hence block) or should just be fired and forgotten. It's not nessecary that all commands are properly implemented by yout context wrapper/worker thread class, even though it makes your application more predictable in terms of performance. I implemented a non-blocking, generic command queue in Java here that uses Runnables and Callables as Commands. The principle is also explained more detailed here.

 Extractor pattern

Extractor pattern is a term one never finds on the internet when searching for information about rendering. The first time I heard this, was when a (professional graphics programming) collegue of mine explained the basics of renderer architectures to me many many years ago. Fancy name for a simple thing: If you share data between two consumers (not exactly threads in this case), you either have to synchronize the access somehow, or you extract a copy of the data to pass it to the renderer in this case. That implies, that you need a synchronized/immutable command-like data structure, that represents all the information you need to push a render command. That plays nicely with the chapter above, you know, command queue etc. In languages other than Java where you have true value types, the implementation could be easier - although having very large renderstate objects copied over and over might be a drawback here. The chapter below shows another way of handling this problem.

Multibuffering

Let's assume that the GPU only consumes renderstate, but doesn't alter it. That means if the GPU calculates any fancy physic, or results that should be passed to the next render cycle, than it isn't seen as part of our traditional renderstate for now.
That means the only producer of renderable state is the CPU. Since we don't want to render a scene where the first half objects are in timestep n and the second half is still in n-1, we have to be able to access a state of our gamestate, that is concise somehow. This is easily achievable with a task-based architecture, that basically has only one update thread, that sequentially crunches down what you want to do in parallel. For example if you have 10 game objects, the updatethread pushes 10 update-commands into a pool of worker threads and after all of them returned a result, we have a concise world state we can use and used all threads our system can push. I don't think, there's an alternative approach to multithreading that can be used in a general purpouse game engine, but I may be wrong. Instead of creating a new renderstate object and pushing a command to the renderqueue, we use something that's known in multithreading environments for ages: multibuffering.

This should be confused with double or triple buffering that is used to display images on your monitor, although the principle behind it is the same.

I skip explaining why double buffering isn't enough to satisfy our needs. Just use triple buffering and be happy, if you can afford the additional memory consumption. This works the following way.

You have three instances of your renderstate. You have a wrapper, that encapsulates these three instances. Instance A is the currently readable state. The renderer uses this state for rendering, so while rendering is in progress, this instance mustn't be touched by anyone else than the renderer. Instance B is the current staging copy of the renderstate. It is the next state, that the renderer will use for the next frame, after the current frame is finished. Copy C is the isntance the update is currently applied on.
Now while this sounds simple, the important thing is, how to swap those instances correctly. Our main purpouse is to keep the GPU busy and to be able to render maximum fps, although one could theoretically (and practically) limit the framerate somehow (vsync, fps cap etc.).

I realized this with a ever-running thread that does push and wait for a render-scene-drawcall constantly. After the frame is finished, the read copy is swapped with the staging copy. It's important, that the renderer get's a fresh copy on every swap. If the update thread isn't fast enough to provide a new updated renderstate copy, the renderer would render a state that is older than the one just drawn. This will cause flickering objects. I realized this with an ongoing counter, and I prevent swapping to a state with a lower number then the current state has. Let's assume your engine and the machine is capable of pushing 60cps update and 60fps framerate.
Let's face the other swap: The update thread updates the write copy C constantly as fast as possible. When finished, the write state becomes the new staging state B, so we have to swap them. We can't do this if the renderer is currently swapping his read and the staging state. So we need a lock for swap(A,B) and a lock for swap(B,C). If rendering can be done with okayish framerates (which we assume), the updatethread can wait (block) until swap(A, B) is finished. If you are using only coherent memory access, than you're donw here. But you wouldn't search for state-of-the-art rendering, if you don't use incoherent access. So let's get to the final ingredient we need for our super fast renderer.

Lowering draw calls and persistent mapped buffers synchronization

At the very beginning, there is the need to reduce the amount of work the gpu has to do, in order to render your scene. There are paths thorugh the OpenGL API that are more expensive than others, and there are paths with very very small overhead, if you can afford the loss of flexibility that comes with using it. This can give you a rough overview about how costly API calls are. I suggest we skip a long explenation and you read this. Summed up: You want unsynchronized buffers for all your data and handle synchronization by yourself. This results in no driver or gpu work for the synchronization, which is crucial for maxing out your performance. Remember that Vulkan was designed to exactly kill the driver overhead of OpenGL. Furthermore, we reduce draw calls and state changes with bindless resources and one of the most important things: indirect and instanced rendering.

Having all these things implemented, there's one last thing to add to our triple buffering: synchronization. Since the driver works asynchronous and the gpu, too, there's no guarantee that the gpu doesn't use the buffer of our current write copy, that the update threads currently writes to. To adress this, one has to insert a fence sync object into the pipeline and associate it with the state, whenever a state becomes the current write state. When the render thread uses this thread and pushing all render commands of the current frame are pushed, the fence signal is pushed. Whenever this signal is processed by the gpu, the update thread can write to the associated state without any problems. The update thread can prepare the renderstate update and wait until the prepared data can be applied. This effectively blocks the update thread, if the renderer is more than 2 frames behind the update thread.
Special care has to be taken if you have any timestep-dependent calculations in your rendeirng step: Since rendering is decoupled form the update-step, you have to calculate the timestep by yourself somehow in the renderthread, based on the last tick and the time passed since then.

That's it

•  Using the fastest possible paths through the OpenGL API. You need shader storage buffers and indirect (and inntanced) rendering, probably combined with bindless textures
• Model a renderstate class that can be your complete draw command
• Run a thread permanently, that permanently pushes rendercommands

Voxel Cone Tracing Global Illumination in relatime

Quite a while ago, I implemented Voxel Cone Tracing global illumination for my own custom engine. This technique can be quite impressive. Used Java, OpenGL, running on a GTX 770 desktop GPU.



Here's another video of the famous cornell box, where I added a different diffuse tracing calculation. Same engine (newer version), Running on a GTX 1060 mobile gpu.


And another run with a point light only configuration on the mobile gpu.

Rendering 4 Mio. vertices with 340000 cubes in Java

Someone has to fight the battle against the "Java is so slow, all Java games have low fps" myth. Maybe Java is not the best language for game development because it lacks value types and easy and zero overhead native code integration .... but the results one could achieve with using a zero-driver-overhead path like modern graphics APIs recommend for so many years, can be on par with what you can achieve in C or C++ as in Unreal, CryEngine or Unity. The secret is: Bindless ressources, indirect and instanced rendering, persistent mapped buffers, direct state access and of course good old multithreading.

So, the most important things first: Use indirect rendering (to minimize calls the cpu has to issue!) and a shared global vertex buffer (to reduce state changes). Use large uniform or shader storage buffers for your object's properties and material properties. For each object, push a command into the buffer. Massive object counts cause massive command counts and large command buffers, so one can now use instancing to further reduce command count. With a clever structure (a offset buffer for example), one can easily have unique properties per object instance (for example dedicated textures per instance!) sourced from a large uniform buffer, when it's okay to share a common geometry (vertices). Et voila, 2ms cpu time to fire a render command that draws 340.000 instanced cubes with 4 million triangles at 60 fps in Java. Each object can have it's own textures and properties etc.

glDrawElementsBaseVertex demystified

In days of Vulkan and DX12, I started investigating how much time my cpu spends on pushing the OpenGL draw calls I need. Turns out: too much if you ask me. On my desktop machine, there's no problem at all, but on weaker hardware, it might be relevant. And at all, my CPU should not have work to do at all, or at least I don't want it to spend more time on pushing a command than the command actually takes to complete on the GPU.

One very evil thing you mostly can not avoid is vertex buffer binding. If you want to have it flexible, every model could have its own vertex buffer, which you can unload, modify and stuff. However, most of your scene contents share a common vertex format - for example you mostly have position attributes, normals and texture attributes. Heavier stuff like bone matrices and material parameters could be unload to seperate buffer objects and accessed via an index, so using a shared vertex and index buffer for all your scene's geometry really really should pay off.

My way was using a pinned memory buffer (aka persistent mapped) that automatically doubles when too small. Registering a new model in the scene simply appends the unique vertex attributes of this model to the global buffer. The same for the indices. Drawing could now be done with glDrawElementsBaseVertex.

Turns out there's a lot of confusion about the usage of this beast and some people even can't figure out how it is different from glDrawElements with a index buffer offset passed. The magic is: the base vertex attribute of glDrawElementsBaseVertex adds a value to the contents of your index buffer. Indeed, it's not an offset that is used to retrieve the current index from the index buffer, it is an offset that is actually added to the value of your index. Why? Because this way, you don't have to take care of adjusting indices by yourself when appending indices to a global index buffer. Here's an example:

You have a plane, constisting of two triangles - 4 unique vertices (the corners) and 6 indices (2 triangles a 3 vertices). After you put those in your global buffers, you want to add another plane. If you append the new plane's 4 unique vertices to the global vertex buffer, you have 8 vertices in it. The first vertex can be accessed with index 4+0, the second one with 4+1 and so on. This offset has to be added to the indices of the new plane, or you can't use its indices with the global buffers. Since this is dumb work, OpenGL offers glDrawElementsBaseVertex. In this case, one could use it as follows to draw your two planes:

glDrawElementsBaseVertex (GL_TRIANGLES, 3*2, GL_UNSIGNED_INT, 4*0, 0);
glDrawElementsBaseVertex (GL_TRIANGLES, 3*2, GL_UNSIGNED_INT, 4*4, 4);

where the second parameter is the index count - 2 triangles times 3 indices per triangle, the fourth parameter is the byte offset into the index buffer and the last parameter is the value that should be added to each index value of your current draw call.

Puh, took me a while. Now we're prepared for indirect drawing.

Simple deferred translucent foliage rendering

Translucency seems to be one of the new illumination features that every graphics engine has to provide these days. The effect is most noticeable on organics, like skin (ears for example) and plant leafs. For the latter, there's a nice and easy way to implement it - this solution only works for very thin objects. This means double sided triangles, like paper, curtains, or leafs. Other cases would be more complicated.

You should have a mechanism to support multiple material types in your deferred renderer already. Then you need to disable backface culling, or the back side of the object won't be rendered at all...Let's talk about direct illumination first. Light traveled through the object and direct light is treated at the same time. Since we assume that our translucent objects are infinitely thin, we don't need to determine a thickness, like you would have to, regularly. An artificial thickness can be provided via per-object parameter or through a texture. One could assume, that texture coordinates are the same for the point on the front and the back side of the currently rendered fragment. For normals, another assumption is possible: The normal on the back side should be the current fragment's normal, but just negated. Using different normals or diffuse texture for the backside is not easy to add here, since you can't know if you are currently rendering the front or the back side ob the object.

After calculating the regular lighting, calculate the lighting with the artificial backface normal und multiply it with the object's thickness at this point. Add the two values and you're done. Even though the usage is limited to very thin and simple-colored objects, this can look very nice, for example with a curtain:

Look at the subtle shadow of the sphere above the curtain that you can now see from below the curtain.

Bindless texture with LWJGL

The classic bound texture paradigma is so 1990. But getting bindless textures to work is not that easy, especially when you're using something other than C or C++. For all the LWJGL or Java users out there, here are some notes that may help you removing the classic texture pipeline from your engine. I think you can read about the technique in the internet, so I'll keep it mostly short.

First of all, you need a handle (some kind of pointer again, but don't think of it as a pointer, yucks) for a given texture.

 long handle = ARBBindlessTexture.glGetTextureHandleARB(textureID);  

Afterwards, the handle has to be made resident. I think that's something you need for the combination with partially resident textures.

 ARBBindlessTexture.glMakeTextureHandleResidentARB(handle);  

This was the easy part. Now, you have to use the handle in our shaders somehow. The easiest way would be to use it as a uniform.

 ARBBindlessTexture.glUniformHandleui64ARB(location, handle);  

Inside your shader, you have to use the handle. But what datatype should one use? Maybe I missed something elementary, but  there's only one proper way, namely use a datatype made available through an nvidia extension. Since bindless textures are available through an extension as well, here are both calls that you (probably) need:

 #extension GL_NV_gpu_shader5 : enable  
 #extension GL_ARB_bindless_texture : enable  

And now, you can use tha datatype uint64_t. So your uniform would be a uint64_t.

That would work. But most probable, you want to have your data in a uniform or storage buffer, probably together with some other data and datatypes. So here's what I did.

Use a DoubleBuffer (Java native buffer, available via BufferUtils.createDoubleBuffer(int size)) for your data. Since doubles are twice the size of a float, which is 4 byte, we have 8 bytes per texture handle, which is 64 bits, which is the same as a uint64's size, so that's enough. Now one has to take the generated handle's bits and put them into the buffer (for example a ssbo) as they are. This can be done like so:

 GL15.glBufferSubData(GL43.GL_SHADER_STORAGE_BUFFER, offset * primitiveByteSize, values);  

-
Where primitiveByteSize is 8 in this case. Since we use the underlying buffer as a DoubleBuffer, we have to provide a double value for the handle (or use it as a byte buffer, but nevertheless we need the correct bits or bytes). You can convert a Java long to and from a double like this:

 Double.longBitsToDouble(longValue)  

 Double.doubleToLongBits(doubleValue)  

Afterwars, the shader can take the value as a said uint64_t and cast it to a sampler. Sounds ugly, maybe it is.

 color = texture(sampler2D(uint64_t(material.handleDiffuse)), UV);  

That is the whole story, took me a while to figure it out.

Single Pass Omnidirectional Shadow Mapping Evaluation

Layered rendering is possible since geometry shaders entered the OpenGL stage. The rough idea of a good omnidirectional shadow mapping (for example for pointlights) is to render the complete shadow map with a single draw pass, in order to reduce the amount of rendercalls, compared to six-pass rendering with traditional rendering to a cubemap. Therefore, the geometry shader emits a vertex for each incoming vertex of the non-culled scene geometry to a face of the cubemap with something called layered rendering. While the idea is well described all over the internet already, every now and then, there is discussion about the efficiency of this method.

My first idea was to evaluate omnidirectional shadow mapping with dual paraboloid shadow maps since it's the easiest way to achieve pointlight shadow: No viewmatrices, no projectionmatrices, two textures per pointlight, let's go. Without view frustum culling, I draw the complete geometry twice and passed a "backside" flag as a uniform variable for the second rendering. No layered rendering, just two depth buffers and two color attachments (can be ommited if no variance shadow mapping is used) for each pointlight (front and back). It's possible to do dpsm with a single-pass rendering in a single texture - I doubt it to be efficient because you would have to handle the depth buffer somehow. Or it will be efficient, but much more complicated.

Rendering a single (two texture) shadowmap for the famous sponza atrium takes ~1.5 ms on my GTX 770.

Layered rendering however, took me a while for the implementation. Because I use array textures in order to be able to use many shadow mapped lights, I had to fight with the strange array indices for cubemap arrays. For everyone who is interested, here's how you create a cubemap array rendertarget and use it to draw your shadow maps:

 framebufferLocation = GL30.glGenFramebuffers();  
 // Create rendertarget
 GL30.glBindFramebuffer(GL30.GL_FRAMEBUFFER, framebufferLocation);  
 IntBuffer scratchBuffer = BufferUtils.createIntBuffer(colorBufferCount);  
 for (int i = 0; i < colorBufferCount; i++) {  
      GL32.glFramebufferTexture(GL30.GL_FRAMEBUFFER, GL30.GL_COLOR_ATTACHMENT0 + i, cubeMapArrays.get(i).getTextureID(), 0);  
      scratchBuffer.put(i, GL30.GL_COLOR_ATTACHMENT0+i);  
 }  
 // Use glDrawBuffers(GL_NONE) if you don't need color attachments but depth only
 GL20.glDrawBuffers(scratchBuffer);  
 CubeMapArray depthCubeMapArray = new CubeMapArray(AppContext.getInstance().getRenderer(), 1, GL11.GL_LINEAR, GL14.GL_DEPTH_COMPONENT24);  
 int depthCubeMapArrayId = depthCubeMapArray.getTextureID();  
 GL32.glFramebufferTexture(GL30.GL_FRAMEBUFFER, GL30.GL_DEPTH_ATTACHMENT, depthCubeMapArrayId, 0);  

Although some implementations are hidden because of my framework, the idea should be clear. Don't forget to initialize the cubemap array texture correctly or your framebuffer object won't be complete.

Since the geometryshader decides which face to render to, you can pass the pointlight index as a uniform variable. The layer will then be gl_Layer = 6*lightIndex + layer.

The quality is very nice, but unfortunetly, rendering sponza into the cubemap takes ~8.3ms on my GTX 770. The additional color attachment is not responsible for the expensiveness, since I tried to remove it. I'm pretty sure that I didn't do a major mistake in the implementation. It has to be the poor performance of the geometry shader that is responsible for the high amount of time this method takes.

Conclusion

• Dual paraboloid shadow map rendering ~1.5 ms
• Single pass cubemap shadow map rendering ~ 8.3 ms

I used 512*512 per dspm face or cubemap face. Would be nice to hear anyone alse's experience with omnidirectional shadowmapping.

Compute shader advices

Recently, I had a lot of pleasure with OpenGL's compute shaders. With this lot of pleasure came a lot of pain because I made some (rookie) mistakes. So I wanted to share my experience and some advices I have, just in case you have troubles too:

• The first thing you should check are your texture formats! No, really, double check it, don't repeat my mistakes. In your compute shaders, you could use your images (not textures) with
  
  
  glBindImageTexture(unit, textureId, 0, false, 0, GL_WRITE_ONLY, GL30.GL_RGBA16F);
  
  
  of OpenGL version 4.2 as an output texture. Of course you could use GL_READ_ONLY or GL_READ_WRITE if you use the texture differently. Also keep in mind that this call binds an image, not a texture. And that's why you have to provide a mipmap level you want to attach. I used the wrong format once, namely rgba32f, which my rendertarget attachments didn't have, and it resulted in non existent output from my compute shader. Very frustrating but correct behaviour.
• Keep in mind that you could use your regular textures via samplers in your compute shaders, too. Simply bind the texture and have a  similar line to this in your shader
  
  
  layout(binding = 1) uniform sampler2D normalMap;
  
  That's helpful if you want to access mip levels easily.
• Since even in the OpenGL super bible is a typo that doesn't help to understand the compute shaders built-ins, I recapture them.
  With dispatchCompute you have to provide three variables that are your group counts. A compute shader pass is done by a large number of threads and defining clever group counts/sizes will help you to process your data. In graphics related cases, mostly you will need compute shaders to render to a texture target. So it would be clever to have a certain, two-dimensional amount of threads, wouldn't it? Define your group sizes corresponding to your image size: a 320*320 image could be devided into 10*10 groups, or tiles - and each will have 32*32 pixels in it. So you should define your group size as 32, 32, 1. Now you can dispatch 320/group size threads, which will be 10 groups, for x and y dimension. In your shader, you will be able to use the built-in gl_WorkGroupSize to have this information in every invocation of your shaders main method. To uniquely identify your invocation, you can use the gl_GlobalInvocationID. If you use your shader like I said in this example, this would contain your texel's position the invocation would have to write. And that's how you can use compute shaders to manipulate your textures. Additionally, there is a gl_WorkGroupID, that identifies your tile/group of the invoation, and gl_LocalInvocationID, that is your pixels position in its tile. Sometimes, it could be useful to use a flattened identifier - for example if you have a task that requires performing an action just 12 times, but has to be done in the compute shader - and therefore you can use gl_LocalInvocationIndex. You can use it as a conditional to limit some code paths like
  
  

  if(gl_LocalInvocationIndex < MAX_ITEMS) { processItem(); }
  
  For a better understanding, have a look at this post, which has a nice picture and another explanation of the group layout.

What else? Compute shaders are awesome! I like how easy it is to invoke them, independent of something like the graphics pipeline. Use compute shaders!

Quick look at my OpenGL engine

I just want to share a small screenshot of my OpenGL rendering project. Includes physical based rendering, a global illumination concept daveloped by myself and realtime (glossy) reflections as you can see on the screenshot. Realtime of course - seen on my GTX 770 in full hd at max 300 fps.

Java 8 default methods for your game engine transformations

Although transformation and class hierarchies in game engines are a topic for itself for sure, I finally arrived at a point where I just want every single of my regular game objects to be a transformable entity. Those objects that don't act as something that can be transformed are outside of my interest, they need some default behaviour I don't care about. I think that is the way the Unity engine took, too. A shared super class might look like a good idea, but we are often warned about such kind of class hierarchies.

While C++ offers multiple inheritance to implement things like this very easily, in languages like Java, you probably have to use composition - which should be favored over inheritance nevertheless. The problem is that sometimes, you get lost in interfaces and class fields... and see yourself write interface implementations again and again while delegating interface calls to field objects.

The last sentence is the catch-word: I tried to learn about Java 8's new stuff and stumbled across default methods. While mainly created to guarantee binary compatibility when changing interfaces, they offer a nice way to implement transformations within one file, without the need of (nearly) no other implementations or stuff. Here's how I did it:

I have a class called Transform. This holds a position, an orientation, a scale and is mostly a data holder. Additionally, I created an interface called Transformable. If this would be a class, I should have implemented the state (which now is in Transform class) in this class. But it's an interface. My gameobjects implement this interface, so I would have to implement all those move(vec3 amount)-etc-methods in this implementation. With default methods, I can now provide implementations on the interface tiself - combined with a pattern I don't know the name for anymore, this could be powerful: methods implemented by the interface can be called by the interface. This means I can use a non-default-implemented method getTransform() on the interface in my default methods.

For all classes that implement Transformable, it's sufficient to provide a transformation, because it needs getTransformation() to be implemented. That's because interfaces are not allowed to have state and one has to add the field to the implementing class.

Where this really shines is in situations where you would use (method)-pointers in C++: When you have an object besides your regular game objects that has to be a transformable, but is attached to another object that controls their transformation, you can implement the getTransform()-method with returning a field object's transform. Best example is for gameobjects that have a physics componentn attached, that should win every transformation war.

Additionally, I made the gameobject interface a subclass of my transformable interface, so that I can have different entity types for game objects, lights, environment probes etc. Some of them are not movable, like a directional light - therefore, I can override the directional light's move-methods to not do anything. And then, the subclass interface can call it's superclass interface's methods, too: For example the default implementation of the game object interface's isDirty()-method uses the superclass interface's hasMoved()- and its own animationHasPlayed()-method, if it's an animated entity.

So for the maximum price of a method call, that the interface has to do on itself when a transformation changes, you can have your transformations interfaced. My experience is, that I have very few problems with undesired class hierarchies in the means of "oh no, now I have to implement x or subclass y, but it's not clean code". As always, I'm still a bit uncertain about if this is good use of default methods. But at least I gave them a try and I don't regret my design descision - let me know why this stuff is bullshit, I'm curious :)

Copy textures in OpenGL

Often, people need to postprocess textures, like with a blur or something. While it's sometimes possible to render to and sample from the same texture in OpenGL, it's not recommended, as long as rendering and sampling uses the same mipmap level. However, some cards and drivers let you do exactly this, but I guess most of the times you want to use kernels, you're screwed, because pixels are processed in parallel.

One common approach is to use somthing that is called ping-ponging. You bind the texture to sample from to a texture unit and render to another texture. However, all other application components have to be aware that your fist texture doesn't contain the result they need, thus have to use the other texture, means the other texture handle id. This is sometimes very inconvenient and I didn't want to clutter my code - so I checked an alternative approach that modern OpenGL provides us: copy textures.

With earlier OpenGL version, you had to do a fullscreen quad render pass or a framebuffer blit to duplicate textures, with version 4.3 you can do the equivalent of a memcopy. I duplicated my texture, set my source texture as a color attachment of a temporary rendertarget and set the duplicated texture to a texture unit for sampling. My method looks like (Java, used lwjgl):


I modified the code so that it doesn't take a texture object but the attributes you know from OpenGL. Copying a 1280, 720-Texture takes around 0.2 ms on my GTX 770. I'm pretty sure it doesn't take much more time for a larger texture, but if you want me to test it, just leave a comment. Or if you need additional explanations. Somewhere I saw people having trouble with this simple functionality and most of the times it was because their textures were incomplete. That's why I added all those filter attributes etc.

Bound

Was ist Bound?

Bound ist ein erzählendes 2D-Jump'n'Run, das ich zusammen mit einigen Kollegen als Studienprojekt begonnen habe und nun als Hobbyprojekt weiterführe. Da es in den kommenden Posts für das ein oder andere Beispiel herhalten wird, ist es vielleicht schön mal ein Bild davon zu sehen.

Gemacht haben wir es mit XNA 4.0, verwenden Farseer Physics, Mercury Particles, Irrklang Audio und ... nein, das wars glaub ich. Die Engine haben wir von Grund auf selbst gebaut. Wir haben Animationsklassen, Physik-Objekte, ein Eventsystem, ein generisches Layer-System, haufenweise Effekte, Gamestate-Funktionalitäten... uuund einen überkrassen Editor, mit dem wir Level bauen und Spielabläufe skripten können (sc ist der King!). Unserer Kreativabteilung belieferte uns mit schicken Grafiken, Storyboards und schönen Rendersequenzen, die die Geschichte des Spiels erzählen. Unsere letzten Errungenschaften sind einige Effekte, wie die Godrays, die im Bild zu sehen sind und deren Implementierung ich wahrscheinlich im nächsten Post vorstellen werde, ein Lua-Hook zum Einbinden von Skripten.. und überhaupt ist immer alles im Umbruch und noch mehr im Kommen.

Spielen tut man übrigens die niedliche kleine Spinne, die ihr im Bild seht. Die Animationen sind der Hammer, vielleicht gibts irgendwann mal ein Video.

Einfache SSAO-Implementierung (XNA 4.0)

Wozu SSAO?

Normalerweise gibt es in Spielen nur lokale Beleuchtung, das heißt Licht wird für einzelne Punkte von Flächen ausgewertet. Dabei ist nur wichtig, ob ein Strahl einen Punkt trifft, so etwas wie eine Verdeckung, also eine Wechselwirkung von Flächen und Punkten untereinander spielt für dieses Modell keine Rolle. Wenn man aber mal in die echte Welt schaut, ist die Beleuchtung um einen Gegenstand, der auf einer ebenen Fläche steht und Beleuchtet wird nicht sauber, sondern in der nähe des Gegenstandes stellenweise dunkler. Mal ganz abgesehn davon, dass Flächen selten wirklich glatt sind - die meisten Materialien sind rau und uneben. Screen Space Ambient Occlusion ist deswegen aus vielen Gründen eine richtig feine Sache: Erstens ist es so einfach, dass sogar ich es ansatzweise verstanden habe (hoffentlich) und zweitens wertet es das Gesamtbild eines Videospiels als Pixelshader merklich auf, ohne dass die restliche Architektur des Spiels groß oder überhaupt angefasst werden muss - es funktioniert also mit fast jedem Spiel. Der dritte Grund ist, dass der Shader eventuell sogar noch um indirekte Beleuchtung (SSIL) erweitert werden kann, dazu aber in einem späteren Post mehr.

Wer verstehen möchte, wie man SSAO implementieren kann, der sollte Schritt für Schritt den Post lesen und versuchen den Code zu verstehen. Die Kommentare sollten dies einfach machen. Wie bei allem heutzutage gilt: Angucken und Können ist nicht - es braucht ein bisschen Zeit und Konzentration um zu verstehen und Übung um es zu implementieren. Wer von Computergrafik absolut keine Ahnung hat, der braucht nicht weiterlesen, weil Begriffe fallen werden, mit denen er nichts anfangen kann. Wer schon mal einen Shader geschrieben hat und ansatzweise versteht worum es geht, der kann mit den folgenden Absätzen sicherlich etwas anfangen.

Wie funktionierts?

Ich steh nicht so auf pure Theorie, daher ist meine Umgebung abgesteckt: XNA 4.0 und HLSL, weil ich damit grade n Projekt machen "musste". Wir brauchen sonst nicht viel, um SSAO mit HLSL zu implementieren: Den Depthbuffer, die Colormap und sowas wie ne Noisemap. Außerdem integrieren wir unsere Normalen. Das wars schon. Meine Implementierung basiert übrigens auf mehreren Internetblogs, zum Beispiel diesem hier. Grob gesprochen nehmen wir jeden Pixel des Bildes und betrachten den Tiefenwert der umliegenden, nahen Pixel, die wir in einer Halbkugel um die Normale des Pixels zufällig auswählen. Sind diese weiter vorne als der Pixel, ist davon auszugehen, dass der ursprüngliche Pixel räumlich hinter ihnen liegt, also von ihnen verdeckt wird. Je nach dem, wie groß die Strecke zwischen dem aktuellen Pixel und einem umliegenden ist, kann man hier eine Verdeckung sichtbar machen.

Schritt 1: Samples holen, Zufall schaffen

Es kommt natürlich ein bisschen darauf an, wie ihr euer Spiel baut - ich habe eine Deferred Rendering Engine aus irgendeinem XNA-Tuto nachgebaut und liefere meine Maps als Texturen an meine Posteffekt-Shader.

float depth = tex2D(depthSampler,input.TexCoord).r;

float3 color = tex2D(colorSampler,input.TexCoord).rgb;

float3 noise = tex2D(noiseSampler,input.TexCoord).rgb;

float3 norm = tex2D(normalSampler,input.TexCoord).rgb;

Jetzt brauchen wir noch so etwas wie einen Zufall. Ich habe hier ein paar Random-Vektoren in ner Kugel mit 1er Durchmesser.

float3 pSphere[16] = {float3(0.53812504, 0.18565957, -0.43192),float3(0.13790712, 0.24864247, 0.44301823),float3(0.33715037, 0.56794053, -0.005789503),float3(-0.6999805, -0.04511441, -0.0019965635),float3(0.06896307, -0.15983082, -0.85477847),float3(0.056099437, 0.006954967, -0.1843352),float3(-0.014653638, 0.14027752, 0.0762037),float3(0.010019933, -0.1924225, -0.034443386),float3(-0.35775623, -0.5301969, -0.43581226),float3(-0.3169221, 0.106360726, 0.015860917),float3(0.010350345, -0.58698344, 0.0046293875),float3(-0.08972908, -0.49408212, 0.3287904),float3(0.7119986, -0.0154690035, -0.09183723),float3(-0.053382345, 0.059675813, -0.5411899),float3(0.035267662, -0.063188605, 0.54602677),float3(-0.47761092, 0.2847911, -0.0271716)};

Als nächstes generieren wir Normalen, die wir später zur Reflektion verwenden. Hier kommt die Noisemap zum Einsatz. Für jeden Bildpunkt kriegen wir also eine zufällige Normale. Das Noise Sample kann noch ein zusätzliches Offset kriegen.

float3 fres = normalize(noise*2) - float3(1.0, 1.0, 1.0);

Meine Quelle speichert sich Fragmentkoordinaten etwas komfortabler ab, daher mach ich das mal nach. Was wir dann in einer Variable haben sind die UV-Screenkoordinaten, also Pixel nach x und y und den Tiefenwert im Bildschirmplatz...den Depthbuffer.

float3 ep = float3(input.TexCoord,currentPixelDepth);

Außerdem macht er den Radius wo wir samplen von der tatsächlichen Tiefe abhängig. Je größer die Tiefe, desto größer der Radius (glaube ich :) ). Wahrscheinlich, damit man einen stärkeren Effekt im Hintergrund hat, der ist ansonsten schlecht wahrnehmbar. Für rad wählt meine Quelle den Wert 0.006. Kann man machen, rumspielen hilft.

float radD = rad/currentPixelDepth;

Schritt 2: Die Schleife

Nun müssen wir für jeden Pixel des Bildes einige Schritte durchführen. Bevor wir die Schleife schreiben, definieren wir mal unsere benötigten Variablen und die Variable, wo wir das Ergebnis speichern.

float bl = 0.0f;

float occluderDepth, depthDifference, normDiff;

Wie oft die Schleife durchlaufen wird, muss man selbst entscheiden. Je mehr desto besser ist die Qualität des Effekts. Bei mehr als 16 Samples braucht man logisccherweise mehr Zufallsvektoren in pSphere.

for(int i=0; i<9;++i)

   {

      // Einen Vektor in der Kugel holen und reflectieren (mit dem Noise-Vektor)

      float3 ray = radD*reflect(pSphere[i],fres);

      // Wenn der Strahl aus der Halbkugel rausgeht, wird Richtung geändert

      float3 se = ep + sign(dot(ray,norm) )*ray;

      // Tiefenwert des verdeckenden Bildpunktes holen

      float3 occluderFragment = tex2D(depthSampler,se.xy);

      // Normale des verdeckenden Bildpunktes holen

      float3 occNorm = tex2D(normalSampler,se.xy).rgb;

      // Wenn die Diff der beiden Punkte negativ ist, ist der Verdecker hinter dem aktuellen Bildpunkt

      depthDifference = currentPixelDepth-occluderFragment.r;

      // Berechne die Differenz der beiden Normalen als ein Gewicht

      normDiff = (1.0-dot(occNorm,norm));

      // the falloff equation, starts at falloff and is kind of 1/x^2 falling

      bl += step(falloff,depthDifference)*normDiff*(1.0-smoothstep(falloff,strength,depthDifference));

   }

Der letzte Schritt dient, um den Verdeckungswert weich abfallen zu lassen. Wer genauer wissen möchte, was passiert, kann hier schauen. Für falloff wählt meine Quelle den Wert 0.000002. Hab ich genauso.

Schritt 3: Fertigmachen

Nach der Schleife wird der Verdeckungswert nochmal mit einem Steuerwert für die Stärke multipliziert und durch die Samples geteilt. strength ist bei "uns" 0.07, totStrength 1.38. Prinzipiell würde es reichen, wenn wir einen float zurückgeben, aber ich hab den Effekt später noch für was anderes nutzen wollen, daher is mein Rückgabewert etwas verschwenderisch.

float ao = 1.0-totStrength*bl*invSamples;

return float4(ao,ao,ao,ao);

Das war ja einfach

Das wars auch schon. Wieviele Zeilen Code waren das? 15? Diese Technik ist nicht völlig frei von Fehlern. Wenn man ein bisschen zu viel am Radius spielt, kriegt man seltsame Ergebnisse und die Nachteile des Viewspace (nur was auf dem Bildschirm sichtbar ist, wird beachtet...) lassen sich auch nicht verneinen. Ich hab bei zu großen Unterschieden bei den Tiefenwerten (?) ein hässliches Kantenbluten gehabt, das ich auch mit einem zusätzlichen Blur-Pass nicht wegbekommen habe (in den Bildern unten erkennbar). In XNA 4.0 hat man außerdem nur Shadermodel 3.0, das bedeutet limitierte Instruktionszahlen. Gerade wenn man den Effekt noch um indirekte Beleuchtung erweitern will, macht sich das sehr schnell bemerkbar. Ein zusätzlicher Blurpass sollte also fast schon eingeplant werden, wenn man SSAO irgendwie ins XNA bringen will. Ich habe hier jetzt keinen, das Ergebnis kann sich trotzdem sehen lassen:

mit SSAO

ohne SSAO

SSAO

Von meiner Seite aus ein allgemeines Danke an alle Leute bei Nvidia, in den Blogs, meine Quelle und auch sonst jeden, der Informationen zum Thema ins Internet gestellt hat, denn er hat diesen Post mit Sicherheit irgendwie beeinflusst.

Viel Spaß beim Nachmachen, für einen der kommenden Posts habe ich noch eine kleine, aber effektive Erweiterung des Shaders. Wenn irgendwas hier falsch oder zu ungenau ist, kommentiert einfach, die Chance ist hoch, dass ich mich Kommentaren annehme :)