Two-phase occlusion culling (part 2) - instance aware culling

I described the basics about the two-phase occlusion culling technique in the first part post, but there is one thing, that doesn't work well with this simple setup: Instancing or sometimes viewed as batching.


The problem is very simple, the solution however is not. Only a few engines managed to implement the technique, but it is unavoidable if one needs to render large batches or large amounts of instances. Quick recap, under the assumption that we compromise flexibility in favor of speed and use OpenGL 4.5 level graphics APIs:

Avoid buffer changes one, two different vertex layouts (for example for static and animated meshed)

Avoid program changes well defined firstpass program (deferred rendering) or subroutines in a global attribute buffer

Minimize drawcalls Usage of instanced and indirect rendering


The usage of draw commands and indirect rendering is nice enough. The amount of drawcommands is reduced by using instanced rendering for entities that have different uniform attributes but the same vertices and indices. This results in the command and entitiy buffers of the following form:

Entities
uint entityId	678
mat4 transform	000...
uint materialIndex	18
uint entityId	1566
mat4 transform	000...
uint materialIndex	2

Commands
vertexOffset	0
indexOffset	0
indexCount	99
instanceCount	2
vertexOffset	1200
indexOffset	99
indexCount	33
instanceCount	3

Firing indirect rendering call is now done with a count of two, because we have two commands. 5 objects will be drawn. The shaders are invoked several times and the entity data can be accessed with an index and fetched from the entities buffer. The resulting indices in the shader will look like

Shader Invocations
DrawId	InstanceId	Entity buffer index
0	0	0
0	1	1
1	0	???
1	1	???
1	2	???

As mentioned in the previous blog post, there's no chance for us to know the entity index when the second command is executed, because the current offset is based on the amount of instances the previous commands draw. The solution is to use an additional offset buffer with size of the draw command buffer. For every command, we set the offset to the appropriate value when creating the commands on the cpu side. With instance based culling, this problem intensifies, because the cpu doesn't know the offset anymore. The calculation has to be done on the GPU now. My solution is still based on vertex shader kernels, but I will tell you later why this is probably problematic. First how it is done, because conceptionally, it will be the same for compute shaders. The layout now looks like this:

Shader Invocations
DrawId	InstanceId	Entity buffer index (command offset + InstanceId)	Offset of the command
0	0	0	0
0	1	1	0
1	0	2	2
1	1	3	2
1	2	4	2

The first step is to determine the visibility for every single instance. Vertex shader kernels have an advantage here, because they can be arbitrarily large (compute shader groups are limited to 1024 or so). A single draw call can determine visibility for dozens of thousands of instances or entities. Combined with instancing, we can use DrawId and InstanceId to index into the command buffer and offset buffer (DrawId) and into the entity buffer (offset + InstanceId). Since the same kernel sizes are applied to every command, invocations are wasted if few commands have many more instances as others. So you might want to launch draw calls without instancing, one per command, which could be faster here.
The visibility is again stored into the visibility buffer, which has to be as large as the entity buffer now. With non-instanced culling, size of the command buffer was sufficient. One important thing has to be done now: Every visible entity thread has to increase a counter that is associated with the corresponding command - since this is done per invocation, atomic operations are needed. So we need another int buffer (just like the offset buffer), that holds the visible instances count per command.

The reason why this is so important is, that this is the bit of information that is used in a second step to append entity data and draw commands in parallel. This is done by an algorithm similar/equal to parallel prefix sum - you can google it. Tldr: Each command has to know how many visible instances all previous commands produce in total, so that it can append its own instances there.

I call the seconds step appending step, while the first one is the visibility computation step, that actually calculates the visibility of an instance. For a massive amount of instanced commands, one would probably want to launch a two-dimensional kernel of the size n*m where n is the amount of commands and m is max(maxInstanceCount, someArbitraryValue) or something. Again, with vastly different instanceCounts per command, multiple shader calls could give a benefit.

So now we need a target buffer that holds the entity data, a target buffer for the commands and a target buffer for the entity data offsets. Additionally, we need an atomic counter for the target command index and then an atomic counter per command, that is the current index of the entities per command. Each shader invocation has its own command index in the DrawID built-in and the instance index in the InstanceID  built-in. So we can calculate all information that is needed on the fly:

struct oldCommand = read from the bound command buffer with DrawID
uint  oldCommandEntityOffset = read from the oldOffsetBuffer with DrawID
uint oldEntityIndex = oldCommandEntityOffset + InstanceID
uint visibilityBufferIndex = oldEntityIndex
uint targetCommandIndex = atomicAdd(commandConuter, 1)
uint targetInstanceIndex = atomicAdd(commandCounters[targetCommandIndex], 1)
uint targetEntityIndex =  sumOfVisibleInstancesOfAllCommandsBeforeCurrentCommand

The current entity data can then be written to the targetEntityDataBuffer, as well as the offset for the instance/command. The command can be written by the first active thread (InstanceID == 0). The resulting buffers contain only the visible instances, offsets of the visible instances and drawcommands that can be used to only draw exactly those instances.

 Here's a video demonstrating occlusion and frustum culling with this technique:

Bonus: For the visibility computation step, it's nice to launch a vertex shader kernel, since the kernel can be arbitrarily large in two dimensions at max. Since there's no shared memory involved, this will probably perform better than the compute shader equivalent, maybe at the same speed, but it shouldn't be slower. The appending step needs an atomic operation per invocation, because every invocation increments the counter if the corresponding instance is visible (which the other invocations couldn't know). Instead of "global" shared memory, one could use shared memory of a compute shader group. Shared memory in a group is much faster than atomic operations between a gazillion vertex shader invocations, so I will implement this at some time and may be able to show a performance comparison.

Multithreaded game engines and rendering with OpenGL

There is this one question since beginning of multicore processors: How to multithread your game engine? I can't answer this question in general, but I can answer the question How to multithread the rendering part of my game engine?

Problem

The most simple case is a single-threaded game engine that does this:

 while(true) {  
   simulateWorld();  
   renderWorld();  
 }  

The update step produces a consistent state of the world. After each update, the rendering is done. After the rendering is done, the next cycle is processed. No synchonization needed, everything works. The bad thing: simulateWorld is a cpu-heavy task, renderWorld is a gpu-heavy task. So while the world is updated, the gpu idles, while the gpu renders, the cpu idles (more or less). And the worst thing: Your framerate is limited by both the cpu and the gpu. Your frametime is timeOf(simulateWorld) + timeOf(renderWorld). Our target frametime is timeOf(renderWorld) and to keep this time as low as possible. More precisely the target is timeOf(renderWorld).

Foundations

Okay I lied: The above mentioned problem is only the overall-globally-problem we solve. There's a shitload of other problems, that mostly depend on the language platform, graphics api, hardware, your skillset and so on. Since our lifetime is limited, and I want to give practical, real-world-relevant help, I make some assumptions. One of these is, that you use OpenGL and a pretty new version of it. Let's say 4.3. Multithreaded rendering could probably be done better and faster with DX 12 or Vulkan, but let's stay cross-platform and simpler with OpenGL.

I assume that you already have at least a little bit knowledge about how game engines work, timesteps, scene representations and so on.

Multiple gpu contexts

In order to be able to issue commands to OpenGL, you have to use a thread that has an OpenGL context bound to it. Architecture-wise, there is the CPU, the OpenGL driver and the GPU. The driver holds a queue, where the commands from the SPU are buffered somehow (implementation dependend). Then again, there is a queue for the GPU - that again buffers commands that came from the driver (implementation dependend). First, all commands the CPU issue are asynchronous, which means there's no actual work done on the GPU yet, but the command is queued by the driver. The order is preserved, that's what you can count on. The implementation can decide how many commands are queued up, until the cpu thread will block on command issuing. And the implementation can decide how many commands are buffered on the GPU side of things.

Now, having that said, let's think about multithreading. What we can control is the CPU side (unless you write your own driver, which I doubt you want to do). Multiple cpu threads would require multiple gpu contexts - one for each thread. While this is theoretically possible with OpenGL, it is a very dumb idea in 19 out of 20 cases. As with traditional multithreading, context switching comes with a very high overhead - so you have to ensure that context switching overhead doesn't eat up the performance you gained from using multiple threads. And here's what's wrong: It seems that all OpenGL drivers (where you issue your comamnds at) except the one for iOS, are implemented singlethreaded. That means you can issue commands from multiple cpu threads, but on the driver thread, those commands are processed sequentially, with a context switch in between. You can find a more detailled explanation here, so that I don't have to lose many further words, but: Don't use multiple contexts for anything else than asynchronous resource streaming.

I assume we only use one OpenGL context. That means, all OpenGL calls must happen on a single thread. The construct used for this is the command pattern. A few hints: You should issue as few commands to OpenGL as possible. You should issue as lightweight commands as possible. Obviously, using only one thread for command execution limits the amount of work that can be done by this (cpu) thread. Nonetheless, you want a single, inifintely running worker thread that owns your OpenGL context and takes commands out of a non-blocking command queue. For convenience, you can seperate commands that should return a result (hence block) or should just be fired and forgotten. It's not nessecary that all commands are properly implemented by yout context wrapper/worker thread class, even though it makes your application more predictable in terms of performance. I implemented a non-blocking, generic command queue in Java here that uses Runnables and Callables as Commands. The principle is also explained more detailed here.

 Extractor pattern

Extractor pattern is a term one never finds on the internet when searching for information about rendering. The first time I heard this, was when a (professional graphics programming) collegue of mine explained the basics of renderer architectures to me many many years ago. Fancy name for a simple thing: If you share data between two consumers (not exactly threads in this case), you either have to synchronize the access somehow, or you extract a copy of the data to pass it to the renderer in this case. That implies, that you need a synchronized/immutable command-like data structure, that represents all the information you need to push a render command. That plays nicely with the chapter above, you know, command queue etc. In languages other than Java where you have true value types, the implementation could be easier - although having very large renderstate objects copied over and over might be a drawback here. The chapter below shows another way of handling this problem.

Multibuffering

Let's assume that the GPU only consumes renderstate, but doesn't alter it. That means if the GPU calculates any fancy physic, or results that should be passed to the next render cycle, than it isn't seen as part of our traditional renderstate for now.
That means the only producer of renderable state is the CPU. Since we don't want to render a scene where the first half objects are in timestep n and the second half is still in n-1, we have to be able to access a state of our gamestate, that is concise somehow. This is easily achievable with a task-based architecture, that basically has only one update thread, that sequentially crunches down what you want to do in parallel. For example if you have 10 game objects, the updatethread pushes 10 update-commands into a pool of worker threads and after all of them returned a result, we have a concise world state we can use and used all threads our system can push. I don't think, there's an alternative approach to multithreading that can be used in a general purpouse game engine, but I may be wrong. Instead of creating a new renderstate object and pushing a command to the renderqueue, we use something that's known in multithreading environments for ages: multibuffering.

This should be confused with double or triple buffering that is used to display images on your monitor, although the principle behind it is the same.

I skip explaining why double buffering isn't enough to satisfy our needs. Just use triple buffering and be happy, if you can afford the additional memory consumption. This works the following way.

You have three instances of your renderstate. You have a wrapper, that encapsulates these three instances. Instance A is the currently readable state. The renderer uses this state for rendering, so while rendering is in progress, this instance mustn't be touched by anyone else than the renderer. Instance B is the current staging copy of the renderstate. It is the next state, that the renderer will use for the next frame, after the current frame is finished. Copy C is the isntance the update is currently applied on.
Now while this sounds simple, the important thing is, how to swap those instances correctly. Our main purpouse is to keep the GPU busy and to be able to render maximum fps, although one could theoretically (and practically) limit the framerate somehow (vsync, fps cap etc.).

I realized this with a ever-running thread that does push and wait for a render-scene-drawcall constantly. After the frame is finished, the read copy is swapped with the staging copy. It's important, that the renderer get's a fresh copy on every swap. If the update thread isn't fast enough to provide a new updated renderstate copy, the renderer would render a state that is older than the one just drawn. This will cause flickering objects. I realized this with an ongoing counter, and I prevent swapping to a state with a lower number then the current state has. Let's assume your engine and the machine is capable of pushing 60cps update and 60fps framerate.
Let's face the other swap: The update thread updates the write copy C constantly as fast as possible. When finished, the write state becomes the new staging state B, so we have to swap them. We can't do this if the renderer is currently swapping his read and the staging state. So we need a lock for swap(A,B) and a lock for swap(B,C). If rendering can be done with okayish framerates (which we assume), the updatethread can wait (block) until swap(A, B) is finished. If you are using only coherent memory access, than you're donw here. But you wouldn't search for state-of-the-art rendering, if you don't use incoherent access. So let's get to the final ingredient we need for our super fast renderer.

Lowering draw calls and persistent mapped buffers synchronization

At the very beginning, there is the need to reduce the amount of work the gpu has to do, in order to render your scene. There are paths thorugh the OpenGL API that are more expensive than others, and there are paths with very very small overhead, if you can afford the loss of flexibility that comes with using it. This can give you a rough overview about how costly API calls are. I suggest we skip a long explenation and you read this. Summed up: You want unsynchronized buffers for all your data and handle synchronization by yourself. This results in no driver or gpu work for the synchronization, which is crucial for maxing out your performance. Remember that Vulkan was designed to exactly kill the driver overhead of OpenGL. Furthermore, we reduce draw calls and state changes with bindless resources and one of the most important things: indirect and instanced rendering.

Having all these things implemented, there's one last thing to add to our triple buffering: synchronization. Since the driver works asynchronous and the gpu, too, there's no guarantee that the gpu doesn't use the buffer of our current write copy, that the update threads currently writes to. To adress this, one has to insert a fence sync object into the pipeline and associate it with the state, whenever a state becomes the current write state. When the render thread uses this thread and pushing all render commands of the current frame are pushed, the fence signal is pushed. Whenever this signal is processed by the gpu, the update thread can write to the associated state without any problems. The update thread can prepare the renderstate update and wait until the prepared data can be applied. This effectively blocks the update thread, if the renderer is more than 2 frames behind the update thread.
Special care has to be taken if you have any timestep-dependent calculations in your rendeirng step: Since rendering is decoupled form the update-step, you have to calculate the timestep by yourself somehow in the renderthread, based on the last tick and the time passed since then.

That's it

•  Using the fastest possible paths through the OpenGL API. You need shader storage buffers and indirect (and inntanced) rendering, probably combined with bindless textures
• Model a renderstate class that can be your complete draw command
• Run a thread permanently, that permanently pushes rendercommands

Voxel Cone Tracing Global Illumination in relatime

Quite a while ago, I implemented Voxel Cone Tracing global illumination for my own custom engine. This technique can be quite impressive. Used Java, OpenGL, running on a GTX 770 desktop GPU.



Here's another video of the famous cornell box, where I added a different diffuse tracing calculation. Same engine (newer version), Running on a GTX 1060 mobile gpu.


And another run with a point light only configuration on the mobile gpu.

Rendering 4 Mio. vertices with 340000 cubes in Java

Someone has to fight the battle against the "Java is so slow, all Java games have low fps" myth. Maybe Java is not the best language for game development because it lacks value types and easy and zero overhead native code integration .... but the results one could achieve with using a zero-driver-overhead path like modern graphics APIs recommend for so many years, can be on par with what you can achieve in C or C++ as in Unreal, CryEngine or Unity. The secret is: Bindless ressources, indirect and instanced rendering, persistent mapped buffers, direct state access and of course good old multithreading.

So, the most important things first: Use indirect rendering (to minimize calls the cpu has to issue!) and a shared global vertex buffer (to reduce state changes). Use large uniform or shader storage buffers for your object's properties and material properties. For each object, push a command into the buffer. Massive object counts cause massive command counts and large command buffers, so one can now use instancing to further reduce command count. With a clever structure (a offset buffer for example), one can easily have unique properties per object instance (for example dedicated textures per instance!) sourced from a large uniform buffer, when it's okay to share a common geometry (vertices). Et voila, 2ms cpu time to fire a render command that draws 340.000 instanced cubes with 4 million triangles at 60 fps in Java. Each object can have it's own textures and properties etc.

Bindless texture with LWJGL

The classic bound texture paradigma is so 1990. But getting bindless textures to work is not that easy, especially when you're using something other than C or C++. For all the LWJGL or Java users out there, here are some notes that may help you removing the classic texture pipeline from your engine. I think you can read about the technique in the internet, so I'll keep it mostly short.

First of all, you need a handle (some kind of pointer again, but don't think of it as a pointer, yucks) for a given texture.

 long handle = ARBBindlessTexture.glGetTextureHandleARB(textureID);  

Afterwards, the handle has to be made resident. I think that's something you need for the combination with partially resident textures.

 ARBBindlessTexture.glMakeTextureHandleResidentARB(handle);  

This was the easy part. Now, you have to use the handle in our shaders somehow. The easiest way would be to use it as a uniform.

 ARBBindlessTexture.glUniformHandleui64ARB(location, handle);  

Inside your shader, you have to use the handle. But what datatype should one use? Maybe I missed something elementary, but  there's only one proper way, namely use a datatype made available through an nvidia extension. Since bindless textures are available through an extension as well, here are both calls that you (probably) need:

 #extension GL_NV_gpu_shader5 : enable  
 #extension GL_ARB_bindless_texture : enable  

And now, you can use tha datatype uint64_t. So your uniform would be a uint64_t.

That would work. But most probable, you want to have your data in a uniform or storage buffer, probably together with some other data and datatypes. So here's what I did.

Use a DoubleBuffer (Java native buffer, available via BufferUtils.createDoubleBuffer(int size)) for your data. Since doubles are twice the size of a float, which is 4 byte, we have 8 bytes per texture handle, which is 64 bits, which is the same as a uint64's size, so that's enough. Now one has to take the generated handle's bits and put them into the buffer (for example a ssbo) as they are. This can be done like so:

 GL15.glBufferSubData(GL43.GL_SHADER_STORAGE_BUFFER, offset * primitiveByteSize, values);  

-
Where primitiveByteSize is 8 in this case. Since we use the underlying buffer as a DoubleBuffer, we have to provide a double value for the handle (or use it as a byte buffer, but nevertheless we need the correct bits or bytes). You can convert a Java long to and from a double like this:

 Double.longBitsToDouble(longValue)  

 Double.doubleToLongBits(doubleValue)  

Afterwars, the shader can take the value as a said uint64_t and cast it to a sampler. Sounds ugly, maybe it is.

 color = texture(sampler2D(uint64_t(material.handleDiffuse)), UV);  

That is the whole story, took me a while to figure it out.

Single Pass Omnidirectional Shadow Mapping Evaluation

Layered rendering is possible since geometry shaders entered the OpenGL stage. The rough idea of a good omnidirectional shadow mapping (for example for pointlights) is to render the complete shadow map with a single draw pass, in order to reduce the amount of rendercalls, compared to six-pass rendering with traditional rendering to a cubemap. Therefore, the geometry shader emits a vertex for each incoming vertex of the non-culled scene geometry to a face of the cubemap with something called layered rendering. While the idea is well described all over the internet already, every now and then, there is discussion about the efficiency of this method.

My first idea was to evaluate omnidirectional shadow mapping with dual paraboloid shadow maps since it's the easiest way to achieve pointlight shadow: No viewmatrices, no projectionmatrices, two textures per pointlight, let's go. Without view frustum culling, I draw the complete geometry twice and passed a "backside" flag as a uniform variable for the second rendering. No layered rendering, just two depth buffers and two color attachments (can be ommited if no variance shadow mapping is used) for each pointlight (front and back). It's possible to do dpsm with a single-pass rendering in a single texture - I doubt it to be efficient because you would have to handle the depth buffer somehow. Or it will be efficient, but much more complicated.

Rendering a single (two texture) shadowmap for the famous sponza atrium takes ~1.5 ms on my GTX 770.

Layered rendering however, took me a while for the implementation. Because I use array textures in order to be able to use many shadow mapped lights, I had to fight with the strange array indices for cubemap arrays. For everyone who is interested, here's how you create a cubemap array rendertarget and use it to draw your shadow maps:

 framebufferLocation = GL30.glGenFramebuffers();  
 // Create rendertarget
 GL30.glBindFramebuffer(GL30.GL_FRAMEBUFFER, framebufferLocation);  
 IntBuffer scratchBuffer = BufferUtils.createIntBuffer(colorBufferCount);  
 for (int i = 0; i < colorBufferCount; i++) {  
      GL32.glFramebufferTexture(GL30.GL_FRAMEBUFFER, GL30.GL_COLOR_ATTACHMENT0 + i, cubeMapArrays.get(i).getTextureID(), 0);  
      scratchBuffer.put(i, GL30.GL_COLOR_ATTACHMENT0+i);  
 }  
 // Use glDrawBuffers(GL_NONE) if you don't need color attachments but depth only
 GL20.glDrawBuffers(scratchBuffer);  
 CubeMapArray depthCubeMapArray = new CubeMapArray(AppContext.getInstance().getRenderer(), 1, GL11.GL_LINEAR, GL14.GL_DEPTH_COMPONENT24);  
 int depthCubeMapArrayId = depthCubeMapArray.getTextureID();  
 GL32.glFramebufferTexture(GL30.GL_FRAMEBUFFER, GL30.GL_DEPTH_ATTACHMENT, depthCubeMapArrayId, 0);  

Although some implementations are hidden because of my framework, the idea should be clear. Don't forget to initialize the cubemap array texture correctly or your framebuffer object won't be complete.

Since the geometryshader decides which face to render to, you can pass the pointlight index as a uniform variable. The layer will then be gl_Layer = 6*lightIndex + layer.

The quality is very nice, but unfortunetly, rendering sponza into the cubemap takes ~8.3ms on my GTX 770. The additional color attachment is not responsible for the expensiveness, since I tried to remove it. I'm pretty sure that I didn't do a major mistake in the implementation. It has to be the poor performance of the geometry shader that is responsible for the high amount of time this method takes.

Conclusion

• Dual paraboloid shadow map rendering ~1.5 ms
• Single pass cubemap shadow map rendering ~ 8.3 ms

I used 512*512 per dspm face or cubemap face. Would be nice to hear anyone alse's experience with omnidirectional shadowmapping.

Get normal for cubemap texel

Since I came across this problem and wasn't able to find an easy solution on the internet, I decided to write s small recipe to calculate normals when you want to do manual mipmapping/radiance convolution with cubemaps in OpenGL. I use compute shaders, so for geometry/vertex/pixel-pipeline, you could use layered rendering and other stuff.

First of all, the shader needs the current cubemap face index as a uniform variable. I recommend using the standard OpenGL indices (see link below).

Most likely, you are using the standard cubemap layout. If this is not the case, you have to change the vectors in my code. So with a given face index and a given texel position, the problem can be solved:




What happens here is that I calculate the pixel position in texture space with the help of the invoation position. The compute shader is invoked with cubemapfaceResolution.x/16, cubemapfaceResolution.y/16, 1. Knowing which (OpenGL) world axis the view direction of the virtual camera, facing the current cubemap side from the inside (cubemaps origin) is, the other two axis are the two orthogonal axes. These two axes' values grow with the texelcoordinates we already have. But therefore, they have to be remapped from 0 - 1 to -1 - 1. The resulting vector can be used to sample a cubemap as it is. Normalization could be unnecessary.

Compute shader advices

Recently, I had a lot of pleasure with OpenGL's compute shaders. With this lot of pleasure came a lot of pain because I made some (rookie) mistakes. So I wanted to share my experience and some advices I have, just in case you have troubles too:

• The first thing you should check are your texture formats! No, really, double check it, don't repeat my mistakes. In your compute shaders, you could use your images (not textures) with
  
  
  glBindImageTexture(unit, textureId, 0, false, 0, GL_WRITE_ONLY, GL30.GL_RGBA16F);
  
  
  of OpenGL version 4.2 as an output texture. Of course you could use GL_READ_ONLY or GL_READ_WRITE if you use the texture differently. Also keep in mind that this call binds an image, not a texture. And that's why you have to provide a mipmap level you want to attach. I used the wrong format once, namely rgba32f, which my rendertarget attachments didn't have, and it resulted in non existent output from my compute shader. Very frustrating but correct behaviour.
• Keep in mind that you could use your regular textures via samplers in your compute shaders, too. Simply bind the texture and have a  similar line to this in your shader
  
  
  layout(binding = 1) uniform sampler2D normalMap;
  
  That's helpful if you want to access mip levels easily.
• Since even in the OpenGL super bible is a typo that doesn't help to understand the compute shaders built-ins, I recapture them.
  With dispatchCompute you have to provide three variables that are your group counts. A compute shader pass is done by a large number of threads and defining clever group counts/sizes will help you to process your data. In graphics related cases, mostly you will need compute shaders to render to a texture target. So it would be clever to have a certain, two-dimensional amount of threads, wouldn't it? Define your group sizes corresponding to your image size: a 320*320 image could be devided into 10*10 groups, or tiles - and each will have 32*32 pixels in it. So you should define your group size as 32, 32, 1. Now you can dispatch 320/group size threads, which will be 10 groups, for x and y dimension. In your shader, you will be able to use the built-in gl_WorkGroupSize to have this information in every invocation of your shaders main method. To uniquely identify your invocation, you can use the gl_GlobalInvocationID. If you use your shader like I said in this example, this would contain your texel's position the invocation would have to write. And that's how you can use compute shaders to manipulate your textures. Additionally, there is a gl_WorkGroupID, that identifies your tile/group of the invoation, and gl_LocalInvocationID, that is your pixels position in its tile. Sometimes, it could be useful to use a flattened identifier - for example if you have a task that requires performing an action just 12 times, but has to be done in the compute shader - and therefore you can use gl_LocalInvocationIndex. You can use it as a conditional to limit some code paths like
  
  

  if(gl_LocalInvocationIndex < MAX_ITEMS) { processItem(); }
  
  For a better understanding, have a look at this post, which has a nice picture and another explanation of the group layout.

What else? Compute shaders are awesome! I like how easy it is to invoke them, independent of something like the graphics pipeline. Use compute shaders!

Quick look at my OpenGL engine

I just want to share a small screenshot of my OpenGL rendering project. Includes physical based rendering, a global illumination concept daveloped by myself and realtime (glossy) reflections as you can see on the screenshot. Realtime of course - seen on my GTX 770 in full hd at max 300 fps.

Java 8 default methods for your game engine transformations

Although transformation and class hierarchies in game engines are a topic for itself for sure, I finally arrived at a point where I just want every single of my regular game objects to be a transformable entity. Those objects that don't act as something that can be transformed are outside of my interest, they need some default behaviour I don't care about. I think that is the way the Unity engine took, too. A shared super class might look like a good idea, but we are often warned about such kind of class hierarchies.

While C++ offers multiple inheritance to implement things like this very easily, in languages like Java, you probably have to use composition - which should be favored over inheritance nevertheless. The problem is that sometimes, you get lost in interfaces and class fields... and see yourself write interface implementations again and again while delegating interface calls to field objects.

The last sentence is the catch-word: I tried to learn about Java 8's new stuff and stumbled across default methods. While mainly created to guarantee binary compatibility when changing interfaces, they offer a nice way to implement transformations within one file, without the need of (nearly) no other implementations or stuff. Here's how I did it:

I have a class called Transform. This holds a position, an orientation, a scale and is mostly a data holder. Additionally, I created an interface called Transformable. If this would be a class, I should have implemented the state (which now is in Transform class) in this class. But it's an interface. My gameobjects implement this interface, so I would have to implement all those move(vec3 amount)-etc-methods in this implementation. With default methods, I can now provide implementations on the interface tiself - combined with a pattern I don't know the name for anymore, this could be powerful: methods implemented by the interface can be called by the interface. This means I can use a non-default-implemented method getTransform() on the interface in my default methods.

For all classes that implement Transformable, it's sufficient to provide a transformation, because it needs getTransformation() to be implemented. That's because interfaces are not allowed to have state and one has to add the field to the implementing class.

Where this really shines is in situations where you would use (method)-pointers in C++: When you have an object besides your regular game objects that has to be a transformable, but is attached to another object that controls their transformation, you can implement the getTransform()-method with returning a field object's transform. Best example is for gameobjects that have a physics componentn attached, that should win every transformation war.

Additionally, I made the gameobject interface a subclass of my transformable interface, so that I can have different entity types for game objects, lights, environment probes etc. Some of them are not movable, like a directional light - therefore, I can override the directional light's move-methods to not do anything. And then, the subclass interface can call it's superclass interface's methods, too: For example the default implementation of the game object interface's isDirty()-method uses the superclass interface's hasMoved()- and its own animationHasPlayed()-method, if it's an animated entity.

So for the maximum price of a method call, that the interface has to do on itself when a transformation changes, you can have your transformations interfaced. My experience is, that I have very few problems with undesired class hierarchies in the means of "oh no, now I have to implement x or subclass y, but it's not clean code". As always, I'm still a bit uncertain about if this is good use of default methods. But at least I gave them a try and I don't regret my design descision - let me know why this stuff is bullshit, I'm curious :)

Copy textures in OpenGL

Often, people need to postprocess textures, like with a blur or something. While it's sometimes possible to render to and sample from the same texture in OpenGL, it's not recommended, as long as rendering and sampling uses the same mipmap level. However, some cards and drivers let you do exactly this, but I guess most of the times you want to use kernels, you're screwed, because pixels are processed in parallel.

One common approach is to use somthing that is called ping-ponging. You bind the texture to sample from to a texture unit and render to another texture. However, all other application components have to be aware that your fist texture doesn't contain the result they need, thus have to use the other texture, means the other texture handle id. This is sometimes very inconvenient and I didn't want to clutter my code - so I checked an alternative approach that modern OpenGL provides us: copy textures.

With earlier OpenGL version, you had to do a fullscreen quad render pass or a framebuffer blit to duplicate textures, with version 4.3 you can do the equivalent of a memcopy. I duplicated my texture, set my source texture as a color attachment of a temporary rendertarget and set the duplicated texture to a texture unit for sampling. My method looks like (Java, used lwjgl):


I modified the code so that it doesn't take a texture object but the attributes you know from OpenGL. Copying a 1280, 720-Texture takes around 0.2 ms on my GTX 770. I'm pretty sure it doesn't take much more time for a larger texture, but if you want me to test it, just leave a comment. Or if you need additional explanations. Somewhere I saw people having trouble with this simple functionality and most of the times it was because their textures were incomplete. That's why I added all those filter attributes etc.

Einfache SSIL-Implementierung (XNA 4.0)

Wozu SSIL?

Mein vorheriger Post zeigte SSAO als eine einfache Methode zur Bildverbesserung für euer Videospiel und sollte vor diesem hier gelesen werden. Auf Screen Space Indirect Lightning bin ich hier gestoßen - vorher habe ich davon auch noch nie etwas gehört. Aber die Idee ist nicht schlecht: Wenn wir beim SSAO sowieso für Bildpunkte Strahlen in zufällige Richtungen schicken und Tiefenwerte umliegender Pixel zur Verdeckung anschauen, wieso schauen wir dann nicht gleich noch, wie sie sich bezüglich Reflexion von Licht verhalten? Und das ist genau was passiert: Es werden Bildpunkte um einen Pixel gesucht und zusätzlich zum SSAO deren Albedo (Rückstrahlvermögen) betrachtet, um zusätzliches indirektes Licht auf den Bildpunkt zu übertragen wenn die Verdeckung vorhanden ist. So etwas wie indirektes Licht kennt ein lokales Beleuchtungsmodell nämlich nicht.

Schritt 1: Änderungen am SSAO-Shader

Der SSAO-Shader beinhaltet schon fast alles von dem, was wir brauchen. Zu Color-, Depth-, Normal- und Noise-Werten könnten wir noch den Licht-Wert brauchen. Ich habe hier den Vorteil, dass ich eine Deferred Rendering Enginge verwende, da habe ich den Licht-Wert sowieso in einem seperaten Rendertarget und er steht mir als Textur zur Verfügung. Im ersten Schritt kommt also hinzu:

float3 light = tex2D(lightSampler,input.TexCoord).rgb;

Vor der Schleife definieren wir noch eine Variable, wo wir die Information des indirekten Lichts speichern.

float3 resultRadiosity = 0;

In der Schleife kommen vier Zeilen Code hinzu. Zusätzlich zu den Fragmentkoordinaten brauchen wir noch den Lichtwert des verdeckenden Pixels und dessen Albedowert. Die Intensität der Rückstrahlung ist der Farbwert (Vektor), den wir mit Punktprodukt auf einen Skalar abbilden. Der Farbwert des Samples wird je nach Intensität (also Rückstrahlvermögen) auf den ursprünglichen Bildpunkt übertragen. 

float3 occluderLight = tex2D(lightSampler,se.xy);

float3 occluderAlbedo = tex2D(colorSampler,se.xy).rgb;
float intensity = dot(occluderAlbedo,1);

resultRadiosity += step(falloff,depthDifference)*normDiff*(smoothstep(color,occluderAlbedo,intensity));

Anschließend muss die Radiosity noch durch die Anzahl der Samples geteilt werden.

Bei der Betrachtung des Bildes ist mir aufgefallen, dass nun zwar Flächen indirekte Beleuchtung abbekommen, die Lichtfarbe dabei aber nicht berücksichtigt wird. Normalerweise reflektiert ein weißer Pixel, der mit rotem Licht beschienen aber kein weißes indirektes Licht, sondern leicht rötliches Zum Farbwert sollte also noch der Verdecker-Lichtwert (+occluderLight hinter der Klammer) dazurechnen werden. Ich habs bei mir mal so gemacht, die Szene wird dadurch aber merklich heller, das sollte man beachten. Eventuell muss man hier den Alphakanal hinzuziehen, und Bild und indirektes Licht hernach mit Alphablending zusammenführen, oder eine Abschwächung durchführen.

resultRadiosity += step(falloff,depthDifference)*normDiff*(smoothstep(color,occluderAlbedo,intensity)) + occluderLight;

Schritt 2: Fertigmachen

Eigentlich sind wir bereits fertig, da wir für jeden Pixel die Radiosity haben und diese einfach zurückgeben und über das Bild legen können. Jemand, der auch XNA 4.0 verwendet und damit auf SM 3.0 beschränkt ist, wird aber merken, dass die Samplerate sehr niedrig gestellt werden muss, damit er nicht über die Zahl der erlaubten Instruktionen kommt. Besonders da man SSAO- und SSIL-Anweisungen in einem Shader hat, hat man nicht viele Instruktionen übrig. Ich hab zu Demozwecken dafür kurz den SSIL-Shader ausgelagert und 13 Samples eingestellt. Das Ergebnis sieht folgendermaßen aus.

SSIL, kein Blur

Da die Körnung im finalen Bild erkennbar ist, wende ich wie beim SSAO einen Blur an. Lässt sich gut kombinieren. Das Ergebnis sieht schon ein wenig besser aus.

SSIL, mit Blur

Eventuell kann man noch etwas geschickter Normalen- und Tiefenwertvergleich einsetzen, sodass man ein besseres Ergebnis erhält. Ich wüsste im Augenblick aber nicht wie.

Der Vergleich zwischen dem Endergebnis mit und ohne SSIL seht ihr hier. Besonders gut sichtbar ist das Ergebnis am Bauch der Echse, wo ohne SSIL so gut wie überhaupt kein Licht hinkommen würde. Oder an der Unterseite des Raumschiffs. Die Lichtwirkung ist im untersten Bild vielleicht etwas heftig, vielleicht sollte man dies händisch abschwächen.

Bild ohne SSIL

Bild mit SSIL (nur Texturfarbe)

Bild mit SSIL (Texturfarbe und Lichtfarbe)

Bild mit SSIL (Texturfarbe und Lichtfarbe, Wirkung 25%)

Jetzt wo die Helligkeit nicht mehr all zu übertrieben ist, bleibt noch ein letzter Versuch, der allerdings wieder einige Instruktionen beansprucht: Anstatt die Intensität nur am Farbwert des Samples auszumachen, addieren wir noch dot(occluderLight,1), um die Intensität vom einfallenden Licht abhängig zu machen. Wo also wenig Licht hinfällt, wird auch wenig zurückgestrahlt, egal ob die Fläche jetzt sehr hell ist (und somit ohne Berücksichtigung ein hohes Rückstrahlvermögen hätte). Das Ergebnis ist das folgende. 

Bild mit SSIL (Texturfarbe und Lichtfarbe, Wirkung 25%), Rückstrahlvermögen berücksichtigt einfallendes Licht

Auch hier: Flächen an den Unterseiten kriegen normalerweise fast garkein Licht ab, mit SSIL das Umgebungslicht. Die Unterkante des Steins ist ebenfalls merklich anders beleuchtet. Ein sichtbarer Unterschied auch bei dem Pinüppel, dessen Unterseide nun nicht pechschwarz ist.

Bild mit SSIL (Texturfarbe und Lichtfarbe), Intensität berücksichtigt kein einfallendes Licht, keine Abschwächung

Bild ohne SSIL

Beurteilung und Einschränkungen

Ich will nicht erneut auf der hohen Instruktionszahl rumreiten, aber gerade wenn man zusätzliches Feintuning, wie die Intensität vom Lichtwert abhängig zu machen, reinbringt, bleibt nicht mehr viel übrig. Andere Einschränkungen dieser Implementierung sind, dass das indirekte Licht mit nur einem Lichtabprall berücksichtigt wird. Es wird also nur einmal reflektiert - und das funktioniert mit wenigen Samples auch nur dann gut, wenn der Radius der Strahlen für die Samples relativ klein bleibt. Außerdem wird nur berücksichtigt, was auf dem Screen zu sehen ist. Ist etwas, das stark abstrahlen würde, knapp aus dem Bild, ist dem Shader das völlig egal. Ist auch nichts dran zu ändern.

Wenn euch noch etwas auf- oder einfällt, wo ich vielleicht Denkfehler habe oder man Performance rausholen könnte, immer raus damit! Ansonsten gefällt mir das ganze schon ganz gut und ich werde weiter versuchen die Lichtfarbe etwas stärker mit in die Wirkung aufzunehmen.

Danke an alle, die irgendwas zum Thema online gestellt haben - wie immer gilt: Sie haben mich bestimmt beeinflusst.

Einfache SSAO-Implementierung (XNA 4.0)

Wozu SSAO?

Normalerweise gibt es in Spielen nur lokale Beleuchtung, das heißt Licht wird für einzelne Punkte von Flächen ausgewertet. Dabei ist nur wichtig, ob ein Strahl einen Punkt trifft, so etwas wie eine Verdeckung, also eine Wechselwirkung von Flächen und Punkten untereinander spielt für dieses Modell keine Rolle. Wenn man aber mal in die echte Welt schaut, ist die Beleuchtung um einen Gegenstand, der auf einer ebenen Fläche steht und Beleuchtet wird nicht sauber, sondern in der nähe des Gegenstandes stellenweise dunkler. Mal ganz abgesehn davon, dass Flächen selten wirklich glatt sind - die meisten Materialien sind rau und uneben. Screen Space Ambient Occlusion ist deswegen aus vielen Gründen eine richtig feine Sache: Erstens ist es so einfach, dass sogar ich es ansatzweise verstanden habe (hoffentlich) und zweitens wertet es das Gesamtbild eines Videospiels als Pixelshader merklich auf, ohne dass die restliche Architektur des Spiels groß oder überhaupt angefasst werden muss - es funktioniert also mit fast jedem Spiel. Der dritte Grund ist, dass der Shader eventuell sogar noch um indirekte Beleuchtung (SSIL) erweitert werden kann, dazu aber in einem späteren Post mehr.

Wer verstehen möchte, wie man SSAO implementieren kann, der sollte Schritt für Schritt den Post lesen und versuchen den Code zu verstehen. Die Kommentare sollten dies einfach machen. Wie bei allem heutzutage gilt: Angucken und Können ist nicht - es braucht ein bisschen Zeit und Konzentration um zu verstehen und Übung um es zu implementieren. Wer von Computergrafik absolut keine Ahnung hat, der braucht nicht weiterlesen, weil Begriffe fallen werden, mit denen er nichts anfangen kann. Wer schon mal einen Shader geschrieben hat und ansatzweise versteht worum es geht, der kann mit den folgenden Absätzen sicherlich etwas anfangen.

Wie funktionierts?

Ich steh nicht so auf pure Theorie, daher ist meine Umgebung abgesteckt: XNA 4.0 und HLSL, weil ich damit grade n Projekt machen "musste". Wir brauchen sonst nicht viel, um SSAO mit HLSL zu implementieren: Den Depthbuffer, die Colormap und sowas wie ne Noisemap. Außerdem integrieren wir unsere Normalen. Das wars schon. Meine Implementierung basiert übrigens auf mehreren Internetblogs, zum Beispiel diesem hier. Grob gesprochen nehmen wir jeden Pixel des Bildes und betrachten den Tiefenwert der umliegenden, nahen Pixel, die wir in einer Halbkugel um die Normale des Pixels zufällig auswählen. Sind diese weiter vorne als der Pixel, ist davon auszugehen, dass der ursprüngliche Pixel räumlich hinter ihnen liegt, also von ihnen verdeckt wird. Je nach dem, wie groß die Strecke zwischen dem aktuellen Pixel und einem umliegenden ist, kann man hier eine Verdeckung sichtbar machen.

Schritt 1: Samples holen, Zufall schaffen

Es kommt natürlich ein bisschen darauf an, wie ihr euer Spiel baut - ich habe eine Deferred Rendering Engine aus irgendeinem XNA-Tuto nachgebaut und liefere meine Maps als Texturen an meine Posteffekt-Shader.

float depth = tex2D(depthSampler,input.TexCoord).r;

float3 color = tex2D(colorSampler,input.TexCoord).rgb;

float3 noise = tex2D(noiseSampler,input.TexCoord).rgb;

float3 norm = tex2D(normalSampler,input.TexCoord).rgb;

Jetzt brauchen wir noch so etwas wie einen Zufall. Ich habe hier ein paar Random-Vektoren in ner Kugel mit 1er Durchmesser.

float3 pSphere[16] = {float3(0.53812504, 0.18565957, -0.43192),float3(0.13790712, 0.24864247, 0.44301823),float3(0.33715037, 0.56794053, -0.005789503),float3(-0.6999805, -0.04511441, -0.0019965635),float3(0.06896307, -0.15983082, -0.85477847),float3(0.056099437, 0.006954967, -0.1843352),float3(-0.014653638, 0.14027752, 0.0762037),float3(0.010019933, -0.1924225, -0.034443386),float3(-0.35775623, -0.5301969, -0.43581226),float3(-0.3169221, 0.106360726, 0.015860917),float3(0.010350345, -0.58698344, 0.0046293875),float3(-0.08972908, -0.49408212, 0.3287904),float3(0.7119986, -0.0154690035, -0.09183723),float3(-0.053382345, 0.059675813, -0.5411899),float3(0.035267662, -0.063188605, 0.54602677),float3(-0.47761092, 0.2847911, -0.0271716)};

Als nächstes generieren wir Normalen, die wir später zur Reflektion verwenden. Hier kommt die Noisemap zum Einsatz. Für jeden Bildpunkt kriegen wir also eine zufällige Normale. Das Noise Sample kann noch ein zusätzliches Offset kriegen.

float3 fres = normalize(noise*2) - float3(1.0, 1.0, 1.0);

Meine Quelle speichert sich Fragmentkoordinaten etwas komfortabler ab, daher mach ich das mal nach. Was wir dann in einer Variable haben sind die UV-Screenkoordinaten, also Pixel nach x und y und den Tiefenwert im Bildschirmplatz...den Depthbuffer.

float3 ep = float3(input.TexCoord,currentPixelDepth);

Außerdem macht er den Radius wo wir samplen von der tatsächlichen Tiefe abhängig. Je größer die Tiefe, desto größer der Radius (glaube ich :) ). Wahrscheinlich, damit man einen stärkeren Effekt im Hintergrund hat, der ist ansonsten schlecht wahrnehmbar. Für rad wählt meine Quelle den Wert 0.006. Kann man machen, rumspielen hilft.

float radD = rad/currentPixelDepth;

Schritt 2: Die Schleife

Nun müssen wir für jeden Pixel des Bildes einige Schritte durchführen. Bevor wir die Schleife schreiben, definieren wir mal unsere benötigten Variablen und die Variable, wo wir das Ergebnis speichern.

float bl = 0.0f;

float occluderDepth, depthDifference, normDiff;

Wie oft die Schleife durchlaufen wird, muss man selbst entscheiden. Je mehr desto besser ist die Qualität des Effekts. Bei mehr als 16 Samples braucht man logisccherweise mehr Zufallsvektoren in pSphere.

for(int i=0; i<9;++i)

   {

      // Einen Vektor in der Kugel holen und reflectieren (mit dem Noise-Vektor)

      float3 ray = radD*reflect(pSphere[i],fres);

      // Wenn der Strahl aus der Halbkugel rausgeht, wird Richtung geändert

      float3 se = ep + sign(dot(ray,norm) )*ray;

      // Tiefenwert des verdeckenden Bildpunktes holen

      float3 occluderFragment = tex2D(depthSampler,se.xy);

      // Normale des verdeckenden Bildpunktes holen

      float3 occNorm = tex2D(normalSampler,se.xy).rgb;

      // Wenn die Diff der beiden Punkte negativ ist, ist der Verdecker hinter dem aktuellen Bildpunkt

      depthDifference = currentPixelDepth-occluderFragment.r;

      // Berechne die Differenz der beiden Normalen als ein Gewicht

      normDiff = (1.0-dot(occNorm,norm));

      // the falloff equation, starts at falloff and is kind of 1/x^2 falling

      bl += step(falloff,depthDifference)*normDiff*(1.0-smoothstep(falloff,strength,depthDifference));

   }

Der letzte Schritt dient, um den Verdeckungswert weich abfallen zu lassen. Wer genauer wissen möchte, was passiert, kann hier schauen. Für falloff wählt meine Quelle den Wert 0.000002. Hab ich genauso.

Schritt 3: Fertigmachen

Nach der Schleife wird der Verdeckungswert nochmal mit einem Steuerwert für die Stärke multipliziert und durch die Samples geteilt. strength ist bei "uns" 0.07, totStrength 1.38. Prinzipiell würde es reichen, wenn wir einen float zurückgeben, aber ich hab den Effekt später noch für was anderes nutzen wollen, daher is mein Rückgabewert etwas verschwenderisch.

float ao = 1.0-totStrength*bl*invSamples;

return float4(ao,ao,ao,ao);

Das war ja einfach

Das wars auch schon. Wieviele Zeilen Code waren das? 15? Diese Technik ist nicht völlig frei von Fehlern. Wenn man ein bisschen zu viel am Radius spielt, kriegt man seltsame Ergebnisse und die Nachteile des Viewspace (nur was auf dem Bildschirm sichtbar ist, wird beachtet...) lassen sich auch nicht verneinen. Ich hab bei zu großen Unterschieden bei den Tiefenwerten (?) ein hässliches Kantenbluten gehabt, das ich auch mit einem zusätzlichen Blur-Pass nicht wegbekommen habe (in den Bildern unten erkennbar). In XNA 4.0 hat man außerdem nur Shadermodel 3.0, das bedeutet limitierte Instruktionszahlen. Gerade wenn man den Effekt noch um indirekte Beleuchtung erweitern will, macht sich das sehr schnell bemerkbar. Ein zusätzlicher Blurpass sollte also fast schon eingeplant werden, wenn man SSAO irgendwie ins XNA bringen will. Ich habe hier jetzt keinen, das Ergebnis kann sich trotzdem sehen lassen:

mit SSAO

ohne SSAO

SSAO

Von meiner Seite aus ein allgemeines Danke an alle Leute bei Nvidia, in den Blogs, meine Quelle und auch sonst jeden, der Informationen zum Thema ins Internet gestellt hat, denn er hat diesen Post mit Sicherheit irgendwie beeinflusst.

Viel Spaß beim Nachmachen, für einen der kommenden Posts habe ich noch eine kleine, aber effektive Erweiterung des Shaders. Wenn irgendwas hier falsch oder zu ungenau ist, kommentiert einfach, die Chance ist hoch, dass ich mich Kommentaren annehme :)