Roderick's Debug Diary

Embedding v8 Javascript in an MSVC project

Roderick Kennedy — Fri, 20 Dec 2024 12:15:33 GMT

rem building v8
rem following https://medium.com/angular-in-depth/how-to-build-v8-on-windows-and-not-go-mad-6347c69aacd4

echo off
set Path=%cd%\depot_tools;%Path%
set DEPOT_TOOLS_WIN_TOOLCHAIN=0
set GYP_MSVS_VERSION=2022

rem curl -O https://storage.googleapis.com/chrome-infra/depot_tools.zip
rem mkdir depot_tools
rem powershell -command "Expand-Archive -Force '%~dp0depot_tools.zip' '%~dp0depot_tools'"


rem fetch v8
set Path=%cd%\depot_tools\.cipd_bin\3.8\bin\Lib\venv\scripts\nt;%Path%

echo on

rem to activate the venv, we need a pyvenv.cfg in the scripts\nt directory e.g.
rem home = C:/Code/Teleport/Javascript/depot_tools/.cipd_bin/3.8/bin/Lib/venv/scripts/nt
rem include-system-site-packages = false
rem version = 3.8

call %cd%\depot_tools\.cipd_bin\3.8\bin\Lib\venv\scripts\nt\activate.bat

rem activate.bat calls echo off, but we want to see these commands:
echo on

where python

cd v8

rem gclient sync
rem python tools/dev/v8gen.py list
rem python tools/dev/v8gen.py x64.release

rem To embed V8 into our application we need to build it as a static library. To do that, we need to modify default build configuration and add these two flags to args.gn file:

rem is_component_build = false
rem v8_static_library = true

rem  now to out.gn\x64.release/args.gn add:
rem is_debug = false
rem target_cpu = "x64"
rem is_component_build = false
rem v8_static_library = true
rem # disable thin lib shenanigans
rem  thin_lto_enable_cache = false

rem GOT to have Windows SDK 10.0.22621.2428, install by hand if necessary
rem see also https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows_build_instructions.md
rem for building on windows with msvc rather than clang

call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" x86_amd64
set WINDOWSSDKDIR=C:\Program Files (x86)\Windows Kits\10
rem gn args out.gn/x64.release --list
rem build/config/win/BUILD.GN: cflags += [ "-fms-extensions","-fms-compatibility"]
gn gen --ide=vs --ninja-executable=%cd%\third_party\ninja\ninja.exe out.gn/x64.vs.release
rem cd out.gn/x64.release
rem ninja -C out.gn/x64.release
rem -t msvc 

rem python tools/run-tests.py --gn

Don't step into STL function in Visual Studio 2022

Roderick Kennedy — Thu, 08 Feb 2024 10:48:54 GMT

Find your Visual Studio install directory. It is probably something like "C:\Program Files\Microsoft Visual Studio\2022\Community"

Navigate to the subfolder "Common7\Packages\Debugger\Visualizers".

As administrator, edit the file default.natstepfilter.

Insert this in betwen the xml tags:

<Function><Name>std\:\:.* Name><Action>NoStepIntoAction>Function>

Lambdas, or functions as function parameters, in Sfx

Roderick Kennedy — Tue, 26 Sep 2023 09:41:58 GMT

Following the work in my previous post, I happened across this Mastodon post about how good it would be to use lambdas, or function objects, as inputs in shaders.

[

](https://mastodon.gamedev.place/@meuns/111119173437770101)

It occurred to me that in our Sfx framework this might not be difficult to implement.

So I’ve implemented it. Because Sfx is essentially just a big text-processing system with semantic knowledge of shader syntax, it wasn’t that difficult. You declare a function input as follows.

shader vec4 PS_Variants(posTexVertexOutput IN,variant function RenderFunction) : SV_TARGET { return RenderFunction(IN); }

For now, we actually discard the function object’s template parameters, but later on this will be used for error-checking. Then, in the pass declaration we would do, e.g.

SetPixelShaderVariants(ps_variants({PSNeon, PSWhite}));

And this would generate two variants, calling the function PSNeon(posTexVertexOutput IN) or PSWhite(posTexVertexOutput IN).

In the variant-generation stage, where we assemble the source for compilation to your specified language, variant parameters that are functions are replaced in the function content with the specific function for that variant, e.g. for GLSL this generates:

layout(location =0) in Block { posTexVertexOutput BlockData0; } ioblock; layout(location = 0) out vec4 returnObject_vec4;

void PS_Variants() { posTexVertexOutput BlockData0=ioblock.BlockData0; #line 1366 "C:/Teleport/client/Shaders/test.sfx" {returnObject_vec4=PSWhite(BlockData0);return;} }

At the moment, this only works for shader functions, we can’t pass lambdas down to the functions it calls. But I think it can easily be extended - the trick will be that when calling a function that itself has variants, we’ll need an intermediate syntax which eliminates the lambdas from the parameter list, but indicates to the shader constructor that a specialized instance of the called-function must be created.

The Sfx shader language and shader variants

Roderick Kennedy — Tue, 19 Sep 2023 17:57:03 GMT

The Sfx shader language and shader variants

19 Sep

Written By Roderick Kennedy

For some reason there seems to be a lot of interest in custom and alternative game engine development now.

Whatever the cause of this mysterious uptick, I thought I'd post a bit about what we do at Simul.

Handling shaders can be a vexed issue in native development. If you're writing a single shader, or a matched pair of vertex and pixel shaders, things start very simply, whether you're using GLSL or HLSL. You'll have a vertex shader file, like this:

cbuffer CameraConstants { float4x4 viewProj; }; struct VS_IN { float4 a_Positions : POSITION; }; struct VS_OUT { float4 o_Position : SV_Position; }; VS_OUT main(VS_IN IN) { VS_OUT OUT; OUT.o_Position = mul(viewProj,IN.a_Positions); return OUT; }

And a pixel shader file:

struct PS_IN { float4 i_Position : SV_Position; }; struct PS_OUT { float4 o_Colour : SV_Target0; };

PS_OUT main(PS_IN IN) { PS_OUT OUT; OUT.o_Colour = float4(1.0,0,0,0); return OUT; }

This above is HLSL. GLSL looks similar inside the functions, but the surrounding declarations are different, e.g.

#version 450 layout(std140, binding = 0) uniform CameraConstants { mat4 modelViewProj; }; layout(location = 0) in vec4 a_Positions; void main() { gl_Position = modelViewProj * a_Positions; }

#version 450 layout(location = 0) in flat uvec2 i_TexCoord; layout(location = 0) out vec4 o_Colour; void main() { o_Colour = vec4(1.0,0,0,0); }

The trick comes as your number of shaders starts to increase, and you start needing multiple combinations - different materials might need shaders with different inputs. 3D models may have different vertex setups, so you need a vertex shader that exactly matches the vertex layout you're sending. And to combine a vertex and pixel shader, you must ensure that the vertex shader output matches the pixel shader's input.

It can get unwieldy.

At Simul, we developed Sfx, a sort of wrapper or meta- language to help manage shaders. The simple vertex-pixel combo above, in Sfx, looks like this:

cbuffer CameraConstants { float4x4 viewProj; }; struct VS_IN { float4 a_Positions : POSITION; }; struct VS_TO_PS { float4 o_Position : SV_Position; }; struct PS_OUT { float4 o_Colour : SV_Target0; };

VS_TO_PS VS_Main(VS_IN IN) { VS_TO_PS OUT; OUT.o_Position = mul(viewProj,IN.a_Positions); return OUT; }

PS_OUT PS_Main(VS_TO_PS IN) { PS_OUT OUT; OUT.o_Colour = float4(1.0,0,0,0); return OUT; }

VertexShader vs_main = CompileShader(vs_5_0,VS_Main()); PixelShader ps_main = CompileShader(ps_5_0, PS_Main());

technique simpledraw { pass p0 { SetRasterizerState( wireframeRasterizer ); SetDepthStencilState( DisableDepth, 0 ); SetBlendState(AddBlend, vec4(0.0,0.0,0.0,0.0), 0xFFFFFFFF ); SetVertexShader(vs_main); SetPixelShader(ps_main); } }

Nice and simple: one file instead of two. And we've added some useful renderstate setup. But how is this used?

Sfx is a transpiler. It doesn't compile binary shaders, rather it translates its shaders to a target language. We might invoke it with

Sfx.exe simple.sfx -I"shader_includes" -O"C:/Teleport/build_pc_client/shaderbin/$PLATFORM_NAME" -P"Sfx/DirectX12.json" -P"Sfx/Vulkan.json"

This means, transpile the effect file "simple.sfx" into shaders, using the profiles "DirectX12.json" and "Vulkan.json". These setup files tell Sfx how to generate shader code. For DirectX 12, it will generate hlsl shaders for Microsoft's dxc compiler. For Vulkan, it will create glsl shaders for glslangvalidator.exe. In both cases, it will invoke the relevant compiler, and bundle the output into an effect binary, simple.sfxb, and an effect setup file simple.sfxo. The former contains the shaders, the latter contains instructions on how they can be used in passes.

Then at runtime, a small library (specific to your chosen graphics API) loads up the binaries and applies the shaders, and renderstate when needed.

I called "simple.sfx" an effect file. This unfashionable concept was all the rage when shaders were first in widespread use, but fell out of favour as more developers started to use commercial game engines, which handled that sort of thing internally. But I think it still has a lot of value. Although each shader language is essentially a special-purpose version of C, the effect file has a meta-language that surrounds it which helps us build and manage collections of shaders.

Originally Sfx was modelled on the effect file format of Direct3D, which was mothballed some years ago. But it's grown into its own thing now. For example, we can declare shader variants as follows:

shader vec4 PS_Var(vertexOutput IN,variant bool textured) : SV_TARGET { vec4 colour; if(textured) { colour=texture.Sample(wrapSamplerState, IN.texCoords.xy); } else { colour=globalColour; } return colour; }

This shader has two variants, one with a texture lookup and one without. In the pass declaration, we invoke this with:

PixelShader ps_var = CompileShader(ps_5_0, PS_Var()); technique test { pass p0 { SetVertexShader(vs_main); SetPixelShaderVariants(ps_var({false,true})); } }

Sfx will now generate two versions of the pixel shader, one with texture=true, one with false. Although if(textured) looks above like runtime code, each version will be compiled with the value of textured constant: it will be optimized down to static code. At runtime though, we can choose which version we want. We could also specify two or more different shaders, e.g.

SetPixelShaderVariants(ps_var1,ps_var2,ps_debug);

It would have been easy enough to just write two versions of the pass, one with textures and one without. The power of the variant system comes as shaders multiply. Combining, say, three vertex shaders with two pixel shaders:

SetVertexShader(vs_main,vs_alternative); SetPixelShaderVariants(ps_var({false,true}),ps_test);

Now, assuming they are mutually compatible, we would have six possible passes. Consider for example, creating specialized debug views of your 3D objects: for each view, you can just add a pixel shader variant to the pass.

So for larger projects, a system like Sfx can be invaluable in quickly writing efficient GPU code. If you do find yourself programming natively, without the comforts of a game engine, it may be just the ticket.

As of today, Sfx supports not only Vulkan and OpenGL flavours of GLSL and D3D11 and 12 flavours of HLSL, but also certain proprietary platforms that I'm not permitted to name. We codenamed them Spectrum and Commodore, for convenience.

All the source for Sfx is at https://github.com/simul/Platform/tree/main/Applications/Sfx.

Roderick Kennedy

The Sfx shader language and shader variants

19 Sep

Written By Roderick Kennedy

For some reason there seems to be a lot of interest in custom and alternative game engine development now.

Whatever the cause of this mysterious uptick, I thought I'd post a bit about what we do at Simul.

cbuffer CameraConstants { float4x4 viewProj; }; struct VS_IN { float4 a_Positions : POSITION; }; struct VS_OUT { float4 o_Position : SV_Position; }; VS_OUT main(VS_IN IN) { VS_OUT OUT; OUT.o_Position = mul(viewProj,IN.a_Positions); return OUT; }

And a pixel shader file:

struct PS_IN { float4 i_Position : SV_Position; }; struct PS_OUT { float4 o_Colour : SV_Target0; };

PS_OUT main(PS_IN IN) { PS_OUT OUT; OUT.o_Colour = float4(1.0,0,0,0); return OUT; }

This above is HLSL. GLSL looks similar inside the functions, but the surrounding declarations are different, e.g.

#version 450 layout(std140, binding = 0) uniform CameraConstants { mat4 modelViewProj; }; layout(location = 0) in vec4 a_Positions; void main() { gl_Position = modelViewProj * a_Positions; }

#version 450 layout(location = 0) in flat uvec2 i_TexCoord; layout(location = 0) out vec4 o_Colour; void main() { o_Colour = vec4(1.0,0,0,0); }

It can get unwieldy.

At Simul, we developed Sfx, a sort of wrapper or meta- language to help manage shaders. The simple vertex-pixel combo above, in Sfx, looks like this:

cbuffer CameraConstants { float4x4 viewProj; }; struct VS_IN { float4 a_Positions : POSITION; }; struct VS_TO_PS { float4 o_Position : SV_Position; }; struct PS_OUT { float4 o_Colour : SV_Target0; };

VS_TO_PS VS_Main(VS_IN IN) { VS_TO_PS OUT; OUT.o_Position = mul(viewProj,IN.a_Positions); return OUT; }

PS_OUT PS_Main(VS_TO_PS IN) { PS_OUT OUT; OUT.o_Colour = float4(1.0,0,0,0); return OUT; }

VertexShader vs_main = CompileShader(vs_5_0,VS_Main()); PixelShader ps_main = CompileShader(ps_5_0, PS_Main());

technique simpledraw { pass p0 { SetRasterizerState( wireframeRasterizer ); SetDepthStencilState( DisableDepth, 0 ); SetBlendState(AddBlend, vec4(0.0,0.0,0.0,0.0), 0xFFFFFFFF ); SetVertexShader(vs_main); SetPixelShader(ps_main); } }

Nice and simple: one file instead of two. And we've added some useful renderstate setup. But how is this used?

Sfx is a transpiler. It doesn't compile binary shaders, rather it translates its shaders to a target language. We might invoke it with

Sfx.exe simple.sfx -I"shader_includes" -O"C:/Teleport/build_pc_client/shaderbin/$PLATFORM_NAME" -P"Sfx/DirectX12.json" -P"Sfx/Vulkan.json"

Then at runtime, a small library (specific to your chosen graphics API) loads up the binaries and applies the shaders, and renderstate when needed.

shader vec4 PS_Var(vertexOutput IN,variant bool textured) : SV_TARGET { vec4 colour; if(textured) { colour=texture.Sample(wrapSamplerState, IN.texCoords.xy); } else { colour=globalColour; } return colour; }

This shader has two variants, one with a texture lookup and one without. In the pass declaration, we invoke this with:

PixelShader ps_var = CompileShader(ps_5_0, PS_Var()); technique test { pass p0 { SetVertexShader(vs_main); SetPixelShaderVariants(ps_var({false,true})); } }

SetPixelShaderVariants(ps_var1,ps_var2,ps_debug);

SetVertexShader(vs_main,vs_alternative); SetPixelShaderVariants(ps_var({false,true}),ps_test);

All the source for Sfx is at https://github.com/simul/Platform/tree/main/Applications/Sfx.

Roderick Kennedy

Why the Metaverse Failed

Roderick Kennedy — Mon, 11 Sep 2023 20:35:35 GMT

Why the Metaverse Failed

11 Sep

Written By Roderick Kennedy

The research firm Gartner has a way of visualizing the development and adoption of new technologies. After an initial work and investment gold rush, there follows a "trough of disillusionment", where the bold claims of the early days haven't panned out, and investors, journalists and even the technologists themselves feel that things aren't going anywhere - at least going nowhere fast enough to justify the hype.

By Jeremy Kemp at English Wikipedia, CC BY-SA 3.0,

https://commons.wikimedia.org/w/index.php?curid=10547051

In the Gartner Hype Cycle, this Slough of Despond is followed by the "Slope of Enlightenment", where the technology continues to develop and starts to achieve its best practices. Finally, there's a Plateau of Productivity, where it reaches the Nirvana of actual use in the real world and value to the economy.

Of course there are a number of problems with this model, if we can even call it that. The Y-axis is "expectations", which is a pretty nebulous thing to measure. In the full flush of hype, you could be halfway up the peak, or you could be at the top and ready to drop. You only know some time later where you were.

And the whole graph presupposes that this technology will end up being useful. If every Trough of Disillusionment was guaranteed to be followed by an upward slope, why would we even get disillusioned?

So you could equally be in the trough before enlightenment kicks in, or you could be on the tail end of an idea that just isn't going to work, and will never find its Plateau. Think, Apple Newton, or... jetpacks.

Now, we certainly seem to be in one or other of these two situations with the Spatial Internet, or as some prefer to call it, the Metaverse.

The idea of a Spatial Internet goes back a long way. In William Gibson's Neuromancer, he imagined hackers "jacking in" with a brain-computer interface directly to the network itself, experiencing a "shared hallucination", a direct view onto the data and communication structures of the "matrix". Conversely, Snow Crash by Neal Stephenson envisaged a simulated environment: what you see is not the raw data made visual, but a virtual world recreating aspects of the real world: you can own property, do business, socialize. This latter model took hold in the minds of many key thinkers in Silicon Valley and beyond.

In 1994, only 3/4 years after the Web burst onto the scene and made the Internet navigable to ordinary people, David Ragget, Mark Pesce and Brian Behlendorf published a proposal for "VRML: Extending WWW to support Platform Independent Virtual Reality", the first substantial technological effort to make the networked Metaverse a reality.

There was a whole hype cycle around VRML. But by the time of the dotcom crash, the tech writer Clay Shirky noted:

“The Quake 3-D engine is for something, while VRML is a technology in search of a use. The Quake engine’s original use is immaterial; once you have a world-modeling protocol… you’ve got the whole ball of wax — spaces, movement, objects, avatars, even a simple physics… once a tool is good for one thing, it’s easy to see how to make it good for another thing.”

— Quote Source

And so VRML reached its own Trough of Disillusionment, from which it has yet to recover. But contrary to Gartner, this was not the first such trough for VR. In 1984, VPL Research, founded by Jaron Lanier had launched, with products such as the DataGlove and the EyePhone (an early headset). There was much fanfare, and films like The Lawnmower Man built the hype around VR, but the hardware was expensive, and didn't find much use outside of military training.

So what we really see is multiple peaks and troughs. But will we ever see the plateau?

When I think of the Spatial Internet, I tend to imagine something like what Pesce and his cohorts proposed in VRML: a shared, common, navigable network. But what we have instead at present is not that. Many millions of venture dollars and person-years of effort have been put into downloadable, installable applications running on XR hardware. It's a mobile phone model: there's an app store, you buy an app, you run it. If you want to do something different, you run a different app.

Now, there are strong efforts at standardization: file formats like GLTF and USD offer a way for applications to share data, so it's possible for users to port some aspects of their personal data between apps.

What we do not have right now though, is a navigable network. When Tim Berners Lee created the World Wide Web, he presented two interlocking technologies. HTTP transfers whole files from a server to a client. HTML allows textual data to be formatted on arbitrary screens, and provides a set of common guiderails.

Guiderails are key. Everyone knows what a link looks like on a web page. Everyone knows that when you click a link, it takes you to a new server, a new URL, and that this is a one-way system - the receiving site doesn't need to give permission to the linking site, it treats each request afresh. Everyone knows that images can be embedded, and it's up to the browser to choose how to display them, according to a clear set of rules that HTML defines.

So you know when navigating the Web that each page you go to follows these same rules, and will be amenable to how you are used to viewing and interacting with sites.

There are no guiderails for the Spatial Internet. Because there's no protocol: we're still using HTTP for WebXR, when we really ought to have a realtime protocol that's built for spatial use-cases.

What should a link look like? Should it be one-way? I suggest: not always. Spatial applications are much more data-heavy than websites. You need to know whether, when you follow a link or portal to a different site, that site is: a) available, and b) has some way back to where you were before.

In 1997, researchers at the US Naval Postgraduate School proposed VRTP: a protocol for VRML that met the needs of spatial internet applications (https://faculty.nps.edu/brutzman/vrtp/). But because VRML was hitting its own trough soon afterwards, VRTP was not developed to completion.

If there is to be a Spatial Internet, we have to address this fundamental challenge. It's what we're doing with Teleport VR, and working with the standards bodies I hope we'll get Teleport or something like it adopted. From here, it could go either way: the last line of that Gartner curve has not yet been drawn.

https://roderickkennedy.com/virtual-reality/http-for-vr

Roderick Kennedy

Why the Metaverse Failed

11 Sep

Written By Roderick Kennedy

By Jeremy Kemp at English Wikipedia, CC BY-SA 3.0,

https://commons.wikimedia.org/w/index.php?curid=10547051

Now, we certainly seem to be in one or other of these two situations with the Spatial Internet, or as some prefer to call it, the Metaverse.

There was a whole hype cycle around VRML. But by the time of the dotcom crash, the tech writer Clay Shirky noted:

“The Quake 3-D engine is for something, while VRML is a technology in search of a use. The Quake engine’s original use is immaterial; once you have a world-modeling protocol… you’ve got the whole ball of wax — spaces, movement, objects, avatars, even a simple physics… once a tool is good for one thing, it’s easy to see how to make it good for another thing.”

— Quote Source

So what we really see is multiple peaks and troughs. But will we ever see the plateau?

So you know when navigating the Web that each page you go to follows these same rules, and will be amenable to how you are used to viewing and interacting with sites.

There are no guiderails for the Spatial Internet. Because there's no protocol: we're still using HTTP for WebXR, when we really ought to have a realtime protocol that's built for spatial use-cases.

https://roderickkennedy.com/virtual-reality/http-for-vr

Roderick Kennedy

Updating the PHP version on Wordpress

Roderick Kennedy — Fri, 21 Jul 2023 08:11:22 GMT

Wordpress sometimes needs you to update the php version. Outside of a managed hosting, if you’re working with raw Linux, you’re pretty much stuck. Ignoring the official, unhelpful page at https://wordpress.org/support/update-php, what actually has to be done is this:

Using the example of updating from PHP 7.2 to 7.4, and assuming you have installed 7.4 from command-line already, add the line:

AddHandler application/x-httpd-php74 .php

to the .htaccess file in the web root. Now call:

sudo a2dismod php7.2

sudo a2enmod php7.4

sudo systemctl restart apache2

And done.

Introducing WebRTC to Teleport VR

Roderick Kennedy — Fri, 17 Mar 2023 23:06:33 GMT

Introducing WebRTC to Teleport VR

17 Mar

Written By Roderick Kennedy

Getting Teleport running on Linux

It's long been a goal to get a Teleport server running on Linux. The reason to do this is that I want to offer effectively "VR hosting in-a-box", and the way to do that is containerization: we make it possible to create a Teleport server instance in a Docker container, then it can be uploaded to a cloud service like AWS with a minimum of fuss.

If we try to do this with Windows, we come up against licensing problems: if you're running a Windows server, you need a Windows licence. Best to use Linux.

But apart from the Android/Quest client, all Teleport work so far has been on Windows.

So for the past few months I've tried to get this resolved: removing obvious Win32 API dependencies, getting the necessary libraries built. An early issue was we were passing strings to and from Unity as _bstr_t, a Microsoft class that isn't supported in clang/Unix. So I switched over to plain old char* for Unity to communicate with the Teleport dll.

Debugging was also a challenge. I've not yet found a way to debug C# code in Unity on Linux, but C++ works, after some deep dives into the gdb settings (I haven’t yet got lldb working, but gdb works fine for clang code).

So finally we can now connect from a Teleport client to the Linux server in Unity. But the data streams are not working. I could try to fix our UDP-based data stream code for Linux, which uses the 3rd party libraries SRT and EFP.

Streaming Transport

We need streaming transport because in Teleport we stream video, audio and geometry. In the hybrid rendering mode (shown above), far objects are streamed as video in the background. Near objects are streamed as geometry. In this video I've added a grey highlight to the video stream, otherwise it might be difficult to see the join.

As we move through the scene, any geometry that we approach is streamed into the local device. In this way, Teleport enables:

Higher visual quality than an installed application or WebXR
Larger scenes than an installed application or WebXR
Multi-platform support without multi-platform code
Instant deployment to any device.

So under the Teleport protocol we need two network transport layers: one for messaging, one for data streaming. Plus HTTPS for static file downloads.

Introducing WebRTC

To build Teleport, needed a data streaming system. We used UDP packets, a fast way of sending data - much faster than the TCP/IP that browsers use to download web content. UDP is "unreliable": the packets aren't guaranteed to get there the way TCP/IP packets are. So we needed a couple of extra layers to ensure reliable streaming: we used SRT for ordering and EFP to manage "frames" or data chunks. Both good systems. And when we started building Teleport there wasn't a good alternative.

But WebRTC is mature now, and does exactly what UDP+SRT/EFP does. So once it's integrated, that standardizes the Teleport stack around something that's in widespread use.

So if I can, I'm going to switch the transport layer over to WebRTC, which is well-supported, supports arbitrary data streams, and is at least nominally available in C++.

WebRTC was originally intended as a peer-to-peer multimedia protocol, a way to enable (for example) video calls on web pages. But it’s pretty flexible: it supports two-way video tracks, audio tracks, and crucially for us: data channels - arbitrary text or binary data, flowing in either direction, either reliably or not. WebRTC offers us a number of advantages.

Web-compatible: when we implement the web version of the Teleport client, we won’t be able to use UDP because browsers don’t support raw packet streaming. But they do support WebRTC.
Firewall-friendly: the implementations of WebRTC include support for STUN/TURN, which helps navigate restrictive firewalls, simplifying the process of finding the right port and underlying transport.
Well-supported: compared to the UDP-based systems we’ve been using, WebRTC is used a lot more, so it’s easier to find expertise and solutions.

So here we have it: streaming the geometry via WebRTC. I've added a green shader at the front to highlight each object as it streams in.

There's no background because I'd only implemented geometry streaming so far: video/audio streaming is handled differently in WebRTC. But it worked well: very reliable, and more robust to network changes+firewalls than the UDP approach.

And it should work fine in Linux, though that remains to be tested.

As it turned out, it was pretty simple to activate WebRTC video and audio support by just using data channels - rather than WebRTC's native video/audio tracks. This is most likely less efficient, and will need to be reviewed.

Here as well I show the attractive, minimal 2D user-interface menus I've implemented for desktop use: the 3D menus will now only appear if you're in VR. So the app in desktop now looks... a lot like a browser! Which is intentional.

The back half of the video shows more clearly the transition between the video-streamed background and the geometry-streamed foreground - again with a grey cast to highlight the join - this will enable VR apps of truly vast scope to be accessed instantly via Teleport. While still permitting the option of geometry-only streams, which will be comparable in price to running a web server - as against the $dollar+ per user per hour that Unreal or Unity pixel-streaming costs!

Because there's no chance of a true Metaverse developing at those prices. It needs to be cheap enough that anyone can build a site, host content, create services - just as they can on the web. And that's what Teleport delivers.

Roderick Kennedy

Introducing WebRTC to Teleport VR

17 Mar

Written By Roderick Kennedy

Getting Teleport running on Linux

If we try to do this with Windows, we come up against licensing problems: if you're running a Windows server, you need a Windows licence. Best to use Linux.

But apart from the Android/Quest client, all Teleport work so far has been on Windows.

Streaming Transport

As we move through the scene, any geometry that we approach is streamed into the local device. In this way, Teleport enables:

Higher visual quality than an installed application or WebXR
Larger scenes than an installed application or WebXR
Multi-platform support without multi-platform code
Instant deployment to any device.

So under the Teleport protocol we need two network transport layers: one for messaging, one for data streaming. Plus HTTPS for static file downloads.

Introducing WebRTC

But WebRTC is mature now, and does exactly what UDP+SRT/EFP does. So once it's integrated, that standardizes the Teleport stack around something that's in widespread use.

So if I can, I'm going to switch the transport layer over to WebRTC, which is well-supported, supports arbitrary data streams, and is at least nominally available in C++.

Web-compatible: when we implement the web version of the Teleport client, we won’t be able to use UDP because browsers don’t support raw packet streaming. But they do support WebRTC.
Firewall-friendly: the implementations of WebRTC include support for STUN/TURN, which helps navigate restrictive firewalls, simplifying the process of finding the right port and underlying transport.
Well-supported: compared to the UDP-based systems we’ve been using, WebRTC is used a lot more, so it’s easier to find expertise and solutions.

So here we have it: streaming the geometry via WebRTC. I've added a green shader at the front to highlight each object as it streams in.

And it should work fine in Linux, though that remains to be tested.

Roderick Kennedy

Teleport VR: A Protocol for the Metaverse - presentation slides

Roderick Kennedy — Fri, 18 Mar 2022 00:11:46 GMT

Teleport VR: A Protocol for the Metaverse - presentation slides

18 Mar

Written By Roderick Kennedy

These are my slides for my session talk at VRARA’s Metaverse 2.0, which was held over 9th-10th March 2022 online. In the talk, I go over some early history of the Metaverse and VR concepts, then go on to make the case why something like Teleport is needed for an open Metaverse of the type we optimistically envision. I compare our approach to the app model status quo, and to Web-based technologies, detailing how in the first case we would be restricted to walled silos, and in the latter to a static data model. Teleport, while open and browsable like the Web, is a dynamic protocol better suited to a live and dynamic 3D Metaverse.

Download the slides here: VRARA-Metaverse2.0-10-3-2022.pdf.

Embed Block

Add an embed URL or code. Learn more

metaverse teleportvr

Roderick Kennedy

Teleport VR: A Protocol for the Metaverse - presentation slides

18 Mar

Written By Roderick Kennedy

Download the slides here: VRARA-Metaverse2.0-10-3-2022.pdf.

Embed Block

Add an embed URL or code. Learn more

metaverse teleportvr

Roderick Kennedy

OpenXR Interaction Profiles

Roderick Kennedy — Sat, 18 Dec 2021 16:58:04 GMT

I’ve tabulated the standard OpenXR interaction profiles from the Khronos OpenXR spec at https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#semantic-path-interaction-profiles.

A given Teleport server will have a control set it supports. This must be sent to any connecting client, to say “these are the controls I need for interaction to work”. The client must then match these controls to its hardware.

Unfortunately, there seems to be nothing in OpenXR that allows us to query the XR device for what paths it provides - you just have to suggest bindings and that will either succeed or fail.

And we need to handle the case where an essential control is missing, and the case where the suggested binding for two required inputs is the same.

This is the table:

https://docs.google.com/spreadsheets/d/1w4Me9_yG_TNho4Gmelrc9ILYtG1TELoxCXc622xUiVM/edit?usp=sharing

Most hardware should support at least the Khronos Simple Controller, plus its own profile if present. There is also the option of extension paths which include “_ext” in them. But again, no way to query them…

User paths

Only the HTC Vive Pro implements the “head” path, and only for a system button and sound controls. OpenXR does not treat the headset as a controller the way it does for the hands; you have to use xrLocateSpace to get the head pose.

So unless we’re using a gamepad, in most cases we’ll have a user path “/user/hand/left” and “/user/hand/right”, to which we’ll append the input path. For example “/user/hand/left/input/grip/pose” to get the position+orientation (the “pose” in XR jargon) of the left hand. And all of the hand controllers provide two poses per hand: the “grip” which is supposed to be the centre of the palm with the Z-axis pointing down from index finger to pinkie; and the “aim”, where the negative Z-axis points in the “aiming” direction of the controller. These two poses are locked together in all current controllers I know of, they move in sync and the offsets from “grip” to “aim” simply represent the slightly different geometries of the various handsets.

The HTTP of VR

Roderick Kennedy — Sat, 27 Nov 2021 17:41:52 GMT

The HTTP of VR

27 Nov

Written By Roderick Kennedy

There’s been a lot of interest lately in the Metaverse and its potential. To get to that potential, we’re missing a vital piece of the puzzle, without which I think the whole project will never really get off the ground.

The Metaverse needs a network protocol: it needs its own “HTTP”.

In the Beginning

In 1989-90, Tim Berners Lee created the Web. He did this by specifying the two technologies that we still use today: the document format HTML, and the Hypertext Transfer Protocol (HTTP) - the language that web browsers and servers use to communicate. It’s crucial at this distance in time to remember what these technologies were designed for. As a document format, HTML is intended to represent text and associated content: images, tables and so on, but principally, human-readable text. And it’s a linear, one-dimensional format - for the non-linear hopping around that characterizes the Web, we need the other side of the coin, HTTP. And HTTP was designed as a request-response system - the browser or client sends a URI, along with a command (usually “GET” as in, get me this webpage, but also “PUT”, “POST” and so on). It’s a document-retrieval language, intended originally to help scientific institutions freely share data.

It’s discontinuous: each request returns a specific, finite response.
It’s stateless: the server is never required to retain information about a client between requests.
It’s unidirectional: links are one-way, and you don’t need permission from the target server to link to it.

So much of the history of the Web over the last thirty years has been of how tech companies have tried to get around all of these properties to turn a system intended for the free sharing of information into one suited for commerce: how to feed things like ads to clients that didn’t request them; how to record “state” - i.e. track the user and observe their behaviour; how to restrict access so that you can charge for your content. But it’s a testament to the power of the original design that the Web became so important that companies felt they had to work around its restrictions. No-one has seriously proposed returning to the walled-garden online services of 90’s AOL, CompuServe and so on; instead, they’ve tried to recreate miniature walled-gardens within the wider web.

But in VR it’s a different story: every app is a silo, barely connected with the wider Metaverse.

VR on the Web

One of the ways people have tried to extend the Web is to enable content that isn’t a 2D view of a 1D document. Inspired in part by science-fiction visions of a network you could “experience” from the inside, they tried to make it capable of representing 3D spaces.

At the very first World Wide Web Conference, as far back as 1994 - VRML was proposed: a text file format for 3D graphics, sure, but its name, and the paper by David Raggett of HP that proposed it (Extending WWW to support Platform Independent Virtual Reality) revealed its ambition: the Web was to incorporate VR environments, “allowing users to "walk" around and push through doors to follow hyperlinks to other parts of the Web.” The protocol that would deliver these environments? HTTP.

Alongside such document-format technologies, the capability of the Web to support VR rendering has developed. WebXR (formerly WebVR) is a set of Javascript libraries that communicate with a web browser to interact with any VR hardware that the device might have attached. Combined with the 3D rendering capabilities of WebGL (again, Javascript), WebXR gets close to Raggett’s original vision - you can indeed explore parts of the web in VR, and thanks to the capabilities of Javascript, you can even navigate and follow links. Libraries like three.js further extend the power of this approach.

But to get to the level of functionality WebXR provides, you’ve pretty much left the Web behind in terms of its design and intention. A WebXR app is more a Javascript app you’ve downloaded via a web page. By relying so much on the Web to deliver what we need, we’ve lost touch with many of its benefits. In a WebXR app, there are no standards for what a 3D URI looks like, for how the app responds to different hardware capabilities, for how we interact or the relationships between the user and the data model. The DOM, a hugely powerful element of the Web’s design, has little bearing on WebGL, for which only one element, the canvas, is relevant to the 3D environment. Behind the scenes, in Javascript, we are in the realms of code alone: by trying to use the Web for too much, we lose many of its advantages.

The modern inheritor of VRML is perhaps A-Frame, which does away with even the need for a separate file format, and extends HTML itself to support 3D objects. A-Frame is amazing, allowing the hosting of entire 3D scenes in a web page or part of one. Still: the scene is a document. A linear, finite chunk of data returned by a GET request.

There is tremendous power and potential in these technologies. And they will play a vital role in joining the Web with the spatial internet. They cannot deliver the entire vision of the Metaverse. By far the greatest reason to look beyond HTML and HTTP for spatial computing is simply this: these technologies will continue to develop, and will always be driven by their primary purpose: to deliver webpages, websites and static, or marginally dynamic content.

Spatial computing will remain a secondary consideration.

If the Metaverse is to achieve its full potential, it needs its own protocol.

The HTTP of VR

A virtual space is not a document, and its fundamental characteristics are quite different. Much as the Web has become dynamic over the years, an html page is a stationary format subject to step changes when acted on via the DOM - but while a virtual space can be stationary, it could equally be continuously evolving, more akin to an online game than something that can be retrieved and observed in steady-state. While there is no limit on the size of an html document, it is the nature of text that only a finite contiguous segment is viewed at any time in a browser. So it can be viewed without difficulty on minimal hardware. In 3D, it is possible for vastly distant elements of a scene to be visible from any point. This introduces great performance challenges, particularly when we want the viewing device to be light enough to wear.

So let’s propose an outline of an alternative technology that would fit these requirements.

Imagine an application-layer protocol for VR with the following characteristics:

A real-time, dynamic, stateful two-way client-server protocol. As such, it will be if not fully RTP then close to it.
It will carry 3D geometry, 2D textures, materials, audio, video and volumetric data mainly from server to client.
It will carry control inputs, including spatial, binary and analogue inputs; as well as video, audio and other data from client to server.
Almost all of the application logic will be processed at the server.
Parts of the app logic that are latency-sensitive will occur at the client.
Less latency-sensitive parts of the rendering may occur at the server.
The final rendering and compositing will occur at the client.

We split the workload: let the client handle the near stuff, and the server everything else. The whole system of client and server will form essentially two “loops”:

The far loop between client input and server logic, which has network latency.
The near loop within the client, which has local latency.

I think that this system could handle almost every current application of XR nicely, and enable a wide range of apps that would struggle in a standalone, installable format. I speak of “the HTTP of VR” rather than the “HTML” because as a real-time system, most of the data will be in binary form; we’re less interested in a human-readable data format, and more in the language of transmission.

An Open Protocol

Much of the above could be considered a fair description of a few VR apps already: particularly the more generic “social network” applications that download a large amount of content on the fly to represent avatars, scenery and so on. But these apps are over-specified for what I have in mind, the definition of a “thick client”. And crucially, each client is proprietary - the apps don’t talk to each other.

The key is that the protocol must be open.

Imagine a world where every XR device has a lightweight client application, a “VR browser”. Where VR experiences of all types - games, social networks, education, simulation - are all accessible by simply launching the browser and connecting to the appropriate server. Where it doesn’t matter what headset you have, because all the best apps are online via the protocol: no need to download and install, just connect and go. Where as a developer you don’t need to worry about fitting your entire application in a few gigabytes, because it lives on your server/s, not taking up space on a heavy, head-mounted hard-drive. And neither do you have to deploy updates to hundreds or thousands of users - you mostly just update the server-side.

It seems to me that in this way, the Metaverse could really get going: with network effects, discoverability, openness. Low barriers to entry and a free marketplace of content. These are the factors that allowed the Web to grow and thrive.

By moving beyond the Web with its own open, native protocol, the Metaverse can do the same.

So at Simul, for the past few years we’ve been building this protocol: it’s called Teleport VR. Let’s see what we can make with it!

Roderick Kennedy

The HTTP of VR

27 Nov

Written By Roderick Kennedy

The Metaverse needs a network protocol: it needs its own “HTTP”.

In the Beginning

It’s discontinuous: each request returns a specific, finite response.
It’s stateless: the server is never required to retain information about a client between requests.
It’s unidirectional: links are one-way, and you don’t need permission from the target server to link to it.

But in VR it’s a different story: every app is a silo, barely connected with the wider Metaverse.

VR on the Web

Spatial computing will remain a secondary consideration.

If the Metaverse is to achieve its full potential, it needs its own protocol.

The HTTP of VR

So let’s propose an outline of an alternative technology that would fit these requirements.

Imagine an application-layer protocol for VR with the following characteristics:

A real-time, dynamic, stateful two-way client-server protocol. As such, it will be if not fully RTP then close to it.
It will carry 3D geometry, 2D textures, materials, audio, video and volumetric data mainly from server to client.
It will carry control inputs, including spatial, binary and analogue inputs; as well as video, audio and other data from client to server.
Almost all of the application logic will be processed at the server.
Parts of the app logic that are latency-sensitive will occur at the client.
Less latency-sensitive parts of the rendering may occur at the server.
The final rendering and compositing will occur at the client.

We split the workload: let the client handle the near stuff, and the server everything else. The whole system of client and server will form essentially two “loops”:

The far loop between client input and server logic, which has network latency.
The near loop within the client, which has local latency.

An Open Protocol

The key is that the protocol must be open.

By moving beyond the Web with its own open, native protocol, the Metaverse can do the same.

So at Simul, for the past few years we’ve been building this protocol: it’s called Teleport VR. Let’s see what we can make with it!

Roderick Kennedy

A virtual keyboard in Dear ImGui

Roderick Kennedy — Wed, 10 Nov 2021 13:48:14 GMT

A virtual keyboard in Dear ImGui

10 Nov

Written By Roderick Kennedy

There are a few kinks implementing a virtual keyboard in Dear ImGui - the main problem is getting key inputs from the buttons into the InputText item.

Clicking a button changes focus to the button - away from the InputText. You can switch it back on the next pass, but ImGui won’t recognize it as being ready for input until two frames later.
So I implemented a counter called “refocus”. It’s set to zero every time we click a keyboard button. It increments every frame. Only when it gets to 2 or greater do we pull the inputs from out of our buffer.

Our variables are:

std::vector keys_pressed;

int refocus=0;

Here’s the code:

ImGui::Begin("Virtual Keyboard"); io.KeysDown[VK_BACK] = false; if(refocus==0) { ImGui::SetKeyboardFocusHere(); } else if(refocus>=2) { while(keys_pressed.size()) { int k=keys_pressed[0]; if(k==VK_BACK) { io.KeysDown[k] = true; } else { io.AddInputCharacter(k); } keys_pressed.erase(keys_pressed.begin()); } } static char buf[500]; if(ImGui::InputText("", buf, IM_ARRAYSIZE(buf))) { current_url=buf; } refocus++;

We define a simple lambda function to add a line of keys to the keyboard:

auto KeyboardLine = [&io,this](const char key) { size_t num = strlen(key); for (size_t i = 0; i < num; i++) { char key_label[] = "X"; key_label[0] = key; if (ImGui::Button(key_label,ImVec2(46,32))) { refocus=0; keys_pressed.push_back(*key); } key++; if (i

This allows us to add the number keys simply with:

KeyboardLine("1234567890-"); ImGui::SameLine();

Now here’s our backspace arrow button:

if (ImGui::Button(ICON_FK_LONG_ARROW_LEFT,ImVec2(92,32))) { refocus=0; keys_pressed.push_back(ImGuiKey_Backspace); }

Using a Text() call to insert a bit of padding in front of the next row of keys:

ImGui::Text(" "); ImGui::SameLine(); KeyboardLine("qwertyuiop"); ImGui::Text(" "); ImGui::SameLine(); KeyboardLine("asdfghjkl"); ImGui::SameLine(); if (ImGui::Button("Return",ImVec2(92,32))) { refocus=0; keys_pressed.push_back(ImGuiKey_Enter); } ImGui::Text(" "); ImGui::SameLine(); KeyboardLine("zxcvbnm,./"); ImGui::End();

And done.

Roderick Kennedy

A virtual keyboard in Dear ImGui

10 Nov

Written By Roderick Kennedy

There are a few kinks implementing a virtual keyboard in Dear ImGui - the main problem is getting key inputs from the buttons into the InputText item.

Our variables are:

std::vector keys_pressed;

int refocus=0;

Here’s the code:

We define a simple lambda function to add a line of keys to the keyboard:

This allows us to add the number keys simply with:

KeyboardLine("1234567890-"); ImGui::SameLine();

Now here’s our backspace arrow button:

if (ImGui::Button(ICON_FK_LONG_ARROW_LEFT,ImVec2(92,32))) { refocus=0; keys_pressed.push_back(ImGuiKey_Backspace); }

Using a Text() call to insert a bit of padding in front of the next row of keys:

And done.

Roderick Kennedy

Accessing MongoDB from Nextjs

Roderick Kennedy — Fri, 22 Oct 2021 22:18:04 GMT

Following the example of

www.mongodb.com/developer/how-to/nextjs-with-mongodb/ you soon run into difficulty - as of October 2021, the version of the nextjs Mongo integration initialized by

npx create-next-app --example with-mongodb mflix

as the tutorial expects to find the file util/mongodb.js and the function connectToDatabase it contains, which is missing - because create-next-app here checks out github.com/vercel/next.js/tree/canary/examples/with-mongodb to the “canary” branch. You have to either check out the branch “master” of that repo, or manually add the mongodb.js file from that branch into the project it created.

Then, the tutorial skips over a lot of crucial information about how to use the API. The answer to obtaining a single result from the API is to create a file e.g. [movie_id].js in pages/api/movies, and fill it with:

import { connectToDatabase } from "../../../util/mongodb";

const {ObjectId} = require('mongodb');

export default async function handler(req, res)

{

const { movie_id } = req.query;

const { db } = await connectToDatabase();

var o_id = new ObjectId(movie_id);

const movie_info = await db

.collection("movies")

.find({'_id' : o_id})

.toArray();

res.json(movie_info);

}

The key here is getting the definition of ObjectId from the mongodb node module, initializing o_id as an ObjectId from the 24-char movie idm then using it in the find() function of the collection object.

Overall, I’m thinking Mongo/Nextjs is insufficiently documented for my needs - my search for a node data backend continues.

Changing the name of a variable in CMake

Roderick Kennedy — Fri, 27 Nov 2020 14:18:00 GMT

Here's a neat trick in CMake: you want to change the name of a variable, but worry that anyone you've distributed the code to already will lose the option they've selected.

Use the old variable as the default value for the new one:

option(OLD_VARIABLE "Some variable" ON) option(NEW_VARIABLE "Some variable" ${OLD_VARIABLE})

or...

set( OLD_STRING_VARIABLE "Old default" CACHE STRING "Help text" )

set( NEW_STRING_VARIABLE "${OLD_STRING_VARIABLE}" "Help text" )

Passing an array of structs from C++ to C#

Roderick Kennedy — Thu, 12 Nov 2020 14:17:00 GMT

To pass an array of structs from C++ to C#, you can pass a pointer to a C-style array. In C++ you may have a struct, e.g.

#pragma pack(push) #pragma pack(1) struct InputEvent { uint32_t eventId; float floatValue; uint32_t intValue; }; #pragma pack(pop)
The delegate in C++ is:
typedef void(__stdcall* ProcessNewInputFn) (int numEvents, const InputEvent**); Telling C++ what C# function to call:

extern "C" __declspec(dllexport) void SetInputProcessingDelegate(ProcessNewInputFn newInputProcessing) { processNewInput = newInputProcessing; }

Using this from C++

std::vector inputEvents;
const avs::InputEvent *v=inputEvents.data(); processNewInput(inputEvents.size(), &v);

In C# the struct is defined as:

[StructLayout(LayoutKind.Sequential, Pack = 1)] public struct InputEvent { public UInt32 eventId; public float floatValue; public UInt32 intValue; };
Note the packing! It must match what we had in C++. Now C# must declare the the delegate type it will implement:

[UnmanagedFunctionPointer(CallingConvention.StdCall)] delegate void OnNewInput(int numEvents, in IntPtr newEvents);

And declare in C# the C++ function that sets the delegate:

[DllImport("dllname")] static extern void SetInputProcessingDelegate(OnNewInput onNewInput );

This is called with

ok = SetInputProcessingDelegate(ProcessingClass.SetInput);

Where we have a class like this:
class ProcessingClass { public static void StaticProcessInput(int numEvents, in IntPtr inputEventsPtr ) { int EventSize = System.Runtime.InteropServices.Marshal.SizeOf(typeof(avs.InputEvent)); avs.InputEvent[] inputEvents = new avs.InputEvent[inputState.numEvents]; IntPtr ptr= inputEventsPtr; for (int i = 0; i < inputState.numEvents; i++) { inputEvents[i]=Marshal.PtrToStructure(ptr); ptr += EventSize; } } }
Here, we take the C++ style pointer-to-array, and iterate through the array elements, copying each in turn into the C# style array.

Resolving conflicts between Qt versions

Roderick Kennedy — Thu, 14 Nov 2019 14:16:00 GMT

Resolving conflicts between Qt versions

14 Nov

Written By Roderick Kennedy

The trueSKY plugin for Unreal uses Qt dll's for UI. Unfortunately so do some other plugins. Because Windows just uses whichever version of a dll was loaded first, this leads to (for example) crashes in trueSKY UI because it tries to access the wrong parts of a dll loaded by Quixel Megascans.

To solve this we recompile Qt using the switch -qtlibinfix to modify the output filenames. Thus instead of Qt5Core.dll we get Qt5Core_simul.dll etc.

No more conflicts!

UPDATE: To compile Qt, follow the instructions at https://wiki.qt.io/Building_Qt_5_from_Git. For example, for me on Windows, I must git-clone the repo, install perl (!) and call perl init-repository.

Then, I create a subdirectory BUILD_DIR at subdirectory "build/x64". From there, I call:

call ../../configure.bat -qtlibinfix %QT_INFIX% -prefix %BUILD_DIR%\qtbase -skip qtwebengine -developer-build -%reldeb% -force-debug-info -no-warnings-are-errors -L kernel32 -opengl desktop -opensource -make libs -make tools -nomake examples -nomake tests -platform win32-msvc -confirm-license -no-compile-examples -qt-zlib -plugin-manifests -no-angle -qt-freetype -qt-libjpeg -qt-libpng -D U_STATIC_IMPLEMENTATION %INC% %LIBDIRS% %IC%

QT_INFIX is _simul, while INC LIBDIRS and IC are extra compile options.

Finally, we run nMake to build Qt.

Roderick Kennedy

Resolving conflicts between Qt versions

14 Nov

Written By Roderick Kennedy

To solve this we recompile Qt using the switch -qtlibinfix to modify the output filenames. Thus instead of Qt5Core.dll we get Qt5Core_simul.dll etc.

No more conflicts!

UPDATE: To compile Qt, follow the instructions at https://wiki.qt.io/Building_Qt_5_from_Git. For example, for me on Windows, I must git-clone the repo, install perl (!) and call perl init-repository.

Then, I create a subdirectory BUILD_DIR at subdirectory "build/x64". From there, I call:

QT_INFIX is _simul, while INC LIBDIRS and IC are extra compile options.

Finally, we run nMake to build Qt.

Roderick Kennedy

Finding and removing files added to git by accident

Roderick Kennedy — Thu, 02 Aug 2018 13:14:00 GMT

Finding and removing files added to git by accident

2 Aug

Written By Roderick Kennedy

If for example, you've added lib files by mistake to a large git repo, and want to remove them, but don't know the exact paths, use this:

git ls-files *.lib>lib.bat

Then in lib.bat you may have e.g.:
Plugins/Media/Intermediate/Build/Win64/DebugMediaEditor/MediaEditor-Win64-Debug.lib

Add git rm --cached to the front of each line, then run the batch file and commit the result.

Roderick Kennedy

How to make a custom Wizard for Unreal Editor

Roderick Kennedy — Thu, 21 Jun 2018 13:13:00 GMT

How to make a custom Wizard for Unreal Editor

21 Jun

Written By Roderick Kennedy

I wanted to create a wizard in the trueSKY Unreal plugin that would make it easier for users to add trueSKY to UE scenes. I was following this video where Epic's Michael Noland describes various ways to modify the Editor. So I made a custom Property Editor window with settings to select a sky sequence, create a TrueSkyLight etc.

But it didn't look very friendly. And implementing a wizard-style Apply button just put a button in amongst the other settings - not great. After some searching in the UE codebase, I discovered the SWizard class that Unreal Editor uses for its own wizards. Here's what you do:

1. Create a class derived from SCompoundWidget containing a TSharedPtr. Mine looks like this:

DECLARE_DELEGATE_FourParams( FOnTrueSkySetup, bool, ADirectionalLight* ,bool, UTrueSkySequenceAsset *);

#define S_DECLARE_CHECKBOX(name) \ bool name; \ ECheckBoxState Is##name##Checked() const { return name ? ECheckBoxState::Checked:ECheckBoxState::Unchecked;} \ void On##name##Changed(ECheckBoxState InCheckedState) {name=(InCheckedState==ECheckBoxState::Checked);}

class STrueSkySetupTool : public SCompoundWidget { public: SLATE_BEGIN_ARGS( STrueSkySetupTool ) :_CreateTrueSkyLight(false) ,_DirectionalLight(nullptr) ,_CreateDirectionalLight(nullptr) ,_Sequence(nullptr) {} / A TrueSkyLight actor performs real-time ambient lighting.*/ SLATE_ARGUMENT(bool,CreateTrueSkyLight) / TrueSKY can drive a directional light to provide sunlight and moonlight./ SLATE_ARGUMENT(ADirectionalLight,DirectionalLight) / If there's no directional light in the scene, you can create one with this checkbox.*/ SLATE_ARGUMENT(bool,CreateDirectionalLight) / The TrueSKY Sequence provides the weather state to render./ SLATE_ARGUMENT(UTrueSkySequenceAsset ,Sequence) / Event called when code is successfully added to the project */ SLATE_EVENT( FOnTrueSkySetup, OnTrueSkySetup ) SLATE_END_ARGS() / Constructs this widget with InArgs */ void Construct( const FArguments& InArgs );

/* Handler for when cancel is clicked / void CancelClicked();

/* Returns true if Finish is allowed / bool CanFinish() const;

/* Handler for when finish is clicked / void FinishClicked();

...

S_DECLARE_CHECKBOX(CreateTrueSkyLight) S_DECLARE_CHECKBOX(ShowAllSequences)

void SetupSequenceAssetItems();

void CloseContainingWindow(); private: /* The wizard widget / TSharedPtr MainWizard; FOnTrueSkySetup OnTrueSkySetup; ... };

The SLATE_ARGUMENT macros allow initialization of named parameters in this style:

TSharedRef TrueSkySetupTool = SNew(STrueSkySetupTool).OnTrueSkySetup(OnTrueSkySetup1).CreateTrueSkyLight(true);

etc. This is super-useful.

2. Create a callback for the wizard to execute:

FOnTrueSkySetup OnTrueSkySetupDelegate;

3. Create a window for the widget. This function is called when the menu option to start the wizard is selected:

void FTrueSkyEditorPlugin::OnAddSequence() { TrueSkySetupWindow = SNew(SWindow) .Title( NSLOCTEXT("InitializeTrueSky", "WindowTitle", "Initialize trueSKY") ) .ClientSize( FVector2D(600, 550) ) .SizingRule( ESizingRule::FixedSize ) .SupportsMinimize(false).SupportsMaximize(false); OnTrueSkySetupDelegate.BindRaw(this,&FTrueSkyEditorPlugin::OnTrueSkySetup); TSharedRef TrueSkySetupTool = SNew(STrueSkySetupTool).OnTrueSkySetup(OnTrueSkySetupDelegate); TrueSkySetupWindow->SetContent( TrueSkySetupTool );

If the main frame exists parent the window to it. The main frame should always exist...

TSharedPtr< SWindow > ParentWindow; if( FModuleManager::Get().IsModuleLoaded( "MainFrame" ) ) { IMainFrameModule& MainFrame = FModuleManager::GetModuleChecked( "MainFrame" ); ParentWindow = MainFrame.GetParentWindow(); }

bool modal=false; if (modal) { FSlateApplication::Get().AddModalWindow(TrueSkySetupWindow.ToSharedRef(), ParentWindow); } else if (ParentWindow.IsValid()) { FSlateApplication::Get().AddWindowAsNativeChild(TrueSkySetupWindow.ToSharedRef(), ParentWindow.ToSharedRef()); } else { FSlateApplication::Get().AddWindow(TrueSkySetupWindow.ToSharedRef()); } TrueSkySetupWindow->ShowWindow(); }

4. Implement the setup tool:

BEGIN_SLATE_FUNCTION_BUILD_OPTIMIZATION void STrueSkySetupTool::Construct( const FArguments& InArgs ) { OnTrueSkySetup = InArgs._OnTrueSkySetup; CreateTrueSkyLight=InArgs._CreateTrueSkyLight; DirectionalLight=InArgs._DirectionalLight; Sequence=InArgs._Sequence; ...

The interface to build the actual UI is really interesting. By overloading the [] and + operators, Epic lets you specify the widget structure like so:

ChildSlot [ SNew(SBorder) .Padding(18) .BorderImage( FEditorStyle::GetBrush("Docking.Tab.ContentAreaBrush") ) [ SNew(SVerticalBox) +SVerticalBox::Slot() [ SAssignNew( MainWizard, SWizard) .ShowPageList(false) .CanFinish(this, &STrueSkySetupTool::CanFinish) .FinishButtonText( LOCTEXT("TrueSkyFinishButtonText", "Initialize") ) .OnCanceled(this, &STrueSkySetupTool::CancelClicked) .OnFinished(this, &STrueSkySetupTool::FinishClicked) .InitialPageIndex( 0) +SWizard::Page() [ SNew(SVerticalBox) +SVerticalBox::Slot() .AutoHeight() [ SNew(STextBlock) .TextStyle( FEditorStyle::Get(), "NewClassDialog.PageTitle" ) .Text( LOCTEXT( "WeatherStateTitle", "Choose a Sequence Asset" ) ) ] +SVerticalBox::Slot() .AutoHeight() .Padding(0) [ SNew(SHorizontalBox) +SHorizontalBox::Slot() .FillWidth(1.f) .VAlign(VAlign_Center) [ SNew(STextBlock) .Text(LOCTEXT("TrueSkySetupToolDesc", "Choose which weather sequence to use initially.") ) .AutoWrapText(true) .TextStyle(FEditorStyle::Get(), "NewClassDialog.ParentClassItemTitle") ] ] ] +SWizard::Page() [ ... ] ] ] ];

}

So by adding new +SWizard::Page() elements we add pages to the wizard.

5. Finally, implement the callback that the delegate calls when you click "Finish":

void FTrueSkyEditorPlugin::OnTrueSkySetup(bool CreateDirectionalLight, ADirectionalLight DirectionalLight,bool CreateTrueSkyLight,UTrueSkySequenceAsset Sequence) { ... }

The end result looks like this:

Full source for this is at our UE branch, (register at Simul to access).

Roderick Kennedy

How to make a custom Wizard for Unreal Editor

21 Jun

Written By Roderick Kennedy

1. Create a class derived from SCompoundWidget containing a TSharedPtr. Mine looks like this:

DECLARE_DELEGATE_FourParams( FOnTrueSkySetup, bool, ADirectionalLight* ,bool, UTrueSkySequenceAsset *);

/* Handler for when cancel is clicked / void CancelClicked();

/* Returns true if Finish is allowed / bool CanFinish() const;

/* Handler for when finish is clicked / void FinishClicked();

...

S_DECLARE_CHECKBOX(CreateTrueSkyLight) S_DECLARE_CHECKBOX(ShowAllSequences)

void SetupSequenceAssetItems();

void CloseContainingWindow(); private: /* The wizard widget / TSharedPtr MainWizard; FOnTrueSkySetup OnTrueSkySetup; ... };

The SLATE_ARGUMENT macros allow initialization of named parameters in this style:

TSharedRef TrueSkySetupTool = SNew(STrueSkySetupTool).OnTrueSkySetup(OnTrueSkySetup1).CreateTrueSkyLight(true);

etc. This is super-useful.

2. Create a callback for the wizard to execute:

FOnTrueSkySetup OnTrueSkySetupDelegate;

3. Create a window for the widget. This function is called when the menu option to start the wizard is selected:

If the main frame exists parent the window to it. The main frame should always exist...

4. Implement the setup tool:

The interface to build the actual UI is really interesting. By overloading the [] and + operators, Epic lets you specify the widget structure like so:

}

So by adding new +SWizard::Page() elements we add pages to the wizard.

5. Finally, implement the callback that the delegate calls when you click "Finish":

void FTrueSkyEditorPlugin::OnTrueSkySetup(bool CreateDirectionalLight, ADirectionalLight DirectionalLight,bool CreateTrueSkyLight,UTrueSkySequenceAsset Sequence) { ... }

The end result looks like this:

Full source for this is at our UE branch, (register at Simul to access).

Roderick Kennedy

Signing installers with certificates

Roderick Kennedy — Tue, 10 Apr 2018 13:09:00 GMT

Signing installers with certificates

10 Apr

Written By Roderick Kennedy

Windows Defender has recently decided to falsely mark all of our installers as containing some virus or other.

It'll be a long long while before they get around to questioning whether their algorithms are in fact, "full of it", as they say, so let's see what happens if we sign our executables using a root certificate.

First, get a certificate, from Comodo. This takes weeks while they check whether an arbitrary non-governmental organization, Dun and Bradstreet, regards your company as genuine. Just check with Companies House? Way too simple!

So you need to get a DUNS number from D&B, then buy a certificate from tucows/Comodo.

After jumping through their hoops (which don't seem to be very secure to me, just cumbersome), you'll get a .crt file.

Then https://support.citrix.com/article/CTX221295 will tell you how to convert your crt to a pfx.

Finally, use the pfx and signtool.exe (in the Windows SDK) to sign your executable.

Roderick Kennedy

Signing installers with certificates

10 Apr

Written By Roderick Kennedy

Windows Defender has recently decided to falsely mark all of our installers as containing some virus or other.

So you need to get a DUNS number from D&B, then buy a certificate from tucows/Comodo.

After jumping through their hoops (which don't seem to be very secure to me, just cumbersome), you'll get a .crt file.

Then https://support.citrix.com/article/CTX221295 will tell you how to convert your crt to a pfx.

Finally, use the pfx and signtool.exe (in the Windows SDK) to sign your executable.

Roderick Kennedy

Building Unreal Engine projects from the solution using MSBuild

Roderick Kennedy — Fri, 17 Nov 2017 14:08:00 GMT

17 Nov

Written By Roderick Kennedy

If you want to use MSBuild to build UE4 projects, but need to build them from within the solution instead of specifying the vcxproj file (which doesn't always work correctly), you need to use the "Target" /t: command line parameter, like so:

"path to MSBuild.exe" /t:Engine\UE4 /p:Configuration="Development Editor" /p:Platform=Win64 UE4.sln

Key things to note:

the configuration and platform specifiers are Solution-style, with spaces instead of underscores, and Win64 instead of x64 etc.
The solution folder path must be specified in the target parameter, otherwise MSBuild will not recognize the project name.

Roderick Kennedy

Building Unreal Engine projects from the solution using MSBuild

17 Nov

Written By Roderick Kennedy

"path to MSBuild.exe" /t:Engine\UE4 /p:Configuration="Development Editor" /p:Platform=Win64 UE4.sln

Key things to note:

the configuration and platform specifiers are Solution-style, with spaces instead of underscores, and Win64 instead of x64 etc.
The solution folder path must be specified in the target parameter, otherwise MSBuild will not recognize the project name.

Roderick Kennedy

Advanced custom Qt Container Widgets and Qt Designer

Roderick Kennedy — Sat, 03 Jun 2017 13:07:00 GMT

3 Jun

Written By Roderick Kennedy

Qt has a nice UI editor called Designer, and you can create custom widgets that go in Designer's toolkit. But the only example I've ever found is this one in the Qt docs, which doesn't explain how to create container widgets.

The problem is to create a widget that contains some decoration or controls, but also has a sub-window where people can put their own widgets.

For example, I wanted an "accordion" control that had a checkbox at the top to open and close it, then to be able to put any other control inside this.

The way it will be structured is a QAccordion, with a VBoxLayout, will contain a QCheckbox and a QWidget called the content widget. This content widget will have its own VBoxLayout, where the controls will go.

You will create two classes: one called (say) QAccordion, which implements the widget, and one called QAccordionInterface, which tells Designer that it's available.

<#include #include "GeneratedFiles/ui_QAccordion.h" #include "Export.h"

class SIMUL_QT_WIDGETS_EXPORT QAccordion : public QWidget { Q_OBJECT

Q_PROPERTY(QString title READ title WRITE setTitle DESIGNABLE true) Q_PROPERTY(bool open READ isOpen WRITE setOpen DESIGNABLE true) public: QAccordion(QWidget *parent = 0); ~QAccordion(); void setTitle(QString f); QString title() const; void setOpen(bool o); bool isOpen() const; public slots: void on_accordionCheckBox_toggled(); void setSearchText(const QString &); signals: protected: void childEvent ( QChildEvent * event ) override; void paintEvent(QPaintEvent *) override; private: Ui::Accordion ui; bool setup_complete; QWidget *contentsWidget; void hookupContentsWidget(); };

The subclass Ui::Accordion shows that I created the basic class in Designer itself. This is optional, but the QAccordion.ui file is just:

?xml version="1.0" encoding="UTF-8"? Accordion0067133600AccordionAccordiontrue

By putting the layout and checkbox in the ui file, they will be created in code, in Ui::Accordion.

But: if we were to create the whole thing, including the contents widget in here, after we built the class the contents widget would NOT be accessible in Designer, and neither would its layout be recognized. So instead we put these in a function called domXml in QAccordionInterface.

QString QAccordionInterface::domXml() const { return "\n" " \n" " \n" " \n" " 0\n" " 0\n" " 100\n" " 24\n" " \n" " \n" " \n" " \n" " \n" " \n" " .\n" " \n" " \n" " \n" " \n" " 2\n" " \n" " \n" " 2\n" " \n" " \n" " 2\n" " \n" " \n" " 2\n" " \n" " \n" " 2\n" " \n" " \n" " \n" " \n" "\n"; }

By specifying the contents widget and its layout here, Designer will know to dynamically create them when you add a QAccordion, so they'll appear in the editor. You can then drag any control into the contents widget, and it will be correctly positioned. Be careful that you drag it to the contents widget and not the QAccordion itself or a subcontrol. Designer doesn't properly obey its "isContainer" function, so it sees any custom control as a container, not just the ones you indicate.

So now in designer, we can add QAccordions. Without styling they just look like checkboxes with a space below where you can drag controls:

After applying some styling, the final result looks like this:

The accordion elements - Cloud Window, Precipitation etc are inside a searchable property panel, implemented on the same principles.

And here are the files for the final class:

QAccordion.zip