Virtual Pinboard

March 2, 2015 Migrated from Tumblr

<iframe width="560" height="315" src="

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

This project was a virtual pinboard, which takes the depth information from the new RealSense tablet, and spits that into pins coming at you. I haven’t had many projects which have been so many problems for such small output.  #Sterling related it to Playstation's Black Triangle Principal.  

Here is a link to the GITHUB directory

I’m not going to go over everything in these scripts - but here is a high level:

Setting up our References

In DsapiPlugin.Build.cs:

we are going to force exceptions and add Unreal modules.

This is the file where we set our file paths, in our case we’re setting up for DSAPI (the libraries which use the camera data) and libmotion (the libraries we’re using to access the second set of gyro sensors in the tablet)

To work more smoothly with Unreal, we setup our Linker paths to point to our unreal project.  Our x64 path needs to point to UnrealProject/binaries/win64

Our win32 dll needs to be moved to the directory where our built unreal exe is.

We also need to setup our dependencies (in this case libmotiondll) and additional include directories (motion.h)

motion.h and *.lib (from the bin folder) needs to move to DsapiPlugins/plugins/thirdParty/include

Here is a more in depth look into include files, linker, and dlls

C++ and Raw Depth Data

Starting at class image in dsapiInstance.cpp

private:
    uint16_t * pixels;
    int width, height;

Sets up the expected value types of these variables

public:
void ReferenceDepthMap(DSAPI * ds)
{
    pixels = ds->getZImage();
    width = ds->zWidth();
    height = ds->zHeight();
}

This function exists so that you can setup an image to mirror the size of our incoming DSAPI depth map.

void Allocate(int w, int h)
{
    pixels = new uint16_t[w*h];
    width = w;
    height = h;
}

This allows us to setup a second image, specifying the values we want.  The uint16 is essentially a 1 dimensional array of values.  By setting an array of them that is the width * height, we’re creating a chunk of memory that has the appropriate number of pixels.

void SetPixel(int x, int y, uint16_t value)
{
    pixels[y * width + x] = value;
}

y * width + x (steps through our preset uint16 image by the width of the image * the vertical steps we want down, and then adds x, which moves us in the horizontal axis)

It essentially lets us feed an x and y value to access a pixel on an image, and then feeds the information to the computer in a way that lets us access that specific pixel on a 1D array.

uint16_t GetPixel(int x, int y)
{
    return pixels[y * width + x];
}

This does the same thing as the last function, but it returns a value.

int GetWidth() { return width; }

allows us to read the value, while keeping it protected (since its being called from our private section.

int MapLinearRange(int value, int fromLow, int fromHigh, int toLow, int toHigh)
{
    return toLow + (value - fromLow) * (toHigh - toLow) / (fromHigh - fromLow);
}

is one of my favorite math equations, basically it translates one range of values to a secondary range of values, and then allows you to pass a number along, and feeds it back scaled to the new value set.  

void ADsapiInstance::ReceiveTick(float DeltaSeconds)

RecieveTick, when put after :: in a declaring function, means it inherits from that class.  In this case, its the equivalent of putting it in an update loop.

Image depthBuffer;
depthBuffer.ReferenceDepthMap(ds); // Point to DSAPI’s depth image, and make sure we have the right size

Image secondaryImage;
secondaryImage.Allocate(60, 46);

Here we are declaring an Image (depth buffer, which again, C++ is interpreting as a 1D array of uint16_t) and then we’re using ReferenceDepthMap to link it to the camera feed.
We do the same thing with secondary Image, but we are feeding it the size we want.

The purpose of having 2 images is that we want to lower the res of the depth image, and feed it into a lower res image.  Each pixel in that secondary image will control one of the pins on the pinboard.  Originally I had it as low as 20x20, but with some tweaking, I eventually bumped it up to 60 x 46.

for (int y = 0; y < secondaryImage.GetHeight(); ++y)
    {
    for (int x = 0; x < secondaryImage.GetWidth(); ++x)
    {
        int otherY = MapLinearRange(y,   0, secondaryImage.GetHeight() - 1,   0,        depthBuffer.GetHeight() - 1);
       int otherX = MapLinearRange(x,   0, secondaryImage.GetWidth() - 1,                    58, depthBuffer.GetWidth() - 1);

       uint16_t sample1 = depthBuffer.GetPixel(otherX, otherY);
       uint16_t sample2 = depthBuffer.GetPixel(otherX + 1, otherY);
       uint16_t sample3 = depthBuffer.GetPixel(otherX, otherY + 1);
       uint16_t sample4 = depthBuffer.GetPixel(otherX + 1, otherY + 1);
       uint16_t sampleAverage = (sample1 + sample2 + sample3 + sample4) / 4;

       //if (sampleAverage<1500)
       // secondaryImage.SetPixel(x, y, sampleAverage);
       secondaryImage.SetPixel(x, y, sample1);
        //MyDepthValues[y * secondaryImage.GetWidth() + x] = (sample1 +                 sample2 + sample3 + sample4) / 4;
        MyDepthValues[y * secondaryImage.GetWidth() + x] = sample1;
    }
}

This bit implements a poor man’s scale down.  For every pixel in the second image, (the size of which we just declared) we are going to do the following:

create a variable called otherY.  We use our MAP function (which converts one range to another range) and convert the range of our secondary image’s Y to our depth buffer’s Y.  We subtract 1 from our values because an array starts on 0.  This gives us a relational reference between a point in a large image, and where the same point on a small image would be.  (same for other X)

Once we have this point, we feed in the x and y values in our for loop arrays to determine their relational values in the depth map.  If we want to blur it, we can grab the pixels around that one, add them, and divide them to get a rough blurring.  One thing to be cautious of is that on edge case pixels, it will throw you fake data if you’re trying to sample pixels that don’t exist.

We then go in and set the pixel in our new image map of said number crunching.

We then push all this to a Tarray called MyDepthValues, which we are initializing in dsapiInstance.h with this line:

UPROPERTY(Category = “Pixel Values”, BlueprintReadOnly, transient, VisibleAnywhere) TArray<int32> MyDepthValues;

This is how we make a variable public to unreal: it needs to be defined as a UPROPERTY, a series of flags need to be set (the list is here)

Meinwhile in UDK…

Make an actor and have it inherit from our C++ script.  In the blueprint editor this is what you should see:  (Parent class is the important thing here)

Here is how our construction Script will look:

This section instantiates our pins on the start loop.  The first thing we do is clear any instances that might be lurking from previous builds.  Because the instance is all self contained, self as default on this one is correct.

Remember in our C++ file when we nested two for loops (for our x and y)?  We’re going to do the same thing here.

the first for loop pertains to our x and the second to our Y, and we need to match the min and max values to the ones we setup in our C++ file.  (For the first line of our y, run all of our x ones.  For the second line of our Y, run all our x values.  See how that similarly populates our 1D array of values?  Good, cause we’re feeding this logic from our uint16_t array)

So for each of those lines, we multiply the index number by a public variable, feed that into a location direction, make a vector (where this logic feeds our X into our X and our Y into our Z… because coordinate systems aren’t cohesive) and then instantiate at that point.  Changing the scale float will change how far apart they are.

For our Y values, we’re also going to subtract this location value from 0 to invert it.  We want each progressive line to get lower, not higher, because our camera feed pixels start in the upper left hand corner.

The components tab should now look like this.  In our static mesh section, I dropped a 3D model of a pin.

In our construction script, we created an instance for every line of x and y of the 3D model which was loaded in our component tab, which was this one.  Now, not only do we have an array of pins that we didn’t have to lay by hand, but they’re also populating an array which we can now drive.

Here, I’ve grey'ed out the values which are not relevant to updating the pin position.  (I’ll talk about that stuff later)

We start with a for loop where the max number is the max number of pixels in our “second image”…. (so our x * y - 1, to compensate for the fact that arrays start at 0)

From our instance array, we get the instance transform.  (And we seed the instance index with our for loop).  We break the transform and feed the rotation and scale back into a make transform.  We break the location’s vector and make a new vector out of its X and Z values.  This allows us to update ONLY the Y axis of the instances’ transforms, leaving all their other variables at their unique starting position.

To change our Y transforms, we get our depthValues, which is an array populated directly from our C++ script.  We are able to call this, because our script inherits from dsapiInstance.h.  We do a “get” on this array and feed the array index from our for loop index.  We run the output through a clamp, in order to throw away data that is over 2000.  (So, any points from the camera which are further than 2000 mm will be discarded, this should help to cut back on noise a little).  We now have an array of points which range from 0 to 2000, and we want it to be transformed into something that makes sense with our Y coordinates.  We do that with map (this blueprint node is the same as the equation in our C++ script).  We run 0 and 2000 as our old values and 0 and 40 as our new values.

We then feed this as the Y in our make vector, which then gets fed back into our make transform, which feeds our update instance transform.  So, for every index of the C++ array based on our SECOND IMAGE (which is a dumbed down version of our depth map) we grab the value, convert it to the min and max distance we want our pins travelling in Y, and then update the corresponding array of mesh instances, which we populated at runtime.

IMU Data

I’m not going to elaborate deeply on the first rabbit hole I descended.  I wanted to get the gyrometer data from the tablet, so that I could move the camera around the pinboard and see it in 3D.  The first thing I did was download the windows 8 sensor pack from microsoft.  After reading all the documentation and successfully getting a build from Visual Studio which was printing out the values, I found out that the RealSense tablets have a second set of sensors called IMU embedded which are much more reliable.

D2 had a script running which takes these values and exposes them for you, which was super helpful!  Unfortunately, it had a bunch of dependencies which Unreal couldn’t handle, so Sterling showed me how to wrap it all up into a dll. 

We just added the IMU calls into our dsapiInstance scripts, so that we could keep our already running pipeline going.

At the beginning, we made sure to 

#include “motion.h”

In our ADSAPIinstance::RecieveTick loop:

if (sensor && IsStarted(sensor))
{
    GetCurrentFrame(sensor);
    IMUYaw = GetYaw(sensor);
    IMUPitch = GetPitch(sensor);
    IMURoll = GetRoll(sensor);

    float quat[4];
    GetQuat(sensor, quat);
    GetCameraRotation = FRotator(FQuat(quat[0], quat[1], quat[2], quat[3]));
}

All of our calls are inheriting from motion.h.  So here we’re gathering the yaw, pitch and roll, and pushing them to Blueprint values in the dsapiInstance.h with this syntax:

UPROPERTY(Category = “IMU Readings”, BlueprintReadOnly, transient, VisibleAnywhere) float IMUYaw;

Sadly, hooking the yaw pitch and roll into a rotator on the camera were giving me really strange results, largely due to the fact that order REALLY matters when dealing with rotations.  I went back to the C++ script after a lot of failure, and added some values for quaternions, which are a lot more stable.

Relating to the last 3 lines…  GetQuat is a call you can make from motion.h.

Get camera rotation is pushing to our blueprint node, and calling the method FRotator converts the quaternion to a rotator.  This is a necessary step in exposing it, because blueprint doesn’t allow you to play with Quaternions.  The unreal forums explain that they “aren’t artist friendly” Feed it the Fquat, and feed it the quaternions from the sensor.

One final piece we need (outside of the RecieveTick loop) is this:

FRotator ADsapiInstance::FixRotatorBases(FRotator r)
{
    FQuat q1 = r.Quaternion();
    FQuat q2 = FQuat(-q1.Z, q1.X, -q1.Y, q1.W);
    return FRotator(q2);
}

This is being pushed through the blueprint node as a UFUNCTION that outputs an FRotator.  We are changing the call order on the X Y Z’s of our quaternions and flipping the appropriate axis, because (this will annoy me to my death) there is no standardization for XYZ space orientation between programs.  

Back to UDK…

I wanted the gyrometer data to affect the camera, so I made a blueprint script and put our camera in the component editor.  I wanted the camera to pivot around the pinboard object, so I moved it back in the x axis -300 (so that the object itself was offset from its pivot point).'

It turns out, touch events weren’t working on the Windows 8 tablets I was developing for.  My friend at Unreal says that that’s in development, but just not in there yet (not a lot of people develop for windows tablets).  So I made a public bool for touch, and just linked it up to left mouse events.

The first node up there is getCameraRotation, which is our FRotator (originally a quaternion) from our C++ script.  Its going to be putting out WHATEVER rotation its at though, and we want our rotation to start from wherever we’re at once we touch the screen.

in the second image we are setting the node originalRotation with whatever our quaternion’s value is on touch.  We run a negate rotator on that value and combine that with whatever our “current value rotation” is to create a reasonable start point.

We then have to run all that through our UFUNCTION (remember from our C++ script) which changes the rotator order in our quaternions to something reasonable.

The rotation was a bit extreme so I am multiplying it by .8 to slow it down (incidentally, you CAN’T break a rotator and slow down only a certain axis, that breaks everything and creates strange results).  All this is fed into a setActorRotation node, which is called every frame, IF the screen is being touched.  If its not, it reverts back to the original rotation (0, 0, 0)

Fake Raytracing with a shader

The last thing I wanted to do was create a reflection on the pins of the RGB, to give it some feeling of “being in the real world”.  Sterling explained that I chose the hardest case to try to do this in, and that he had no time to explain to me how to do it.  I was ready to say, “ok I don’t need that feature”, but he then proceeded to make it for me in 10 minutes.  I didn’t ask him to, but I also didn’t interrupt him.

Reflections only work in Unreal when they’re reflecting what is currently ON SCREEN.  Once the camera viewpoint is no longer looking at said object, it disappears in the reflection space.  Furthermore, it also takes issue with dynamic objects.  What we need to do in order to reflect a dynamic object not in screen space is to imitate raytracing.

Here’s a really good article on Wiki about it, with some equations

So, he essentially faked a raytracer using a shader:

Later, I sat down and reverse engineered the process.  Here is my journal on the subject:

We start off with the absoluteWorldPosition node.  Hooking this into the baseColor gives us a texture which has 8 different colors, pending on what quadrant you are in (with a center point of 0,0,0).

Breaking the node and feeding only the R into BaseColor gives us a black and white with the division in the X axis… meaning that absolute world division is broken into 3 floats, each changing value along the center axis.

The next major node is the ReflectionVector.  When the first 2 vectors are fed into a texture’s UV map, it changes the image’s angle as it relates to the camera angle.

First, we create a new float, and subtract the absoluteWorldPosition from this new float.  This changes the center point of our UVW space grid.

Next, we break out the components and use only the R value.  Like before, this isolates our vector space to the X axis.

Multiplying it by -1 reverses the direction of the UV mapping

Dividing this reversed coordinateSpace.x float from the x axis of the reflection vector changes the x coordinate, but only as it relates to the camera in the x axis.

Now we multiply this by the reflectionVector.  Multiplying the negative X value by a positive X value reverses the effects and adds back in the Y and the Z absoluteWorld Coordinate system.

I’m not 100%, but I suspect this moves the coordinate system from a (-1,1) system to a (0,1) system (see below)

Now we break off the Y and Z (or G and B) values and feed them into the UVs of a texture map.

Dividing the floats by our image width and image height scale them to the point where the image is recognizable.

Subtracting the (G B) UV map by 0.5 offsets it by half.  If we subtracted it by 1, we’d return to the original offset of 0.

Breaking out our UV values and reversing the second ( by subtracting from -1) reverses the image and sets it to right.

Now we start to go into our Coordinate logic.

if the U axis of our UV map is greater than 0, render the texture.  If it is less, render a color.

When we run that if logic through all for positions, we get a good result.  The image is rendered, at the right size and in relation to the camera position, and it is isolated.  Areas above, below and to the left and right of the image show grey.

This last node prevents the camera from reflecting the image if its looking at the pixels from behind

Conclusion

Being in world space, (overriding any natural uvs and making them dependent on world) we can apply this to our instanced pins, and it just spreads over all of them.  (Regardless of the fact that they’re copies of each other.  Score!)