Editing a Point Cloud

Project - Oct 6, 2013

Mar 03, 2024

Major learning takeaways from this project:

Unity limits the amount of faces a mesh can have to 60,000
Using math to convert a color to its greyscale
Microsoft Linq
Implementing new methods using static class, and ArgMin

Originally when designing this program, I imagined the feedback UI to be a mirror-image point cloud of the player, and Seth wrote this point cloud for it.

MeshPointCloud_backup.cs : Seth's original file

MeshPointCloud.cs : What I ended up with

Rather than generating particles, Seth is generating a mesh and setting its parameters. This is how it looks in action:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

My goal today was to take this and give it more visual cues, so the user knew where they were in space. He recast the

(Engine.DepthSize[0] - dx - mOffset.x) * CloudScale,
(Engine.DepthSize[1] - dy - mOffset.y) * CloudScale, (lmap((float)Engine.DepthBuffer[pid], 0, 2000, MinSceneDepth, MaxSceneDepth)) * CloudScale);

Section to a new vector3 and named it Pos.

_c is still calling the RGB pixels from the camera’s video stream He called the add function

mVerts.Add(pos-axisA-axisB);
mColors.Add(_c);

four times with a mix up of – and + so as to manifest a block of points instead of one, and then did some wizardry here:

mIndices.Add(vid+0);
mIndices.Add(vid+1);
mIndices.Add(vid+2);
mIndices.Add(vid+0);
mIndices.Add(vid+2);
mIndices.Add(vid+3);
vid+=4;

With that, he showed me how these additions allowed me access to the opacity and size channels of the point cloud, and from there I added 4 new components.

First, I played with all the available numbers to create this look:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

Unity limits the amount of faces a mesh can have to 60,000, so raising the number on the point cloud resolution was a good idea anyways. If you pump up the Point cloud size, a resolution of 2 or 3 look good. I went with 2 in this one, with a size of 0.032f. That kept my average point cloud mesh amount to around 10k, whereas at 1 it was bordering on ~55k, and would sometimes pop out when it went over 60k. I also changed the shader from “Diffuse” to “Transparent/VertexLit”.

Next, I threw in a sliding scale to affect the point size:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

var sizeTemp = lmap (pos.z,handPosMin,handPosMax,0.025f,0.035f);
float mySize = sizeTemp;
var axisA = new Vector3(1,0,0) * mySize;
var axisB = new Vector3(0,1,0) * mySize;

The lmap function at the bottom of the script remaps one set of values to another along the first input, and mySize affects the scaling of the PointCloud. handPosMin and Max are some magic numbers which I determined to be “as far back and forward as I’m comfortable reaching”, to give it a range to test things out in.

I did much the same thing for the opacity:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

var alphaTemp = lmap(pos.z,handPosMin,handPosMax,.25f,1f);
_c.a = alphaTemp;

And then Sterling showed me this concept- changing the greyscale on a slider.

var greyscaleTemp = lmap(pos.z,handPosMin,handPosMax,0,1);
float gray = _c.r * 0.2126f + _c.g * 0.7152f + _c.b * 0.0722f;
Color _cGray = new Color(gray, gray, gray, 1);
var _c1 = Color.Lerp(_c, _cGray, 0.1f); // Mostly c
var _c2 = Color.Lerp(_c, _cGray, 0.5f); // Halfway there
var _c3 = Color.Lerp(_c, _cGray, 0.9f); // Mostly cGray
_c = Color.Lerp(_c, _cGray, greyscaleTemp);

It’s based on this principal

The idea seemed suss, so I opened up Photoshop and a calculator and ran the math. On the bottom is the wiki formula. The far left is a random color I selected, the middle grey what happens when you convert to greyscale using Photoshop, and the one on the right is the color I get when I let the math equation dictate.

Here’s how it looked in practice:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

From that basis, I did a really simple pass with editing the color brightness.

When I did that, it turned my point cloud white. That made me look up Unity’s Color function, which I found out is in 0-1 color space, not 0-255. I converted the difference in photoshop to what that difference would be in (0,1) space, and plugged those numbers in instead.

var colorTemp = lmap(pos.z,handPosMin,handPosMax,0,1);
Color brighterColor = new Color(_c.r+0.16f,_c.g+0.18f,_c.b+0.09f,1);
_c = Color.Lerp(_c,brighterColor,colorTemp);

Result:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

I took these edits home with me, and after tweaking on it for a few days to get a decent feedback loop, which is linked as "MeshPointCloud.cs" in my Git Repository

The first thing I did was comment back in the size/distance distribution, and add a “bounding box” region to the alpha map, so that the hand would fade as you got close to the edge of the camera’s vision.

Alpha Slider

var rightLimit = lmap(pos.x,4.6f,5f,1f,0.5f);
var leftLimit = lmap(pos.x,-2.6f,-3f,1f,0.5f);
if(pos.x>=4.6f)
_c.a = rightLimit;
if(pos.x<=-2.6f)
_c.a = leftLimit;

Size Slider

var sizeTemp = lmap (pos.z,handPosMin,handPosMax,0.028f,0.044f);
float mySize = sizeTemp;
var axisA = new Vector3(1,0,0) * mySize;
var axisB = new Vector3(0,1,0) * mySize;

The next challenge was to create a localized ball of light within the hand, so that I could create feedback when the user was near/touching a reactive object. This part was a bit more difficult.

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

In the Update loop I wrote:

float fadeOffset = lmap (PrimaryColliders[0].transform.position.z,-2.5f,-7.3f,0.9f,2f);
var x_1_scale = lmap(PrimaryColliders[0].transform.position.z,-1f,-7.3f,0.7f,-1.1f);
var x_2_scale = lmap(PrimaryColliders[0].transform.position.z,-1f,-7.3f,2f,3.4f);
var y_1_scale = lmap(PrimaryColliders[0].transform.position.z,-1f,-7.3f,1.6f,0f);
var y_2_scale = lmap(PrimaryColliders[0].transform.position.z,-1f,-7.3f,3.3f,5.8f);
float x_mid = PrimaryColliders[0].transform.position.x + ((x_2_scale-x_1_scale)/2+x_1_scale);
float y_mid = PrimaryColliders[0].transform.position.y - ((y_2_scale-y_1_scale)/2+y_1_scale);
float z_offset = 3.5f;

This runs a check on where the top bounding box (x1) and bottom bounding box (x2) for both the x and y axis are on the hand, fade offset is a chunk of space over which the blend between hot and normal occur, and z offset how far backwards in space before the bounding box area is cutoff. I’m running the math in this way so that the “bounding box” scales as you move forward and backward in the scene. The figures themselves are just magic numbers I tweaked till it felt right.

Within the if (cx >= 0 && cx < Engine.RGBSize[0] && cy >= 0 && cy < Engine.RGBSize[1]) loop, (this runs a loop through each manifested “pixel” and applies to them individually) I added this:

Color hotspot = new Color(_c.r+brightnessAdditions,_c.g+brightnessAdditions,_c.b+brightnessAdditions,1);
var x_1_fade = lmap(pos.x,PrimaryColliders[0].transform.position.x +
x_1_scale+fadeOffset,PrimaryColliders[0].transform.position.x + x_1_scale,1,0);
var x_2_fade = lmap(pos.x,PrimaryColliders[0].transform.position.x + x_2_scale,PrimaryColliders[0].transform.position.x+x_2_scale+fadeOffset,1,0);
var y_1_fade = lmap(pos.y,PrimaryColliders[0].transform.position.y - y_1_scale-fadeOffset,PrimaryColliders[0].transform.position.y - y_1_scale,1,0);
var y_2_fade = lmap(pos.y,PrimaryColliders[0].transform.position.y -
y_2_scale+fadeOffset,PrimaryColliders[0].transform.position.y - y_2_scale,1,0);
if(pos.x <
PrimaryColliders[0].transform.position.x+x_1_scale+fadeOffset
&&pos.y < PrimaryColliders[0].transform.position.y-y_1_scale - (fadeOffset/2)
&&pos.y > PrimaryColliders[0].transform.position.y-y_2_scale + (fadeOffset/2)
&&pos.z < PrimaryColliders[0].transform.position.z - z_offset)
_c = Color.Lerp(_c,hotspot,x_1_fade);
if(pos.x > PrimaryColliders[0].transform.position.x+x_2_scale
&&pos.y < PrimaryColliders[0].transform.position.y-y_1_scale - (fadeOffset/2)
&&pos.y > PrimaryColliders[0].transform.position.y-y_2_scale + (fadeOffset/2)
&&pos.z < PrimaryColliders[0].transform.position.z - z_offset)
_c = Color.Lerp(_c,hotspot,x_2_fade);
if(pos.y > y_mid
&& pos.x > PrimaryColliders[0].transform.position.x + x_1_scale + fadeOffset
&& pos.x < PrimaryColliders[0].transform.position.x + x_2_scale
&&pos.z < PrimaryColliders[0].transform.position.z - z_offset)
_c = Color.Lerp(_c,hotspot,y_1_fade);
if(pos.y < y_mid
&& pos.x > PrimaryColliders[0].transform.position.x + x_1_scale + fadeOffset
&& pos.x < PrimaryColliders[0].transform.position.x + x_2_scale
&&pos.z < PrimaryColliders[0].transform.position.z - z_offset)
_c = Color.Lerp(_c,hotspot,y_2_fade);

BrightnessAdditions is a little variable I’ll address in a minute. When I wrote this originally, it was set to 0.7f, which pushed the color channel up to a hot white when triggered.

The rest of the code sets some bounding boxes with fade amounts on how much “color lerping” is happening on a per-pixel basis. The whole thing makes for an imperfect circle. Noticeable now, but less-so when we are only implementing it selectively.

So at this point, I got some more Sterling help, and he introduced me to Microsoft Linq.

First thing is that in the header, we need to add:

using System.Linq;
using System;

and in in the initializing statements we’re throwing in an enum:

public enum                          TouchingState {not,close,closer,touch};
public TouchingState           touchingState;|
public float                            brightnessAdditions;

Now for the Linker stuff!

if(interactiveObjects.Any(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position) <= distMax/distMax)&&interactiveObjects!=null)
                touchingState=TouchingState.touch;
if(interactiveObjects.Any(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position) <= distMax-(distMax/2f))
                && !interactiveObjects.All(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position)<=distMax/distMax)&&interactiveObjects!=null)
                touchingState=TouchingState.closer;
if(interactiveObjects.Any(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position)<=distMax)
                && !interactiveObjects.All(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position)<=distMax-(distMax/2f))&&interactiveObjects!=null)
                touchingState=TouchingState.close;
if(interactiveObjects.All(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position) > distMax-(distMax/2f))&&interactiveObjects!=null)
                touchingState=TouchingState.not;

Now, here are my initial notes on how Linq works:

var addFunc = (a,b) => a+b;
var negateFunc = a => -a;
addFunc(5,negateFunc(7));*/

So you start by declaring your variable/name, and setting it to your equation. In this case, => is Linq shorthand for ‘take these variables and throw them into this context’. In this example, Sterling shows how you can put one Linq equation into another.

Now, the real juicy part of Linq is the list sorting methods.

if(interactiveObjects.Any(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position) <= distMax-(distMax/2f))
&& !interactiveObjects.All(g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position)<=distMax/distMax)&&interactiveObjects!=null)
touchingState=TouchingState.closer;

In this example, we’re sorting through all the objects in “interactiveObjects”, and checking if any of them are less than half of the variable “DistanceMax”, and checking that !all are less than 1f (distMax/distMax). Anything that falls into that category triggers the enum to “close”. The reason I had these flags just changing an enum (as opposed to setting a per item trigger) is that the rendering on the hand can only represent four states, and that’s what I’m building this enum / Linq system to represent.

In our per-vertex loop, above the hotspot declaration we’re going to put:

if(touchingState==TouchingState.not)
brightnessAdditions = 0f;
if(touchingState==TouchingState.close)
brightnessAdditions = 0.28f;
if(touchingState==TouchingState.closer)
brightnessAdditions = 0.55f;
if(touchingState==TouchingState.touch)
brightnessAdditions = 0.8f;

The result being that as we near any of the items in the “interactive objects” list, it will change the amount of tweaking happening in our pixel rendering. The result is that we have fading both on the intensity of the palm, and also fading according to how close we are to our objects.

Finally, I wanted to brighten the objects themselves, so that you could tell which one you were touching. The more information the better. I’m almost thinking I need some audio feedback.

Originally, I was thinking of figuring out through my script which object was closest, since I was already running the numbers with Vector3.Distance. Sterling became irritated that Unity C# doesn’t include ArgMin, so he looked up a workaround. In a separate script called Utils, Sterling copied:

using UnityEngine;
using System.Collections.Generic;
using System;
public static class Utils
{
//a function normally found in other languages which gives you the object with the lowest converted number in a list
public static TSrc ArgMin<TSrc, TArg>(this IEnumerable<TSrc> ie, Converter<TSrc, TArg> fn)
where TArg : IComparable<TArg>
{
IEnumerator<TSrc> e = ie.GetEnumerator();
if (!e.MoveNext())
   throw new InvalidOperationException("Sequence has no elements.");
                TSrc t_try, t = e.Current;
if (!e.MoveNext())
   return t;
                TArg v, min_val = fn(t);
do
{
if ((v = fn(t_try = e.Current)).CompareTo(min_val) < 0)
{
   t = t_try;
   min_val = v;
}
}
while (e.MoveNext());
return t;
}
}

He explained to me the history of Unity in regards to static classes. TLDR: in creating a static class in this manner, the developer is able to make methods that can be accessed from other scripts.

Back in my MeshPointCloud.cs file under the Update loop…

var minObj = interactiveObjects.ArgMin (g => Vector3.Distance(g.transform.position,PrimaryColliders[1].transform.position));

So in this case, we’re creating a variable called minObj, sorting through all members of the interactiveObjects list, and running the ArgMin method (which simulates a C++ function which converts all members to their representative numbers, chooses the smallest number, and then returns the name of said thing).

I barely ended up using this tool, because brightening the materials in the way I was originally thinking did not look pretty. I wanted to include the process in the blog, however, because the lesson learned was very valuable.

I ended up making a list of lights which corresponded to the interactiveObjects list. I then used the ArgMin method to sort through and see if any of the members matched the minObj name, and then turn on the brother light. It rather feels like using a sledgehammer on a nail… but as the sledgehammer was already built… meh?

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

for(int i=0;i<interactiveObjects.Length;i++)
{
if(minObj.name==interactiveObjects[i].name && touchingState!=TouchingState.not)
touchLights[i].enabled = true;
else
touchLights[i].enabled = false;
}

Here is the result:

title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

This is what I eventually came up with. We’ll see how it fares through the council of critique.

Annie’s Substack

Discussion about this post