ViewFinder拍照重建

复刻viewfinder核心玩法————拍照并重建(2)

Github源码传送门————>CaptureRebuild_ViewFinder

Bilibili教程传送门————>小祥带你实现Viewfinder照片实体化功能(网格裁剪+JobSystem)

整体思路

我们捕获照片的时候有说到过，相机渲染在屏幕上的内容，就是相机视锥体内的物体投射到近裁剪平面的结果，那么逆向思维想一想我们要重建照片，实际上就是复原拍照时相机视锥体内的所有物体。所以我们在捕获照片时，记录当时相机视锥体内物体，如果物体有超出视锥体的部分，进行网格裁剪只保留视锥体内的网格即可。这样当我们拿出照片Rebuild的时候，将切割后的物体放置在相机视锥体内，就实现了照片实体化的效果。所以本期的重点内容就是网格裁剪以及使用JobSystem进行计算加速。

1.捕获视椎体内物体

private void CaptureObjectsInFrustum()
{
    Plane[] frustumPlanes = GeometryUtility.CalculateFrustumPlanes(viewCamera);

    IEnumerable capturableObjects =
        FindObjectsOfType<Capturable>().Where(a =>
        {
            return a.GetComponent<Capturable>().isCapturable &&
            a.TryGetComponent(out Renderer renderer) &&
            renderer != null &&
            GeometryUtility.TestPlanesAABB(frustumPlanes, renderer.bounds);
        });

    foreach (Capturable capturableObject in capturableObjects)
    {
        Mesh mesh = temp.GetComponent<MeshFilter>().mesh;
        foreach(Plane plane in frustumPlanes)
        {
            CutMesh(mesh, plane, capturableObject.transform);
        }
    }
}

• GeometryUtility.CalculateFrustumPlanes方法 得相机视椎体对应的六个平面(Plane)
• 寻找场景内所有挂载了Capturable组件的物体，并使用GeometryUtility.TestPlanesAABB判断当前物体是否在视椎体(传入的平面组)内，完全不在视椎体内则返回false，全部或部分在视椎体内返回true
• 筛选出的在视椎体内的物体从MeshFilter获取对应网格，传入网格和视椎体平面进行网格切割

2. 初始化网格数据

List<Vector3> meshVertices = new();
mesh.GetVertices(meshVertices);
NativeArray<float3> jobMeshVertices = CommonTools.ToNativeFloat3Array(meshVertices, Allocator.TempJob);

List<Vector3> meshNormals = new();
mesh.GetNormals(meshNormals);

int validUVCount = 0;
List<List<Vector2>> meshUVs = new();
for (VertexAttribute attr = VertexAttribute.TexCoord0; attr <= VertexAttribute.TexCoord7; attr++)
{
    if (mesh.HasVertexAttribute(attr)) validUVCount++;
}
for (int i = 0; i < validUVCount; i++)
{
    meshUVs.Add(new List<Vector2>());
    mesh.GetUVs(i, meshUVs[i]);
}

int subMeshCount = mesh.subMeshCount;
List<List<int>> meshTriangles = new();
for (int i = 0; i < subMeshCount; i++)
{
    meshTriangles.Add(new List<int>());
    mesh.GetTriangles(meshTriangles[i], i);
}

• 遍历TexCoord0-7，使用 mesh.HasVertexAttribute 方法判断UV通道是否有效，计算合法UV通道数量

3.预留新网格数据结构

List<Vector3> newVertices = new List<Vector3>(meshVertices.Count);
List<Vector3> newNormals = new List<Vector3>(meshNormals.Count);
List<List<Vector2>> newUVs = new List<List<Vector2>>(meshUVs.Count);
List<List<int>> newTriangles = new List<List<int>>(meshTriangles.Count);
List<(Vector3, Vector3)> newEdgePoints = new List<(Vector3, Vector3)>();
foreach (List<Vector2> uvs in meshUVs)
{
    newUVs.Add(new List<Vector2>(uvs.Count * 2));
}
foreach (List<int> triangles in meshTriangles)
{
    newTriangles.Add(new List<int>(triangles.Count * 2));
}

• 提前设置容量，避免或减少扩容次数以提高性能
• newEdgePoints：网格切割之后会产生新的顶点及新边，这个数组用来存所有新产生的边的顶点坐标对

4.本地空间转世界空间

Matrix4x4 localToWorldMatrix = objectTransform.localToWorldMatrix;
VertexTransformJob vertexLocalToWorldJob = new VertexTransformJob
{
    transformMatrix = localToWorldMatrix,
    meshVertices = jobMeshVertices
};
JobHandle vertexLocalToWorldJobHandle = vertexLocalToWorldJob.Schedule(meshVertices.Count, 64);
vertexLocalToWorldJobHandle.Complete();
meshVertices = CommonTools.ToVector3List(jobMeshVertices);

• JobSystem的应用场景：
  • 需要处理大量重复数据(比如这里所有顶点坐标从本地空间转为世界空间)
  • 数据之间不相互依赖，不需要线程同步(顶点坐标转换过程中没有数据相互依赖)
  • 参与运算的数据结构尽量简单且可序列化(传入的参数是简单的float3数组)

[BurstCompile]
public struct VertexTransformJob : IJobParallelFor
{
    [ReadOnly] public float4x4 transformMatrix;
    public NativeArray<float3> meshVertices;

    public void Execute(int index)
    {
        float4 v = new float4(meshVertices[index], 1f);
        meshVertices[index] = math.mul(transformMatrix, v).xyz;
    }
}

• transformMatrix：传入的转换矩阵，做其他转换(世界转本地)传入对应的变换矩阵即可
• 尽量使用Unity.Mathematics这个专门为JobSystem优化的轻量高性能数学库
  • Vector3->float3 Matrix4x4->float4x4 等做好类型转换
  • 使用 math.mul math.dot 等向量化运算方法

5.判断顶点在平面内外侧情况

BurstPlane burstPlane = new BurstPlane(plane.normal, plane.distance);
NativeArray<bool> jobSideResults = new NativeArray<bool>(meshVertices.Count, Allocator.TempJob);
CheckPlaneSideJob checkPlaneSideJob = new CheckPlaneSideJob
{
    burstPlane = burstPlane,
    meshVertices = jobMeshVertices,
    sideResults = jobSideResults
};
JobHandle sideHandle = checkPlaneSideJob.Schedule(meshVertices.Count, 64);
sideHandle.Complete();

List<bool> sideResults = CommonTools.ToBoolList(jobSideResults);
jobMeshVertices.Dispose();
jobSideResults.Dispose();
if (!sideResults.Contains(true) || !sideResults.Contains(false)) return;

• 判断网格所有顶点全部在平面内或平面外，说明不需要进行后续耗时的切割操作，直接返回即可

[BurstCompile]
public struct BurstPlane
{
    public float3 normal;
    public float distance;

    public BurstPlane(float3 normal, float distance)
    {
        this.normal = normal;
        this.distance = distance;
    }

    public bool GetSide(float3 point)
    {
        return math.dot(normal, point) + distance > 0;
    }
}

[BurstCompile]
public struct CheckPlaneSideJob : IJobParallelFor
{
    [ReadOnly] public BurstPlane burstPlane;
    [ReadOnly] public NativeArray<float3> meshVertices;
    [WriteOnly] public NativeArray<bool> sideResults;

    public void Execute(int index)
    {
        sideResults[index] = burstPlane.GetSide(meshVertices[index]);
    }
}

• Unity原生Plane结构体包含引用字段比如Vector3等可以转换为性能更高的数据结构，我们将其Burst化来更好等应用于Job任务中。Burst化的时候只关注需要的属性，比如Plane的法线和距离，同时我们需要方法的也只有GetSide，来判断传入点在平面内测还是外侧。其他组件如果也需要Burst化话也类似，关注Job需要用到的内容即可

6.组织顶点数据

struct VertexKey : IEquatable<VertexKey>
{
    public Vector3 vertex;
    public Vector3 normal;
    public Vector2[] uvs;

    public VertexKey(Vector3 vertex, Vector3 normal, List<Vector2> uvs)
    {
        this.vertex = vertex;
        this.normal = normal;
        this.uvs = uvs.ToArray();
    }

    public bool Equals(VertexKey other)
    {
        if (!vertex.Equals(other.vertex) || !normal.Equals(other.normal)) return false;
        if (uvs.Length != other.uvs.Length) return false;
        for (int i = 0; i < uvs.Length; i++)
        {
            if (!uvs[i].Equals(other.uvs[i])) return false;
        }
        return true;
    }

    public override bool Equals(object obj) => obj is VertexKey other && Equals(other);

    public override int GetHashCode()
    {
        int hash = vertex.GetHashCode();
        hash = (hash * 397) ^ normal.GetHashCode();
        foreach (var uv in uvs)
        {
            hash = (hash * 397) ^ uv.GetHashCode();
        }
        return hash;
    }
}

• 将顶点信息(vertex,normal.uvs)组合成一个键，方便网格切割后顶点去重
• 重写 Equals(object obj)，首先判断 obj 是否为 VertexKey 类型，否的话返回false，是的话转为 other 传入 Equals(VertexKey other) 方法进行比较

7.网格切割

for (int i = 0; i < subMeshCount; i++)
{
    for (int j = 0; j < meshTriangles[i].Count; j += 3)
    {
        triangleVertices = new List<int> 
        { 
            meshTriangles[i][j], 
            meshTriangles[i][j + 1], 
            meshTriangles[i][j + 2] 
        };
        verticesInside = triangleVertices.Where(x => sideResults[x]).ToArray();
        verticesOutside = triangleVertices.Where(x => !sideResults[x]).ToArray();
        switch (verticesInside.Count())
        {
            case 1:
                if (triangleVertices.IndexOf(verticesInside[0]) == 1)
                    Check1In2Out(verticesInside[0], verticesOutside[1], verticesOutside[0], i);
                else
                    Check1In2Out(verticesInside[0], verticesOutside[0], verticesOutside[1], i);
                break;
            case 2:
                if (triangleVertices.IndexOf(verticesOutside[0]) == 1)
                    Check2In1Out(verticesInside[0], verticesInside[1], verticesOutside[0], i);
                else
                    Check2In1Out(verticesInside[1], verticesInside[0], verticesOutside[0], i);
                break;
            case 3:
                CheckAllIn(triangleVertices[0], triangleVertices[1], triangleVertices[2], i);
                break;
        }
    }
}

• 以三角形面片为基本单位进行网格切割，有三种情况(三角形的1/2/3个顶点平面内侧)分别进行处理
• 传顶点参数之前要调整顶点顺序，确保新三角形的顶点顺序正确，具体看后面的示意图

void Check1In2Out(int index1In, int index2Out, int index3Out, int submesh)
{
    int newVertex1Index = CutEdge(index1In, index2Out);
    int newVertex2Index = CutEdge(index1In, index3Out);
    newEdgePoints.Add((newVertices[newVertex1Index], newVertices[newVertex2Index]));

    int oldVert1Index = AddOldVertex(index1In);
    int[] trianglesToAdd = new int[] { oldVert1Index, newVertex1Index, newVertex2Index };

    newTriangles[submesh].AddRange(trianglesToAdd);
}

void Check2In1Out(int index1In, int index2In, int index3Out, int submesh)
{
    int newVertex1Index = CutEdge(index1In, index3Out);
    int newVertex2Index = CutEdge(index2In, index3Out);
    newEdgePoints.Add((newVertices[newVertex1Index], newVertices[newVertex2Index]));

    int oldVertex1Index = AddOldVertex(index1In);
    int oldVertex2Index = AddOldVertex(index2In);
    int[] trianglesToAdd = new int[] 
    {
        oldVertex1Index, newVertex1Index, oldVertex2Index,
        newVertex1Index, newVertex2Index, oldVertex2Index
    };

    newTriangles[submesh].AddRange(trianglesToAdd);
}

void CheckAllIn(int index1In, int index2In, int index3In, int submesh)
{
    int oldVertex1Index = AddOldVertex(index1In);
    int oldVertex2Index = AddOldVertex(index2In);
    int oldVertex3Index = AddOldVertex(index3In);
    int[] trianglesToAdd = new int[] { oldVertex1Index, oldVertex2Index, oldVertex3Index };

    newTriangles[submesh].AddRange(trianglesToAdd);
}

• Unity默认以顺时针顶点顺序来定义三角形正面，切割之后顶点会有不同情况，如下图给出不同情况：
  

int CutEdge(int vertexInIndex, int vertexOutIndex)
{
    Vector3 vertexIn = meshVertices[vertexInIndex];
    Vector3 vertexOut = meshVertices[vertexOutIndex];
    Vector3 direction = vertexOut - vertexIn;

    Ray ray = new Ray(vertexIn, direction);
    plane.Raycast(ray, out float distance);
    float ratioIntersection = distance / direction.magnitude;

    List<Vector2> newUVs = new();
    for (int i = 0; i < validUVCount; i++)
    {
        Vector2 uvIn = meshUVs[i][vertexInIndex];
        Vector2 uvOut = meshUVs[i][vertexOutIndex];
        newUVs.Add(Vector2.Lerp(uvIn, uvOut, ratioIntersection));
    }

    Vector3 normalIn = meshNormals[vertexInIndex];
    Vector3 normalOut = meshNormals[vertexOutIndex];
    Vector3 newNormal = Vector3.Lerp(normalIn, normalOut, ratioIntersection).normalized;

    Vector3 newVertex = ray.GetPoint(distance);
    int newVertexIndex = AddNewVertex(newVertex, newNormal, newUVs);

    return newVertexIndex;
}

• 通过射线检测判断当前边与平面的交点(newVertex)以及交点比率(ratioIntersection)
• 根据交点比率插值计算新的uv坐标以及法线向量

8.查找循环边生成多边形

Vector3 polygonNormal = plane.flipped.normal;
List<List<Vector3>> newPolygons = new();
List<Vector2> polygonUVs = Enumerable.Repeat(Vector2.zero, validUVCount).ToList();

int edgeIndex = -1;
while (newEdgePoints.Count != 0)
{
    if (edgeIndex == -1)
    {
        newPolygons.Add(new List<Vector3>());
        edgeIndex = 0;
    }
    var (p1, p2) = newEdgePoints[edgeIndex];
    newEdgePoints.RemoveAt(edgeIndex);
    newPolygons.Last().Add(p1);
    edgeIndex = newEdgePoints.FindIndex(x => x.Item1 == p2);
}

foreach (List<Vector3> polygon in newPolygons)
{
    for (int i = polygon.Count - 2; i >= 1 ; i--)
    {
        if (CommonTools.CheckCollinear(polygon[i - 1], polygon[i], polygon[i + 1]))
        {
            polygon.RemoveAt(i);
        }
    }

    int[] indices = polygon.Select(vertex => AddNewVertex(vertex, polygonNormal, polygonUVs)).ToArray();
    int pivotIndex = indices[0];
    for (int i = 1; i < polygon.Count - 1; i++)
    {
        newTriangles.Last().AddRange(new int[] { pivotIndex, indices[i + 1], indices[i] });
    }
}

• 前面切割之后产生了新的边，需要查找循环边将这些边组织起来形成新的多边形
• newEdgePoints内是所有新边对应的端点，遍历所有边将所有能够首尾相接的边添加进一个列表
• 排除共线的点后以第一个顶点为锚点将多边形转化为三角形，依旧需要注意三角形顶点顺序，如下图：
  

9.回填网格数据

mesh.Clear();
mesh.SetVertices(newVertices);
mesh.SetNormals(newNormals);
for (int i = 0; i < validUVCount; i++)
{
    mesh.SetUVs(i, newUVs[i]);
}
mesh.subMeshCount = subMeshCount;
for (int i = 0; i < subMeshCount; i++)
{
    mesh.SetTriangles(newTriangles[i], i);
}

• mesh.Clear() 回填数据前需要清空mesh数据，否则顶点数和三角形索引数无法匹配会导致异常
• 将处理好的新顶点，法线，UV，三角形索引回填到mesh，一次切割就完成了

优化方向

• 光照和阴影目前没有处理，从照片实体化为模型后产生阴影会有异样感，可以将光照提前烘焙到LightMap，关闭生成物体的阴影，这样就可以让实体化的模型和照片完全保持一致
• 网格裁剪的过程是最耗费性能的，但是因为要处理的数据比较杂，同时需要使用字典进行去重操作，没有加入JobSystem进行计算加速，实际上也可以通过拆分任务，写一个线程安全的顶点去重方案来进一步优化这里的性能

完结撒花~