Performance Profiling & Debugging

01 · Problem & scope

This runbook covers diagnosis on the Windows client and dedicated server, including CPU work, rendering, RAM/VRAM pressure, asset loading, animation, AI, physics, UMG/CommonUI, replication and persistence stalls. It describes a proposed investigation procedure; it does not approve optimization changes or certify current performance.

Source of truth: this HTML owns the profiling procedure. Runtime settings remain in source/configuration and the running build. Final hardware, resolution, frame-rate and server targets remain in ART-DIRECTION → performance budget, tracked by TBD-BFF08134. Example measurement windows below are procedural suggestions, never approved performance budgets.

02 · Current evidence & verification limits

03 · Investigation architecture

1. Reproduce
2. Identify process
3. Triage
4. Capture
5. Follow critical path
6. Change one cause
7. Repeat & report

Game Thread, Render Thread, RHI submission, workers and GPU queues overlap and synchronize. Frame time is an end-to-end interval; adding all their durations double-counts overlapping work. A long wait is a clue about a dependency, not proof that the waiting function is expensive.

Treat these routes as hypotheses. A local server, two clients and the Editor on one machine compete for CPU, GPU, RAM and disk; label that run as a contention scenario.

04 · Establish a comparable baseline

1. Write the symptom before capturing. Record the map, camera/route, action, network role, reproduction frequency and expected behavior. Separate steady-state slowdown, first-use hitch and long-session degradation.
2. Identify the actual binary. Record engine/build configuration, source revision and local modifications. After C++ changes, stale Live Coding DLLs can invalidate the comparison. The owner closes the Editor, builds LambeerEditor Win64 Development and reopens when needed.
3. Choose a build appropriate to the question. PIE is useful for locating causes. Use a packaged Development build for representative diagnostic captures when available. Record if using Editor/Standalone instead. Do not compare Editor frame time with a packaged baseline as though only code changed.
4. Fix the environment. Record CPU/GPU/RAM, driver/OS, power/thermal state, storage, display resolution, render resolution, upscaler/dynamic resolution, quality presets, VSync, frame limit and background applications. Keep the same focused window and viewport size.
5. Fix the workload. Use the same save/seed if supported, position, visible actors, player/AI count, inventory contents, lighting/weather and action sequence. If a system is not implemented, mark that scenario unavailable.
6. Separate cold and warm runs. First load and first shader/PSO use answer a different question from a warmed route. For steady state, wait for loading/compilation to settle; retain cold-run results separately. Do not clear caches as a routine shortcut.
7. Repeat the same window. A suggested initial sample is three repetitions of a 30–60 second steady-state route, plus short targeted hitch captures. Adjust duration to the actual reproduction; report the chosen duration and sample count.
8. Measure collection overhead. Compare a lightweight run and the diagnostic capture with the same workload. Heavy memory, named-event and verbose network tracing can change the result. Disable visual debug overlays for the final comparison after triage.

Caps: query t.MaxFPS and r.VSync and record their current values. For a separately labelled uncapped diagnostic experiment, the owner may use t.MaxFPS 0 and r.VSync 0, then restore the recorded values. Also check game user settings, smoothing and driver limits. Do not silently change the normal play configuration.

Units: use milliseconds per frame. frame budget (ms) = 1000 / target FPS; for illustration, 60 FPS is about 16.67 ms and 30 FPS about 33.33 ms. Neither is an approved Lambeer client target. Server active simulation time and tick interval are separate measurements; an idle wait between ticks is not simulation cost.

05 · First capture: owner manual procedure

Start with a short CPU/GPU capture. These are console commands entered one at a time in the intended running instance, not PowerShell commands.

1. The owner opens Lambeer in UE 5.8, selects the intended map and starts the chosen test mode. For a PIE reproduction, record the client window title, server role, process IDs and whether Run Under One Process is enabled.
2. Focus the intended client window; open its console. Run stat unit and stat unitgraph, reproduce once and record whether Game, Draw, GPU or a periodic spike dominates. Treat the overlay as triage.
3. Run Trace.Status. Check whether a trace is already active, the destination and enabled/available channels. Preserve an existing useful capture before stopping it; do not assume a fresh trace state.
4. If no capture is active, run the sequence below. Replace the example filename/run label for each capture. Close the console before performing the action; keep console-opening time outside the analysis interval.

Trace.File Lambeer_Client1_Baseline_R01.utrace cpu,gpu,frame,bookmark,log
Trace.Status
Trace.Bookmark Baseline_Begin

// Close console; perform the fixed reproduction sequence.
// Reopen console only after the observation window ends.

Trace.Bookmark Baseline_End
Trace.Stop

Lines beginning with // are explanatory notes; do not paste them into the console. Trace.File and Trace.Bookmark syntax was checked in local UE 5.8 Core/Private/ProfilingDebugging/TraceAuxiliary.cpp. Trace.Start is deprecated there; this guide uses Trace.File.

5. Read the log for the actual output path and successful start/stop. For a filename-only trace, the normal project location is Saved/Profiling/. Packaged or remote runs may use another writable Saved directory; the log is authoritative.
6. The owner opens D:/Epic Games/UE_5.8/Engine/Binaries/Win64/UnrealInsights.exe and opens the saved .utrace from its session browser/file-opening control. Wait for analysis to finish.
7. Confirm process/session identity, non-empty frame/CPU tracks, expected duration and bookmarks. Check that the symptom actually occurred. An empty or wrong-process capture is an invalid experiment.
8. Attach the trace, exact start command, capture log, run manifest and a screenshot of the selected time interval to the investigation handoff.

Epic reference: Unreal Insights commands and channels. Verify runtime availability with Trace.Status; documented support is not a live verification.

06 · Capture profiles & command reference

Use the smallest profile that answers the current question. Startup argument fragments below belong on the actual game/server/Editor process command line. They do not launch a build by themselves. Executable locations and launch settings must be verified for the selected build.

Destination choice: file recording is convenient for one process; a trace store is useful for live sessions. Choose one intended route, record it, and check the log. Give server, Client1 and Client2 different filenames. Late-start timing capture cannot reconstruct earlier events or allocations.

No commands match. Reset the search or choose another category.

Sources: Epic Stat Commands; local CsvProfiler.cpp, TraceAuxiliary.cpp and UnrealEngine.cpp. Memory startup requirements: Memory Insights. Plugin/channel support: Trace overview.

07 · CPU: read the critical path

1. Find the interval. Open Timing Insights, locate the reproduction bookmarks, select the affected frame and include several frames before/after it. Compare with an ordinary frame from the same phase.
2. Find the delayed dependency. Inspect Game Thread, Render Thread/RHI and worker lanes together. If a thread is waiting, follow the task, resource or preceding work that must complete. A busy worker is relevant only if it delays the observed outcome.
3. Separate inclusive and exclusive cost. Inclusive time contains child scopes; exclusive time excludes those children. Sort Timers for the selected interval by total cost, then inspect event count, average and maximum. Many small calls can exceed a rare large call.
4. Inspect callers/callees where available. Follow the expensive event down to a scope you can associate with source. Do not blame a broad engine tick scope merely because it encloses everything else.
5. Check workload scaling. Relate event count to actual players, AI, inventory entries or visible widgets. Record count and cost together; a faster run with fewer actors is not a comparable improvement.
6. Form a falsifiable explanation. Example hypothesis: inventory notifications cause repeated full presentation rebuilds during one action. Verify the event count and call chain before changing update logic.

Blueprint, animation, AI and physics drilldown

• Blueprint: use named events for a focused capture; locate repeated event/tick/function work. Count timer/delegate invocations, casts, searches and collection traversals. Avoid converting an entire Blueprint to C++ without measured evidence.
• Animation / MetaHuman: correlate skeletal-mesh count, LOD, animation updates, evaluation workers, bone/skin work and attachment complexity. Observe both local and remote characters. A camera-distance change also changes rendering, so annotate it.
• AI / StateTree: inspect active agents, perception, navigation requests, task transitions and query frequency where implemented. Change one frequency or workload factor per experiment; verify behavior afterwards.
• Physics / collision: inspect query count, overlap/event bursts, active bodies and component updates. Measure a reproducible interaction instead of disabling collision globally and treating the result as a fix.

Analysis procedure proposed for Lambeer. Tool overview: Unreal Insights. Prefer the exact timer names recorded by the build; this guide does not promise every scope will exist.

08 · GPU, rendering and VRAM

1. Confirm the slowdown is visible in GPU timing and that the resolution, upscaler, dynamic resolution and frame cap are known. Read CPU submission tracks too; a render-submission bottleneck is different from expensive GPU execution.
2. Capture several representative frames in Insights. Use ProfileGPU for a selected frame, then read its log hierarchy. The installed RHI/Private/GPUProfiler.cpp supports log output even when the UI is unavailable or disabled.
3. Inspect the expensive recorded passes: candidates include base/shadow work, Lumen, hair/skin, translucency, post-processing and UI composition. Pass labels depend on the actual renderer. Parent/child timings and parallel GPU queues can overlap; do not add them into a false frame total.
4. Run a controlled resolution test: temporarily lower render resolution using the supported settings while keeping camera/content fixed, then restore it. A strong GPU-time response supports pixel-related cost; little change calls for investigation of geometry, bandwidth, fixed costs or CPU limits. This is evidence, not a complete diagnosis.
5. Inspect one suspected subsystem at a time using a reversible diagnostic setting or a duplicate test scenario. Record the visual difference and repeat with the normal configuration before accepting a change.
6. Check VRAM pressure alongside GPU time: resource residency, streaming changes and host memory movement can make a route hitch. A texture-pool warning alone does not describe all GPU allocations; blindly increasing the pool can worsen overall pressure.

RenderDoc, PIX or a vendor GPU profiler may be useful when an exact draw/resource question remains after Insights. Availability and workflow are unverified here; select the tool for the actual platform/RHI. Capture injection changes timing, so validate improvements again without it.

09 · Memory growth, allocation churn and garbage collection

Use a fresh process with memory tracing enabled at startup. A late timing trace cannot reconstruct the allocation history required by Memory Insights. Keep matching symbols and enough disk space; short reproductions are easier to analyze. The metadata,assetmetadata channels support asset/class attribution where available. Epic setup and queries.

1. Choose repeatable phases: A = warmed idle; B = after opening/using the feature; C = after closing it and returning to equivalent idle. Repeat the full cycle; note GC and asynchronous cleanup timing.
2. Inspect tracked bytes and live allocation counts over time. Query allocations alive at the chosen points; group by callstack, tag and available asset/class metadata.
3. Separate temporary churn from retention. High allocate/free volume can waste CPU even when total memory returns to baseline. A rising plateau needs investigation of still-live allocations and owners.
4. Compare equivalent lifecycle points. Caches may retain useful resources; a rise after first use is not automatically a leak. Persistent growth after repeated equivalent cycles is stronger evidence.
5. Capture memreport -full separately at comparable points and inspect object/resource counts. Correlate suspected retained widgets, delegates, async handles or assets with source ownership.
6. For periodic hitches, correlate GC scopes with object churn and reference retention. A forced GC is a separate diagnostic experiment and must be labelled; it does not represent normal frame pacing.

Three distinct measurements: process working set/committed memory from the OS, engine-tracked allocations from Insights, and GPU resource memory from RHI/platform tools. Their scopes differ; do not expect equal totals. Missing symbols or asset tags reduce attribution and must be recorded as an analysis limit.

10 · Loading, streaming, shader/PSO and storage hitches

Capture before travel or first interaction with loadtime,file plus CPU/frame/bookmarks. In Timing/Asset Loading views available in the build, align the hitch with file activity, package serialization, object creation, post-load work, resource initialization and thread waits.

1. Repeat the same route once cold and again warm. Record which cache state is known; do not call every first run “cold” without describing what was restarted.
2. If disk work overlaps a hitch, determine whether the Game Thread actually waits for it. Concurrent I/O is not proof that I/O caused the stall.
3. If a UI action first loads textures/classes, trace from input through load completion and widget creation. A warm open can hide the original problem.
4. If a new material/effect or scene first appears, check shader/PSO creation events and corresponding log messages in that build. Distinguish Editor shader compilation from a packaged first-use pipeline hitch.
5. If traversing the world stalls, correlate streaming transitions, registration/component work, GC and GPU residency. Record travel speed, direction, visibility and storage device.
6. If saving stalls the server, measure serialization, queue handoff, database work and completion separately where implemented. Link to the canonical persistence design; do not assume its full runtime implementation is present.

Candidate remedies need evidence: preloading, async loading, batching, dependency reduction or PSO preparation may help different causes. Moving work to a worker is insufficient if the Game Thread immediately waits for it. Preserve failure handling and ownership when proposing a fix.

11 · Lambeer UI, inventory and HUD drilldown

Use TECH-HUD-UI-IMPLEMENTATION → Performance & Failure Handling for the existing design rules. This page defines how to collect supporting evidence, without assigning an unapproved UI millisecond budget.

If the Slate channel and Slate Insights analysis are available, inspect invalidation, layout, paint and widget activity around the same bookmarks. First verify plugin/channel support; an empty Slate view is not evidence of a cheap UI. Compare idle UI with the exact action and dataset. Investigate unnecessary full-tree updates, repeated construction, per-frame bindings, timers or unremoved listeners only when the trace/source supports that explanation.

Reproduction commands: before diagnosing a HUD or GAS symptom, verify the gameplay command actually ran in that client. Command not recognized: Lambeer.Damage means the cheat path did not execute. Confirm LogLambeer output; UI_G3NotifyBlood exercises the overlay path while Lambeer.Damage exercises GAS plus overlay. Do not classify an absent trigger as a rendering/performance failure. The owner performs all such PIE actions.

12 · Dedicated Server + Client1 + Client2

1. Write a process roster: host, PID, executable/build, role, client/window and connection. Use separate capture names, for example Server_R01.utrace, Client1_R01.utrace and Client2_R01.utrace.
2. The owner applies network tracing arguments from section 06 to each actual process before launch. In PIE launch settings, check which fields apply to the server and to separate clients; argument propagation is not assumed. Confirm each process log and trace session.
3. Capture the same action with a shared run ID. Bookmarks in one process do not automatically appear in another. Match gameplay sequence IDs/log events when possible; timestamps from different processes are not automatically a synchronized latency measurement.
4. In Networking Insights select the intended Game Instance, Connection and incoming/outgoing direction. Inspect the packet interval, then the replicated object/property/RPC hierarchy and event size/count. Keep units explicit: packet content may be shown in bits, other totals in bytes.
5. Compare server CPU simulation/serialization with each client's rendering and receipt/update work. Identify whether all clients pause together, one client alone stalls, or visual corrections occur while local frames stay smooth.
6. Repeat with controlled populations where the build supports them. The current two-client setup is a debugging scenario, not proof of the intended full-server capacity. Preserve authority, relevancy and correctness in any proposed optimization.

Networking panels and launch syntax: Epic Networking Insights. Reported packet sizes are before compression, so they are not necessarily actual wire bandwidth. A local loopback run does not represent internet latency/loss.

13 · Add instrumentation only where attribution is missing

The following is an illustrative C++ pattern, not added to Lambeer source. Place one scope around the actual operation and bookmarks at sparse scenario boundaries. Use stable names and record role/world identifiers where necessary to distinguish shared-process PIE worlds.

#include "ProfilingDebugging/CpuProfilerTrace.h"
#include "ProfilingDebugging/MiscTrace.h"

// Inside a measured operation; name is illustrative.
TRACE_CPUPROFILER_EVENT_SCOPE(Lambeer_InventoryPresentation);
// Existing operation executes within this scope.

// At a rare scenario transition, not every Tick:
TRACE_BOOKMARK(TEXT("Lambeer_Profile_InventoryOpened"));

• Measure a meaningful operation, then subdivide only if the first scope remains too broad. Keep the scope lifetime equal to the work being measured.
• Use counters for workload size, rebuild frequency, queue depth or bytes when helpful. A duration without workload/context often cannot explain scaling.
• A synchronous scope around dispatch measures dispatch, not asynchronous completion. Instrument the worker and completion dependency separately.
• Avoid dynamically constructed scope labels, unbounded per-entity names and per-frame log spam. Keep before/after instrumentation equivalent.
• Compile the intended target, confirm the running binary includes the change, then capture with the required channels. Source edits alone do not verify emitted events.

Macro spelling checked against installed UE 5.8 Engine/Source/Runtime/Core/Public/ProfilingDebugging/CpuProfilerTrace.h; bookmarks use MiscTrace.h. Runtime trace compile switches/build support still need checking for the selected target.

14 · Capture manifest & evidence handoff

Copy this read-only template into an investigation record maintained by editing HTML source. Raw traces/reports are evidence, not a second project-specification source. The final cause, accepted decision and regression status belong in the relevant canonical HTML document.

Investigation ID / symptom:
Status: Not measured / Investigating / Improved / Regressed / Inconclusive
Build: engine, configuration, executable, source revision + local changes
Host: OS, CPU, GPU/driver, RAM, storage, power/thermal state
Role/topology: server/client, PID, host, PIE one-process setting
Runtime settings: RHI, resolution, render scale/upscaler, quality, caps/VSync
Scenario: map, start position/camera, route/actions, save/seed if available
Workload: players, AI, visible characters, widgets/items/effects as relevant
Cache state: cold/warm; startup/compilation/loading exclusions
Capture: exact arguments, channels, destination, start/stop log, symbols
Sample: run IDs, duration, frame count, bookmarks and selected interval
Baseline: metric names/units, per-run values, median/p95/p99/max as applicable
Suspected cause: scope/pass/object/packet; evidence and competing explanations
Change under test: exactly one independent change; visual/behavior tradeoff
After: same metric/window/workload; absolute delta; percentage where valid
Regression checks: client roles, correctness, memory, visuals, cold/warm behavior
Evidence paths: .utrace, logs, CSV/memreport, screenshots, matching build symbols
Limits: missing tracks, overhead, sample variance, unavailable scenarios
Owner-run result: reporter/date and exact reported outcome
Target decision: TBD-BFF08134 unless/ until canonical targets are approved
Next action / decision owner:

Naming suggestion: Lambeer_<scenario>_<role>_<build-label>_<phase>_R<run>. This is a proposed evidence naming convention; replace all placeholders. Keep captures out of ordinary source commits unless a storage policy explicitly calls for them. Share the actual file destination from the log and preserve enough evidence to reproduce the conclusion.

Retention/ownership: storage location, retention period and performance review owner are Unassigned / not yet specified. Agree these before instituting a recurring benchmark gate; this runbook does not create one.

15 · Before/after validation & acceptance

Compare distributions and representative traces from equivalent runs. Report frame-time median plus p95/p99 where sample size supports them, the maximum, and a defined hitch count. A short capture cannot establish a stable tail percentile. Keep startup frames, menu time and intentional pauses separate; document all exclusions.

Metric conventions: specify the percentile method and sampled frame population. A p99 frame time is the time at/below which 99% of sampled frames fall; it is not automatically the same definition as “1% low FPS.” Use 1000 / mean(frame_ms) for average FPS over the sampled frames, not the arithmetic mean of per-frame FPS. Report each run before aggregating to expose variance.

delta_ms = after_ms - before_ms; negative means faster for timing metrics. improvement_percent = (before_ms - after_ms) / before_ms × 100 only when the baseline is positive. Improvement within normal run-to-run noise is inconclusive.

Suggested scenario matrix

• Warmed idle; fixed traversal; first/repeated inventory and map opening; one inventory action; HUD effect sequence.
• Server + both clients during the same interaction, join-in-progress and respawn where available; separate role results.
• Dense visible characters/AI and long-session repeated interaction only with an implemented reproducible fixture; record actual counts.
• Cold startup/loading versus warmed play; changed UI closed again; memory after equivalent lifecycle points.

Completion wording: “measured improvement in scenario X” is allowed with evidence. “Performance target passed” requires approved targets and a matching platform/workload. An agent may record PIE results only when explicitly reported by the user.

16 · Failure handling & common misdiagnoses

Manual boundary: if Editor inspection is needed and Unreal MCP is disconnected, the owner opens UE 5.8, enables/connects the project's MCP plugin and confirms the endpoint http://127.0.0.1:8000/mcp. Agents discover available toolsets before inspecting. If a required profiling UI action is not exposed, the owner performs that action and supplies the resulting capture or log. This document does not assert current MCP connectivity.

17 · Decisions, open requirements & status

Existing decision TBD-BFF08134 remains open. Resolve final hardware/resolution/frame-rate and server targets in ART-DIRECTION, then synchronize DEC-BACKLOG. This guide does not duplicate or replace that decision.

Before declaring a shipping pass, agree the target workload, quality mode, memory/VRAM limits, frame-time/tail/hitch criteria, server workload and supported host specification as part of the target review. Capture owner, benchmark fixture, evidence retention and regression tolerance are not specified. Their absence does not prevent an explicitly labelled diagnostic comparison.

Revision — 2026-09-12: added a detailed profiling/debugging procedure at the user's request; checked against canonical HTML, selected local source/configuration and Epic references. Status is draft pending practical owner use. No new performance targets, gameplay decisions or test passes are approved by this documentation update.

18 · Relationships & source references

• ART-DIRECTION — canonical performance target decision.
• DEC-BACKLOG — preserves TBD-BFF08134.
• TECH-HUD-UI-IMPLEMENTATION — UI design rules and implementation/owner verification status.
• TECH-NETWORK-ARCH — authority, replication and persistence design.
• TECH-STACK-ENGINE — project systems; use verified UE 5.8 source paths.
• PROJ-OVERVIEW — deployment scope.

External references checked 2026-09-12

Reference links require internet only when opened. The document itself has no external runtime assets. Epic documentation is living documentation; the installed UE 5.8 source checks below clarify version-sensitive command behavior.

• Epic: Unreal Insights — tool suite and capture/analysis overview.
• Epic: Unreal Insights Reference — channels, trace destinations and console controls.
• Epic: Stat Commands — stat groups for first-pass inspection.
• Epic: Memory Insights — startup requirements, allocation tracking and attribution.
• Epic: Networking Insights — arguments, connection/packet views and size interpretation.
• Epic: Trace — channel and plugin context.

Local source checks

Engine-relative paths below are under D:/Epic Games/UE_5.8/Engine/. They were read from disk; no engine or gameplay code was changed.

• Source/Runtime/Core/Private/ProfilingDebugging/TraceAuxiliary.cpp — Trace.File, Trace.Status, Trace.Bookmark and deprecated Trace.Start.
• Source/Runtime/Core/Private/ProfilingDebugging/CsvProfiler.cpp — CsvProfile START/STOP parsing.
• Source/Runtime/Core/Public/ProfilingDebugging/CpuProfilerTrace.h — scope macro syntax.
• Source/Runtime/RHI/Private/GPUProfiler.cpp — GPU hierarchy/log output and UI control.
• Source/Runtime/Engine/Private/UnrealEngine.cpp — memory report destination.