PHYS Open-Oven Handoff and Final Conclusion¶
历史单 case 调试记录。 本页中的旧 worktree、只读主仓库约束、环境变量和启动命令已经 失效，不得用于当前实验。当前方法状态看 Physics Baselines，运行方式看 Physics Studio 使用指南，正式数值看 Physics Results。下文只保留用于追溯 2026-04-29 的失败诊断。
Date: 2026-04-29
This document is a self-contained handoff for another agent. Assume no other chat context is available.
Hard Constraints¶
Main repo:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D
Main repo rule:
Treat the main repo as read-only for code and experiments.
Do not git switch / git checkout / edit implementation files under the main repo.
Do not run PHC experiments from the main repo.
This handoff file is stored in main docs only as documentation.
PHC/open-oven worktree:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven
Worktree rule:
All PHC / PHC-X / passive-object / repair experiments must happen in this worktree.
All new code, reports, plots, videos, and temporary configs for this line of work should go here.
Expected worktree branch:
phys/phc-open-oven-exp
If the worktree is missing, recreate it from the main repo without changing the main repo branch:
cd /DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D
git worktree add /DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven phys/phc-open-oven-exp
cd /DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven
Environment¶
Default environment for reconstruction / contact refinement work:
The arthoi4d environment is the default environment to use for the next stage of visual contact refinement and local hand/arm repair.
Use the IsaacGym environment only when reproducing PHC / PHC-X / IsaacGym rollout commands that require IsaacGym bindings:
Known IsaacGym command environment:
PATH=/DATA/intern/hoi4d/miniconda3/envs/isaacgym/bin:${PATH:-}
LD_LIBRARY_PATH=/DATA/intern/hoi4d/miniconda3/envs/isaacgym/lib:${LD_LIBRARY_PATH:-}
PYTHONPATH=/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/submodules/InterMimic/isaacgym/python:${PYTHONPATH:-}
IMAGEIO_FFMPEG_EXE=/usr/bin/ffmpeg
Do not reinstall, downgrade, or upgrade PyTorch in either environment unless explicitly asked.
Current Case¶
Target reconstruction:
result.pt:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D/output/bench_d3dhoi_oven_ours_b009_0001_track3d_sam3d_body_regression/recon/output/result.pt
object asset:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D/data/d3dhoi/processed_cads/oven/101940
experiment/worktree root:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven
Task context:
Object:
articulated oven
Desired physics behavior:
humanoid hand should contact the oven moving link / handle
passive hinge qpos_sim should be moved by contact
object qpos must not be overwritten during passive validation
Observed failure:
PHC tracks the human, contact force appears on the moving link, but hinge qpos_sim remains almost zero.
Final Diagnosis¶
The root cause is visual reconstruction / reference contact pose error.
It is not:
not PHC tracker failure
not coordinate-system conversion failure
not renderer failure
not missing object collider
not hinge DOF indexing failure
not a stuck hinge asset
not simply disabled physics
The current reconstructed reference places the active hand too far from the moving link / handle. Physics cannot recover a 20-40cm reference contact error.
Evidence¶
PHC Tracking Is Working¶
PHC-X / PHC+ tracker-only runs completed successfully:
PHC-X primary SMPL-X:
success_rate: 1.0
mpjpe_g: 36.332 mm
mpjpe_l: 42.707 mm
mpjpe_pa: 28.071 mm
PHC+ keypoint SMPL:
success_rate: 1.0
mpjpe_g: 27.347 mm
After switching visualization back to PHC native smplx_humanoid embodiment, the human no longer appears horizontally rotated or drifting in the rendered videos.
Correct PHC embodiment files already generated in the worktree:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/phc_assets/phc_smplx.xml
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/phc_assets/phc_smplx.yaml
Object Collider and Hinge Are Working¶
The staged oven.urdf has base and moving-link collisions. Torque tests prove link_00_joint can move:
isolated damping=0/friction=0, +1Nm:
qpos_final ~= 1.57080 rad
PHC-X full env diagnostic hinge torque +10Nm:
qpos_sim max ~= 1.57084 rad
So this is not an asset import, collider, DOF indexing, or hinge-lock issue.
Passive Rollout Confirms the Object Is Not Moving¶
Saved passive-object physics rollout:
default physics rollout:
qpos_sim max ~= 2.06e-6 rad
joint-free damping/friction ablation:
qpos_sim max ~= 1.03e-4 rad
moving link contact force max:
default ~= 1519.52 N
joint-free ~= 1638.64 N
Interpretation:
Contact force exists on the moving link,
but the hand contact is geometrically wrong and does not generate useful hinge torque.
Hand-Handle Distance Is Too Large¶
After fixing PHC native visualization, the measured hand-object distances are:
left hand -> moving link:
active mean ~= 31.95 cm
active min ~= 23.04 cm
8cm contact ratio = 0
left hand -> handle75 candidate:
active mean ~= 38.41 cm
left hand -> handle90 candidate:
active mean ~= 46.17 cm
This is far outside the range needed for passive contact. The target for physical rollout should be:
active hand-handle mean distance < 3-5 cm
min distance < 2 cm
penetration < 1 cm
stable contact over manipulation window
Key Existing Artifacts¶
Reports:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/passive_object_tracker_report.json
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/phc_x_pnn_passive_object_default_eval_summary.pkl
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/phc_x_pnn_passive_object_joint_free_eval_summary.pkl
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/phc_x_pnn_passive_object_diag_torque10_eval_summary.pkl
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/phc_x_native_hand_object_distance.json
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/phc_x_native_tracked_human_physics_rollout_default_visualization_report.json
Videos:
PHC tracked human + object replay, correct PHC embodiment:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/videos/phc_x_phc_smplx_tracked_human_object_replay_full_body.mp4
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/videos/phc_x_phc_smplx_tracked_human_object_replay_closeup.mp4
PHC tracked human + saved passive-object physics qpos_sim replay:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/videos/phc_x_phc_smplx_physics_rollout_default_closeup.mp4
Physics rollout state source:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/reports/phc_x_pnn_passive_object_default_eval_summary.pkl
VLM contact sidecar already available; do not rerun VLM for the first iteration:
VLM contact root:
/mnt/cvda_mnt/xuyuan/ArtHOI4Dmine/output/contact_combo_2026-04-26/c10_r10_gpt_5_5_parallel075
coarse tracker contact:
/mnt/cvda_mnt/xuyuan/ArtHOI4Dmine/output/contact_combo_2026-04-26/c10_r10_gpt_5_5_parallel075/exported_contact/tracker_contact_ref.npz
fine contact sequence:
/mnt/cvda_mnt/xuyuan/ArtHOI4Dmine/output/contact_combo_2026-04-26/c10_r10_gpt_5_5_parallel075/contact3d_video2d3d/contacts_3d.json
raw VLM frame verdicts:
/mnt/cvda_mnt/xuyuan/ArtHOI4Dmine/output/contact_combo_2026-04-26/c10_r10_gpt_5_5_parallel075/video2d3d_provider/vlm_raw_responses.json
review video:
/mnt/cvda_mnt/xuyuan/ArtHOI4Dmine/output/contact_combo_2026-04-26/c10_r10_gpt_5_5_parallel075/vis/vlm_quad_panel.mp4
Observed from the sidecar:
contact_human shape: (150, 52)
contact_obj shape: (150, 1)
contact_obj active frames: 24-148
dominant hand: left
early active slots: L_Thumb3, L_Index3, L_Middle3, L_Ring3
Important:
VLM contact onset is around frame 24.
TAPIP3D / object motion onset is around frame 37-41.
Treat these as two different gates:
contact gate: hand may touch / grip before object moves
motion / torque gate: object should only be required to move after visual motion onset
The contacts_3d.json was generated from an older xuyuan result path. Use its frame-wise semantic labels and finger-slot labels, but recompute object-side contact points / moving-link candidates from the current result.pt before optimizing.
Fast-Iteration Experimental Route¶
The current goal is not to redesign PHC. The goal is to repair the visual reconstruction reference quickly by post-processing the existing open-oven result.pt, then validate whether PHC passive-object tracking starts to work.
Core rule for this round:
Start from the existing open-oven result.pt.
Use the existing VLM tracker_contact_ref.npz directly.
Do not rerun VLM.
Do not rerun the full visual reconstruction pipeline.
Do not optimize the full body or object root in the first pass.
Write each variant as a new result.pt under the worktree.
Recommended worktree output layout:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D-phc-open-oven/experiments/phc_open_oven/contact_refine/
input/
result_baseline.pt
tracker_contact_ref.npz
vlm_raw_responses.json
contacts_3d.json
p0_diagnostics/
p1_local_refine/
p2_grasp_repair/
p3_visual_losses/
p4_external_priors/
validation_phc/
Each variant should write:
result_<variant>.pt
metrics_before_after.json
contact_distance_timeseries.csv
overlay_before_after.mp4
notes.md
Only run PHC-X passive validation after a variant passes local visual/contact metrics. Local iteration should be minutes, not hours.
Priority P0: Data Prep and No-Optimization Diagnostics¶
Purpose:
Make sure all later losses are driven by the correct frame labels, active hand, active fingers,
current articulated object FK, and current result.pt geometry.
P0.1 Copy immutable inputs into the worktree:
copy current result.pt -> contact_refine/input/result_baseline.pt
copy tracker_contact_ref.npz -> contact_refine/input/tracker_contact_ref.npz
copy vlm_raw_responses.json -> contact_refine/input/vlm_raw_responses.json
copy contacts_3d.json -> contact_refine/input/contacts_3d.json
P0.2 Parse and report VLM contact:
active hand by frame
active finger slots by frame
contact_obj active frame intervals
contact onset frame from VLM contact_obj
dominant active slots and their frequency
P0.3 Recompute all 3D geometry from the current result_baseline.pt:
SMPL-X joints / hand vertices in world
articulated oven full mesh per frame
active moving-link mesh per frame
moving-link normals
joint origin / axis from the current articulated spec
TAPIP3D motion onset fallback from track3d if summary has -1
P0.4 Baseline metrics:
VLM active finger joints -> active moving link distance
VLM active finger joints -> candidate contact region distance
left hand mesh -> full object signed distance
penetration p50/p95/max
revolute torque proxy sign and magnitude after object-motion onset
object qpos raw pre-motion noise
P0.5 Candidate region ablation without optimizing:
candidate_a_current_nearest:
current active-link vertices nearest to VLM active finger joints
candidate_b_lever_arm:
high lever-arm vertices on the moving link, far from hinge axis
candidate_c_motion_extreme:
vertices with largest displacement between closed and opened qpos
candidate_d_visible_surface:
active-link vertices projected inside object mask and visible in camera
candidate_e_vlm_object_points:
object_points_world from contacts_3d.json, only if visually aligned with current result
candidate_f_mixture:
weighted union of nearest + lever-arm + visible + motion-extreme
P0 acceptance:
metrics and overlay exist
the active hand / active slots match VLM raw responses
the selected candidate region is on the moving link, not the static body
no optimization has modified result.pt yet
Priority P1: Low-Risk Post-Process Visual Recon Refinement¶
These are the first variants to run. They should only edit local upper body / active hand in a copied result.pt.
Common variables:
optimize:
left collar / shoulder / elbow / wrist
left SMPL-X hand pose
optional after first success:
small clavicle / upper-spine compensation
fixed:
pelvis/root translation and orientation
lower body
right body / right hand
object root pose
object scale
object qpos after cleanup
Common losses:
L_contact_region:
active VLM finger joints / local hand vertices close to selected contact candidate region
L_penetration:
InterAct point2point_signed hand-object signed distance, penalize inside-object hand vertices
L_temporal:
first and second differences of optimized arm / hand pose
L_init_anchor:
keep visible pose close to input result.pt
L_hand_prior:
InterAct HandPrior, weak or delayed polish only
P1.1 Qpos cleanup only:
Use TAPIP3D / track3d motion onset as primary source.
Set qpos before visual motion onset to q0.
Smooth qpos after motion onset.
Do not move human.
Run metrics and PHC import sanity check.
Reason:
The current object qpos has early noise. Even if it does not fix contact, it prevents
the refinement loss from chasing a false pre-motion door pose.
P1.2 InterAct-style local contact refine, generic candidate region:
Use tracker_contact_ref.npz contact_human as the frame/finger gate.
Use candidate_f_mixture as the object target.
Use InterAct point2point_signed for penetration.
Use weak HandPrior only in the final polish iterations.
Expected result:
active VLM finger -> candidate mean distance drops below 8cm
no large root or torso drift
penetration p95 stays below 1cm
P1.3 Finger-slot-specific contact loss:
Do not pull the whole hand to the object.
Only supervise active VLM slots:
L_Thumb3, L_Index3, L_Middle3, L_Ring3, etc.
Map slots to SMPL-X joints or nearby fingertip vertices.
This should be preferred over whole-hand Chamfer if the hand is occluded and the fingers are noisy.
P1.4 Contact gate vs motion gate:
contact gate:
frames where contact_obj > threshold, starts around frame 24
strong torque / anti-slip gate:
frames after TAPIP3D object motion onset, around frame 37-41
pre-contact:
no attraction
pre-motion contact:
allow light contact / grasp formation, but do not force door-motion consistency
P1.5 Candidate-region loss ablation:
Run the same optimizer against these targets:
Keep all other weights fixed. Pick the candidate policy that best improves distance without penetration or bad elbow motion.
P1.6 HandPrior schedule ablation:
schedule_a:
no HandPrior
schedule_b:
HandPrior weight 1e-4 for all iterations
schedule_c:
no prior for first 70%, 1e-4 to 1e-3 for final polish
Use this to determine whether prior helps polish or prevents contact.
P1.7 Anchor-weight sweep:
arm_anchor: high / medium / low
hand_anchor: high / medium / low
root_anchor: fixed
torso_anchor: fixed or very high
Do not accept a solution that reaches contact by moving the whole body unnaturally.
Priority P2: Stronger Local Grasp Repair¶
Use P2 only if P1 reduces distance but still cannot reach centimeter-level contact or produce useful torque.
P2.1 Wrist/palm target in object local frame:
Choose a stable target point on the contact candidate region.
Define a palm center and wrist pose relative to the moving link local frame.
Optimize left arm IK so palm/fingertips reach the target during the contact interval.
Blend in/out at interval boundaries.
P2.2 Revolute torque proxy:
hinge_axis = articulated spec axis
hinge_origin = articulated spec origin
r = contact_point - hinge_origin
force_dir = palm motion direction or -contact normal
torque_proxy = dot(hinge_axis, cross(r, force_dir))
After visual motion onset, prefer torque_proxy sign that matches qpos direction.
Do not use this as the only contact loss. It is a physical usefulness term, not a contact detector.
P2.3 Palm-facing / surface-normal loss:
Compute palm normal from wrist/index/pinky joints.
Align palm normal with nearby contact surface normal.
Try both signs once; keep the sign that visually matches the left hand.
P2.4 Contact anti-slip:
For frames after motion onset, transform active fingertip points into moving-link local frame.
Penalize large frame-to-frame local sliding except where VLM changes active finger set.
P2.5 Thumb-opposition / curl prior:
Add weak heuristics:
thumb tip should oppose index/middle tips around the contact region
distal fingers can curl toward palm
avoid isolated pinky-only contact
Use this only after the geometry loss can reach the object.
P2.6 IK then optimize:
First solve a coarse left-arm IK target to get the wrist near the contact region.
Then run P1-style differentiable optimization for fingers, penetration, and smoothness.
This is more robust than asking gradient descent to close a 30-40cm gap from scratch.
Priority P3: Add Visual Consistency Terms if P1/P2 Drift¶
Use P3 if local repair reaches the oven but visibly breaks the video alignment.
P3.1 2D projected-joint anchor:
Project optimized SMPL-X finger/wrist/elbow joints back to the original camera.
Keep visible non-contact joints close to the original reconstruction.
Allow active contact fingers to move more.
P3.2 Human mask / silhouette consistency:
Penalize optimized visible hand/forearm vertices projected outside the human mask.
Use only a lightweight sample of vertices.
Do not make this a dense renderer-first project.
P3.3 Object mask / qpos consistency:
If qpos cleanup visibly conflicts with the object mask, run a small qpos-only visual fit.
Keep object root fixed.
Keep pre-motion qpos static.
Only smooth qpos after motion onset.
P3.4 Occlusion-aware anchor:
Where hand is occluded by the oven/object, allow larger 3D repair.
Where hand/arm is visible, use stronger 2D anchor.
Priority P4: External Priors and Heavier Alternatives¶
Use these only after the fast local route has clear failure modes.
P4.1 InterAct HOI correction components:
Reuse:
point2point_signed
penetration loss
temporal smoothness
HandPrior
wrist/arm optimization idea
Do not directly run InterAct optimize.py unchanged:
it assumes rigid object / OMOMO-style human.npz/object.npz
it infers contact from current distance, which is wrong when the current reference is 20-40cm away
it does not understand articulated moving links or hinge qpos
P4.2 HOIFHLI / DexGraspNet-style candidate:
Generate a plausible local hand grasp for the selected contact region.
Use it as a target / prior for SMPL-X hand pose.
Do not replace the full body motion.
P4.3 CARI4D / ArtHOI-style integrated refine:
Use contact + silhouette + penetration + temporal losses in one visual optimization.
This is useful if the post-process route proves too brittle.
It is slower and should not be the first open-oven debugging step.
P4.4 Mark unrecoverable visual reference:
If none of the local repairs can reach contact without destroying video consistency,
record this result.pt as reference_contact_error and do not keep tuning PHC.
Evaluation Gates¶
Every experiment must report the same metrics:
semantic contact metrics:
active VLM slot -> candidate mean / median / min distance
active VLM slot -> moving-link mean / median / min distance
8cm, 5cm, and 2cm contact ratios
geometry safety:
hand-object penetration p50 / p95 / max
number of frames with penetration > 1cm
visual preservation:
root drift
torso drift
left elbow / wrist 2D projection drift if implemented
visible hand mask violation if implemented
object consistency:
pre-motion qpos max deviation from q0
qpos smoothness after motion onset
physics readiness:
revolute torque proxy sign after motion onset
active contact candidate stays on moving link
Only promote a variant to PHC-X passive validation when:
active VLM finger -> contact candidate mean < 5cm
active VLM finger -> contact candidate median < 5cm
min distance < 2cm for most contact frames
8cm contact ratio > 80%
penetration p95 < 1cm
no obvious pre-contact snap in overlay video
no full-body/root cheating
PHC-X validation metrics:
PHC success_rate
mpjpe_g / mpjpe_l / mpjpe_pa
passive qpos_sim max
qpos_sim vs qpos_ref error after motion onset
moving-link contact force
full-body replay video
closeup replay video
Immediate Next Step¶
Implement the fast baseline in the worktree:
1. Create contact_refine/input/ and copy:
- current open-oven result.pt
- existing tracker_contact_ref.npz
- existing vlm_raw_responses.json
- existing contacts_3d.json
2. Add or repair local scripts in the worktree:
- scripts/tuning/eval_articulated_contact_ref.py
- scripts/tuning/refine_articulated_contact.py
- scripts/tuning/render_articulated_contact_overlay.py
3. Run P0 diagnostics first.
4. Run P1.1 qpos cleanup only.
5. Run the first P1 refinement matrix:
- P1.2 mixture candidate, no HandPrior
- P1.2 mixture candidate, delayed HandPrior
- P1.3 finger-slot-specific contact, no HandPrior
- P1.5 current_nearest / lever_arm / visible_surface / mixture ablation
6. Pick the best local visual variant by metrics + overlay video.
7. Only then export that variant to PHC-X passive-object validation.
8. If local visual metrics cannot pass, escalate to P2 IK/grasp repair.
Local Code References¶
Contact-Aware Recon / Joint Optimization¶
CARI4D:
/DATA/intern/hoi4d/projects/xuqin/repos/CARI4D/run_horefine.py
/DATA/intern/hoi4d/projects/xuqin/repos/CARI4D/learning/training/opt_refineout.py
/DATA/intern/hoi4d/projects/xuqin/repos/CARI4D/prep/precompute_contacts.py
Useful components:
contact distance / contact logits
object silhouette loss
penetration loss with VolumetricSMPL
temporal smoothness
initial object/human anchor
human pose + object pose joint optimization
ArtHOI:
/DATA/intern/hoi4d/projects/zhanbaiting/models/ArtHOI/src/train.py
/DATA/intern/hoi4d/projects/zhanbaiting/models/ArtHOI/src/utils/loss_utils.py
Useful components:
infer contact window from object motion
pseudo 3D contact keypoints in contact window
SMPL-X body/hand pose optimization
collision / hand-object proximity loss
Grasp Prior / Hand Repair¶
HOIFHLI / DexGraspNet-style:
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D/submodules/hoifhli_release/grasp_generation/gen_grasp.py
/DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D/submodules/hoifhli_release/grasp_generation/utils/energy.py
Useful components:
E_dis: contact point to object surface distance
E_pen: hand-object penetration
E_prior: MANO hand pose prior
E_spen: hand self-penetration
E_fc: force-closure proxy
OmniGrasp:
Useful components:
object-mesh-conditioned full-body humanoid grasping
later-stage physics controller / policy validation
OmniGrasp is better treated as a later physics/controller stage, not the first replacement for visual reconstruction.
Papers¶
Contact-Aware / Joint Human-Object Reconstruction¶
-
CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction
Monocular metric-scale 4D human-object reconstruction; relevant for HORefine-style contact, silhouette, penetration, and temporal losses. -
ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
Treats articulated HOI synthesis as monocular 4D reconstruction; relevant for contact, articulation, and temporal coherence. -
ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions
Optimization-based monocular hand-articulated-object reconstruction; directly relevant to hand-object contact under occlusion. -
CHORE: Contact, Human and Object REconstruction from a single RGB image
Contact-aware human-object reconstruction from RGB; useful for contact-conditioned fitting. -
PHOSA: Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild
Joint human-object spatial optimization with scale, occlusion-aware silhouette, and interaction layout constraints. -
Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation
Uses contact transfer and joint optimization; useful for contact closeness, normal consistency, and interpenetration constraints.
Sequence / Grasp Refinement¶
-
TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement
Targets noisy hand-object tracking sequences and learns an object-centric temporal prior for plausible contact. -
GRAB: A Dataset of Whole-Body Human Grasping of Objects
Whole-body SMPL-X grasping dataset with object motion, detailed hand pose, and contact; useful for human grasp priors. -
DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation
Differentiable force-closure / contact / penetration style grasp synthesis; useful for candidate generation. -
OmniGrasp: Grasping Diverse Objects with Simulated Humanoids
Object-mesh-conditioned full-body humanoid grasping; suitable as a physics/controller stage after reference repair. -
Human-Object Interaction from Human-Level Instructions
Includes detailed finger-object interactions and physics tracking; local repo has DexGraspNet-derived grasp optimization components.
Do Not Do¶
Do not modify implementation files under /DATA/intern/hoi4d/projects/zhanbaiting/ArtHOI4D.
Do not run PHC experiments from the main repo.
Do not switch the main repo branch.
Do not keep tuning PHC tracker or IsaacGym physics parameters as the primary next step.
Do not rerun VLM for the first fast-iteration pass; use the existing tracker_contact_ref.npz.
Do not rerun the full visual reconstruction pipeline for the first pass; post-process the existing open-oven result.pt.
Do not treat VLM contact onset and object motion onset as the same frame.
Do not hard-code frame 37 except as a debug override; read TAPIP3D / track3d onset when available.
Do not trust old contacts_3d object_points_world blindly if they were generated from a different result.pt; recompute object-side geometry from the current result.pt.
Do not treat object replay videos as proof of passive physics success.
Do not overwrite object qpos during passive validation.
Do not jump directly to full-body grasp replacement without preserving video/recon consistency.