# Mobile VLA LoRA Fine-tuning μ‹€ν–‰ μš”μ•½ ## πŸ“‹ μƒμ„±λœ 파일 ### 1. Config 파일 ``` Mobile_VLA/configs/finetune_mobile_vla_lora_20251106.json ``` - RoboVLMs upstream config 기반 - LoRA μ„€μ •: r=32, alpha=16, dropout=0.1 - 2D μ•‘μ…˜ 곡간: action_dim=2 (linear_x, linear_y) - Jetson μ΅œμ ν™”: batch_size=2, fp16, memory_limit=14GB ### 2. 데이터셋 κ΅¬ν˜„ ``` Mobile_VLA/src/data/mobile_vla_h5_dataset.py ``` - HDF5 파일 λ‘œλ“œ (20251106 μ—ν”Όμ†Œλ“œ) - Window size=8, Action chunk=10 - 이미지 μ „μ²˜λ¦¬ (224x224, ImageNet μ •κ·œν™”) - Train/Val λΆ„ν•  (80/20) ### 3. LoRA Fine-tuning 슀크립트 ``` Mobile_VLA/src/training/finetune_lora_20251106.py ``` - Kosmos-2 VLM + LoRA 적용 - AdamW Optimizer + Cosine Annealing LR - Gradient Clipping (max_norm=1.0) - 체크포인트 μ €μž₯ (best + 주기적) ### 4. μ‹€ν–‰ 슀크립트 ``` Mobile_VLA/scripts/run_lora_finetune_20251106.sh ``` - CUDA 확인 - 데이터셋 확인 - LoRA Fine-tuning μ‹€ν–‰ - ν•™μŠ΅ μ‹œκ°„ μΈ‘μ • ### 5. ν…ŒμŠ€νŠΈ 슀크립트 ``` Mobile_VLA/scripts/test_dataset_20251106.py ``` - 데이터셋 λ‘œλ“œ ν…ŒμŠ€νŠΈ - 배치 생성 ν…ŒμŠ€νŠΈ ### 6. κ°€μ΄λ“œ λ¬Έμ„œ ``` Mobile_VLA/README_LORA_FINETUNING.md ``` - 전체 μ‹€ν–‰ κ°€μ΄λ“œ - 문제 ν•΄κ²° 방법 - μ°Έκ³  자료 --- ## πŸš€ μ‹€ν–‰ μˆœμ„œ ### 1단계: 데이터셋 ν…ŒμŠ€νŠΈ ```bash cd /home/billy/25-1kp/vla python3 Mobile_VLA/scripts/test_dataset_20251106.py ``` **μ˜ˆμƒ κ²°κ³Ό:** - βœ… 11개 Training μ—ν”Όμ†Œλ“œ - βœ… 2개 Validation μ—ν”Όμ†Œλ“œ - βœ… 배치 λ‘œλ“œ 성곡 ### 2단계: LoRA μ‹œκ°„ μΈ‘μ • (1 에포크) ```bash # Config μˆ˜μ •: max_epochsλ₯Ό 1둜 λ³€κ²½ vim Mobile_VLA/configs/finetune_mobile_vla_lora_20251106.json # μ‹€ν–‰ bash Mobile_VLA/scripts/run_lora_finetune_20251106.sh # κ²°κ³Ό 확인 cat Mobile_VLA/runs/mobile_vla_lora/logs/training_results.json | grep avg_epoch_time ``` ### 3단계: 전체 ν•™μŠ΅ (50 에포크) ```bash # Config μˆ˜μ •: max_epochsλ₯Ό 50으둜 λ³€κ²½ vim Mobile_VLA/configs/finetune_mobile_vla_lora_20251106.json # μ‹€ν–‰ bash Mobile_VLA/scripts/run_lora_finetune_20251106.sh ``` --- ## πŸ“Š μ˜ˆμƒ κ²°κ³Ό ### ν•™μŠ΅ μ‹œκ°„ (Jetson AGX Orin 16GB) - **1 에포크**: ~5-10λΆ„ (μ˜ˆμƒ) - **50 에포크**: ~4-8μ‹œκ°„ (μ˜ˆμƒ) ### λͺ¨λΈ 크기 - **Full Model**: ~2GB (Kosmos-2) - **LoRA Adapter**: ~50MB (ν•™μŠ΅ νŒŒλΌλ―Έν„°λ§Œ) ### ν•™μŠ΅ νŒŒλΌλ―Έν„° - **Total Parameters**: ~1.3B (Kosmos-2) - **Trainable Parameters**: ~10M (LoRA, <1%) --- ## 🎯 핡심 차이점: RoboVLMs vs Mobile VLA | ν•­λͺ© | RoboVLMs | Mobile VLA | |------|----------|------------| | **Fine-tuning** | Full FT | LoRA | | **Action Space** | 7D (6-DOF + gripper) | 2D (linear_x, linear_y) | | **Dataset** | CALVIN 24K episodes | 13 episodes (20251106) | | **Epochs** | 5 | 50 | | **Learning Rate** | 2e-5 | 1e-4 | | **Batch Size** | 4 | 2 | | **Hidden Size** | 1024 | 512 | | **Trainable %** | 100% | <1% | --- ## πŸ“ 좜λ ₯ ꡬ쑰 ``` Mobile_VLA/runs/mobile_vla_lora/ β”œβ”€β”€ checkpoints/ β”‚ β”œβ”€β”€ best_model.pth # 졜고 μ„±λŠ₯ λͺ¨λΈ β”‚ β”œβ”€β”€ checkpoint_epoch_10.pth β”‚ β”œβ”€β”€ checkpoint_epoch_20.pth β”‚ β”œβ”€β”€ checkpoint_epoch_30.pth β”‚ β”œβ”€β”€ checkpoint_epoch_40.pth β”‚ └── checkpoint_epoch_50.pth └── logs/ β”œβ”€β”€ training_results.json # ν•™μŠ΅ κ²°κ³Ό μš”μ•½ β”œβ”€β”€ events.out.tfevents.* # TensorBoard └── metrics.csv ``` --- ## πŸ” ν•™μŠ΅ λͺ¨λ‹ˆν„°λ§ ### TensorBoard ```bash tensorboard --logdir=Mobile_VLA/runs/mobile_vla_lora/logs # http://localhost:6006 ``` ### μ‹€μ‹œκ°„ 둜그 ```bash tail -f Mobile_VLA/runs/mobile_vla_lora/logs/training_results.json ``` --- ## βœ… 검증 ν•­λͺ© ### 데이터셋 - [x] 20251106 μ—ν”Όμ†Œλ“œ 13개 확인 - [x] HDF5 ꡬ쑰 확인 (images, actions, action_event_types) - [x] Train/Val λΆ„ν•  (11/2) ### λͺ¨λΈ - [x] Kosmos-2 λ‘œλ“œ - [x] LoRA 적용 (r=32, alpha=16) - [x] 2D μ•‘μ…˜ ν—€λ“œ (action_dim=2) - [x] Gripper 제거 ### ν•™μŠ΅ - [x] AdamW Optimizer - [x] Cosine Annealing LR - [x] Gradient Clipping - [x] 체크포인트 μ €μž₯ ### Config - [x] RoboVLMs upstream μ°Έμ‘° - [x] Mobile VLA νƒœμŠ€ν¬ 적응 - [x] Jetson μ΅œμ ν™” --- ## πŸ“š μ°Έκ³  μ½”λ“œ ### RoboVLMs Upstream ``` RoboVLMs_upstream/configs/calvin_finetune/ └── finetune_kosmos_cont-lstm-post_full-ft_text_vision_wd-0_ws-8_act-10.json ``` ### Mobile VLA κ΅¬ν˜„ ``` Mobile_VLA/src/ β”œβ”€β”€ model/mobile_vla_model.py # LoRA 적용된 Kosmos-2 β”œβ”€β”€ data/mobile_vla_h5_dataset.py # HDF5 데이터셋 └── training/finetune_lora_20251106.py # LoRA Fine-tuning ``` --- ## πŸŽ‰ μ™„λ£Œ ν›„ λ‹€μŒ 단계 1. **μΆ”λ‘  ν…ŒμŠ€νŠΈ**: ν•™μŠ΅λœ λͺ¨λΈλ‘œ μΆ”λ‘  2. **μ„±λŠ₯ 평가**: MAE, MSE 계산 3. **μ‹€μ‹œκ°„ ν…ŒμŠ€νŠΈ**: Jetsonμ—μ„œ λ‘œλ΄‡ μ œμ–΄ 4. **100 Dataset μˆ˜μ§‘**: μΆ”κ°€ 데이터 μˆ˜μ§‘ 5. **λ…Όλ¬Έ μž‘μ„±**: RoboVLMs + Robot Manipulator --- **μž‘μ„±μΌ**: 2025-11-06 **μƒνƒœ**: κ΅¬ν˜„ μ™„λ£Œ, μ‹€ν–‰ μ€€λΉ„ μ™„λ£Œ **λ‹€μŒ**: LoRA μ‹œκ°„ μΈ‘μ • (1 에포크)