On-device navigation, built at Kangnam University

EdgeGround-VLA: A Lightweight Model Fusing Heterogeneous Grounding Encoders for Goal-Directed Navigation in Wheeled Robots

EdgeGround-VLA fuses OWLv2 (open-vocabulary detection) and Kosmos-2 (phrase grounding) — both frozen — behind a 0.866M-parameter MLP action head. Only 1.128M of 459M parameters are trained, and the whole pipeline runs entirely on an NVIDIA Jetson Orin NX with no cloud connectivity and no pre-built map.

Min-Woo Lee*, In-Yeop Choi† *Dept. of Artificial Intelligence · †Dept. of Computer Science, Kangnam University

📄 Paper (HWP) 💻 Code 🧩 Architecture Notes
95.0%
Real-robot goal-reaching success
(95/100 · 5 positions × 20 trials)
1.128M
Trained parameters
out of 459M total
97%
Of inference latency is
the perception (grounding) stage
0
Cloud calls, pre-built maps,
or demo-tour videos required
Abstract

Grounding, not language modeling, is the bottleneck

Recent mobile-robot navigation research has pushed toward language-instructed navigation with large Vision-Language Models, exemplified by Mobility VLA. But VLM-based inference is computationally heavy and depends on pre-built infrastructure — demo-tour videos and offline topological graphs — which limits deployment on resource-constrained platforms.

EdgeGround-VLA takes a different path. It fuses the representations of two vision encoders with distinct grounding characteristics — OWLv2 for real-time open-vocabulary object localization, and Kosmos-2's vision encoder for phrase-grounded visual features — and combines them with a lightweight MLP action head that outputs one of 8 discrete movement classes, mapped to a 3-DoF command via a fixed lookup table. Both backbones stay frozen; only the 0.866M-parameter action head (plus a 0.262M projection layer) is trained. Deployed on a Jetson Orin NX with no cloud connectivity, the system reaches a 95.0% goal-reaching success rate across 100 real-world trials — above the 86–90% end-to-end success Mobility VLA reports in comparable settings. Decomposing the pipeline by component further shows that OWLv2 grounding alone accounts for ~97% of inference latency, identifying exactly where future compression effort (quantization, distillation) should go when porting to platforms as constrained as a Raspberry Pi.

Method

Heterogeneous grounding, fused behind a frozen backbone

A 1280×720 RGB frame is processed in parallel by two independently pre-trained, frozen encoders — one built for detection, one built for phrase grounding — and their outputs are concatenated into a single 260-dimensional frame feature.

Fig. 1 — Model Architecture. OWLv2 contributes a 4-channel bbox feature (cx, cy, area, has_bbox); Kosmos-2's vision encoder contributes a 1024-dim feature that is linearly projected to 256-dim and L2-normalized. The two are concatenated into a 260-dim frame feature, six frames are flattened into a 1560-dim input, and a 2-hidden-layer MLP (512 → 128) outputs one of 8 discrete action classes.
EdgeGround-VLA model architecture diagram
OWLv2
0.155B · frozen
Open-vocabulary detection → bbox (cx, cy, area, has_bbox)
+
Kosmos-2 vision encoder
0.303B · frozen
Phrase-grounded 1024-dim feature
→
image_proj
0.262M · Stage 1
Linear 1024→256 + L2 norm
→
MLP action head
0.866M · trained
1560 → 512 → 128 → 8-class logit

The action space is deliberately discrete: 8 classes covering stop, forward, lateral strafe (×2), diagonal strafe (×2), and in-place rotation (×2) — well matched to the robot's 3-axis omni-wheel (Kiwi-drive) chassis, which can move laterally without rotating. Continuous regression or diffusion/flow-matching action experts were considered and rejected: they typically need far more demonstration data than this project's 225-episode dataset provides. The 8-class logit is mapped to a 3-DoF (x, y, z) command via a fixed lookup table:

classactionxyz
0STOP+0.00+0.00+0.00
1FORWARD+1.15+0.00+0.00
2LEFT+0.00+1.15+0.00
3RIGHT+0.00-1.15+0.00
4FWD+LEFT+1.15+1.15+0.00
5FWD+RIGHT+1.15-1.15+0.00
6ROT_L+0.00+0.00+0.25
7ROT_R+0.00+0.00-0.25

Table 1 — 8-class → 3-DoF lookup table (fixed constants; the robot's firmware ignores speed magnitude, so only the direction bits matter).

Fig. 2 — Hardware Platform. Top board carries the Jetson Orin NX, camera, and 3-axis omni-wheel drive; the motor board's Cortex-M4 handles CAN-bus motor control, current limiting, and sensor fusion independently of the inference pipeline.
Serbot II hardware block diagram — top board and motor board
Data collection pipeline diagram
Fig. 3 — Data Collection. 225 teleoperated episodes (16,599 frames) across 5 target directions, collected via joystick with cx-label mean 0.4954 — confirming no left/right spatial bias in the raw dataset.
Stage 1 / Stage 2 training pipeline diagram
Fig. 4 — Two-Stage Training. Stage 1 trains image_proj via direction anchor-sentence matching (5-class), then freezes it. Stage 2 trains only the MLP action head on top of the frozen representation — 1.128M parameters end to end.
ItemValue
Episodes225 (track A 180 + track F 45)
Total frames16,599
Train / val split192 / 33 (seed 42, episode-level)
BBox annotationbbox_dataset_v6_owl.json
Deployed headexp73_owl_stage1v3_v6_mlp.pt

Table 2 — Dataset composition.

ClassFramesShare
STOP1,4528.75%
FORWARD7,64146.03%
LEFT7164.31%
RIGHT8585.17%
FWD+LEFT2,53215.25%
FWD+RIGHT3,11718.78%
ROT_L1360.82%
ROT_R1470.89%

Table 3 — Action class distribution. FORWARD dominance (46.03%) motivates the inverse-frequency class weighting used in Stage 2.

Training

Stage 1 generalizes cleanly; Stage 2 shows the expected imitation-learning gap

image_proj (Stage 1) trains to near-identical train/validation accuracy. The action head (Stage 2) shows a much larger train–validation gap — a well-documented pattern in teleoperation-based imitation learning, and one that episode-level closed-loop control is structurally able to correct for.

DirectionFramesAnchor sentence (Stage 1 contrastive target)Class weight
strong_left3,611"...strongly on the left side of the image"0.91
weak_left3,299"...slightly on the left side of the image"1.00
center2,819"...in the center of the image"1.17
weak_right3,268"...slightly on the right side of the image"1.01
strong_right3,602"...strongly on the right side of the image"0.91

Table 6 — The 5 direction anchor sentences ("The gray basket is <degree> on the <side> of the image") that Stage 1 contrastively matches each frame's vision feature against. Frame counts sum to exactly 16,599, matching Table 2.

image_proj Stage 1 training/validation accuracy curve
Fig. 5 — image_proj Accuracy. Train 96.10% / val 94.09% — a 2.01pp gap.
image_proj Stage 1 training/validation loss curve
Fig. 6 — image_proj Loss. Train 0.107 / val 0.173, both converging smoothly.
Action head training/validation accuracy curve
Fig. 7 — Action Head Accuracy. Train 91.44% / val 74.04% at the deployed checkpoint (epoch 225) — a 17.4pp gap, selected by validation accuracy, not loss.
Action head training/validation loss curve
Fig. 9 — Action Head Loss. Validation loss climbs from 0.926 to 1.953 after epoch 20 even as validation accuracy keeps improving — a known pattern when a model grows more confident without its top-1 prediction changing.

Validation accuracy is a lower bound on real-world competence, not a ceiling on it.

Frame-level scoring can't credit re-decision; closed-loop control can.

Why does a 17.4-point gap not translate into a 17.4-point drop in real-world success? Two reasons. First, frame-level validation accuracy asks whether the model matches the single action the human happened to choose; at any given moment, several actions can be equally valid (e.g. FORWARD and FWD+LEFT both make progress when the target is slightly left). Second, the robot re-observes and re-decides at every timestep — an average of 11.64 decisions per successful episode — so a substitutable frame-level "error" gets corrected on the very next frame. Breaking down the 629 validation errors by severity confirms this directly:

Deployed action head validation confusion matrix, stride=1 scoring
Fig. 7 — Validation Confusion Matrix. Deployed head (exp73_owl_stage1v3_v6_mlp.pt) scored with the same stride=1 protocol used in training — 74.1% overall (n=2,431). The 6 well-sampled classes score 66.7–84.5% fairly evenly; only ROT_L/ROT_R look low, and they are 1.15% of validation (28/2,431), so one or two misclassifications swing their per-class number sharply.
Validation confusion matrix colored by error severity
Fig. 8 — Same Confusion Matrix, by Error Severity. 81.4% of errors are confusions between actions with the same directional tendency (substitutable) — only 1.97% of all predictions are a critical left/right flip that a closed loop cannot self-correct.
ModuleStatus
OWLv2frozen (zero-shot)
Kosmos-2 vision_modelfrozen
image_projStage 1 trained → frozen for deploy
text_projStage 1 trained, deploy-unused
MLP action headStage 2 trained (0.866M)
Language decodernever called at deploy time

Table 4 — Training status of every pipeline component.

ParamValue
threshold0.25 default / 0.20 deployed
window6 frames
bbox_scale3.0
optimizerAdamW, lr=5e-4
epochs300, CosineAnnealingLR
class weightinverse-frequency, normalized

Table 5 — Stage 2 action-head hyperparameters.

Real-World Results

95/100 goal-reaching trials, evenly spread across 5 target directions

100 real-robot episodes — 20 per target position — run entirely on-device on the Jetson Orin NX, with no cloud connectivity and no pre-built map.

CENTER
100%
20/20 · 10.00 steps
STRONG LEFT
95%
19/20 · 13.35 steps
WEAK LEFT
95%
19/20 · 11.55 steps
STRONG RIGHT
95%
19/20 · 12.75 steps
WEAK RIGHT
90%
18/20 · 10.55 steps
OVERALL
95.0%
95/100 · avg. 11.64 steps
Real-world test track — overview, target approach, and target capture
Fig. 12–13 — Practical Test. (a) full track overview, (b) target approach, (c) target capture, run on the indoor test course used for all 100 trials.
Deployment

The bottleneck is grounding — not the model doing the reasoning

Decomposing end-to-end latency by pipeline stage reveals a counter-intuitive result: the smaller model is the slower one.

OWLv2 grounding (fp32)
1901.7 ms · ~97%
OWLv2 grounding (fp16)
962.1 ms
Kosmos-2 vision encoding
53.7 ms · ~3%
MLP action head
<1 ms · ~0%

OWLv2 (0.155B params) is responsible for ~97% of end-to-end latency, while Kosmos-2's vision encoder — nearly twice as many parameters (0.303B) — accounts for only ~3%. Parameter count alone does not predict inference latency; OWLv2's cost comes from cross-attention text-query matching and multi-scale anchor processing, not raw FLOPs. Switching OWLv2 to fp16 cuts its latency by ~49% (1901.7ms → 962.1ms), confirming that compressing the grounding stage — not the policy — is where future work should focus when porting to platforms as constrained as a Raspberry Pi.

End-to-end pipeline latency is ~1.96s (fp32) or ~1.02s (fp16) — far from the tens-of-milliseconds a continuous controller would need. But EdgeGround-VLA doesn't need continuous control: it updates a discrete action periodically, re-issuing the last decided command at 10Hz between grounding updates so the robot never actually stops waiting. The 95% real-world success rate is the direct evidence that this control scheme is practical for goal-reaching.

Deployment and runtime process diagram
Fig. 10 — Deploy & Runtime. The trained checkpoints ship to the Jetson Orin NX; a FastAPI dashboard exposes robot control, status, and camera stream over Tailscale VPN.
Runtime inference control logic diagram
Fig. 11 — Runtime Control Logic. A sliding 6-frame window feeds the action head; the resulting 3-DoF command is sent over CAN → MCU → wheels.
Citation

BibTeX

@article{lee2026edgeground, title = {EdgeGround-VLA: A Lightweight Model Fusing Heterogeneous Grounding Encoders for Goal-Directed Navigation in Wheeled Robots}, author = {Lee, Min-Woo and Choi, In-Yeop}, journal = {Journal of The Korea Society of Computer and Information (JKSCI)}, volume = {29}, number = {11}, year = {2026}, publisher = {Korea Society of Computer and Information} }

Kangnam University, Dept. of Artificial Intelligence / Dept. of Computer Science.

More

Project documentation

Architecture

Model Structure & Training Notes

Full parameter breakdown, hyperparameters, and inference control logic, kept in sync with the deployed checkpoint.

 Grounder

OWLv2 Grounder Notes

Why OWLv2 was adopted, threshold/fp16/skip_n behavior, and the real-world grounding-availability data behind it.

 Paper

Revision Briefing

Line-by-line advisor Q&A, reference audit, and before/after diffs tracked through the KCI submission process.

 Research Log

Research Dashboard & Status Log

The full experiment history — offline ablations, chapter-by-chapter findings, and archived legacy results.

 Wiki

Research Wiki (Topic Index)

The 70+ research-log chapters reorganized by topic (grounding, action heads, backbone, real-robot testing, ...) instead of chronologically.

 Code

GitHub Repository

Training scripts, inference server, and figure-generation code referenced throughout this page.

 Related

MoNa-pi (Sister Project)

A π0-style flow-matching VLA baseline explored on the same platform, for comparison.