بَصير

01What it does

Say, in any Arabic dialect — «ناوليني سيروم الشعر» — and the arm sees what's on the table, localizes the item, grasps it with a learned policy, retries if it misses, hands it to a fixed delivery zone, and speaks back in Arabic. If the item isn't there, it doesn't guess — it says aloud what is.

02The full pipeline

Fanar reasons over four tools; deliver() runs a hybrid perception-guided grasp.

1. Listen — phone tap-to-talk → Aura ASR → normalize.py fixes diacritics/brand mishears.
2. Reason — Fanar agent loop (ReAct) emits JSON actions: perceive → deliver → say / ask.
3. See — Fanar-Oryx reads the labels and reports which serums are on the table.
4. Localize — Oryx returns the target's pixel; a calibrated map → arm hover pose above it.
5. Grasp — SmolVLA policy does the final descent + close from the hover.
6. Verify — gripper torque (current) + width say HELD or EMPTY.
7. Retry — on a miss: open, re-home, re-localize, try again.
8. Deliver & speak — scripted waypoints to the hand-off zone, release, confirm in Arabic via Aura.

03Hierarchical VLA

Fanar is a task-level Vision-Language→Action controller (actions = tool calls); SmolVLA is the low-level motor controller. No hardcoded if/else — every semantic decision is Fanar's.

SYSTEM:  أنت بَصير … اعرض المشهد قبل التسليم؛ تكلّم دائماً؛ وضّح عند الالتباس
USER:    ناوليني السيروم
  Fanar ▶ {"action":"perceive_scene"}
OBS:     {"items":["سيروم الشعر","سيروم الوجه"]}
  Fanar ▶ {"action":"ask","args":{"text_ar":"عندي سيروم الوجه وسيروم الشعر، أي واحد تبي؟"}}
USER:    سيروم الشعر
  Fanar ▶ {"action":"deliver","args":{"item":"سيروم الشعر"}}
  Fanar ▶ {"action":"say","args":{"text_ar":"تفضّل، سيروم الشعر أمامك"}}

04Dataset & model

05Use of Fanar

Dialect handling is the star: Gulf / Egyptian / MSA map directly to actions — no English translation step. Oryx keeps perception Fanar-native and supplies the localization that guides the arm.

06Run it

# voice / agent demo (laptop)
pip install -r requirements.txt
cp .env.example .env          # FANAR_API_KEY=...
cd backend && BASEER_PERCEIVE=oryx python server.py   →  http://localhost:8080

# deploy the grasp (laptop with the SO-100)
python backend/robot/calibrate_grasp.py        --port <follower> --id follower_so100
python backend/robot/save_delivery_pose.py     --port <follower> --id follower_so100
python backend/robot/calibrate_localization.py --port <follower> --id follower_so100
python backend/agent/agent4_grasp.py --policy ~/baseer/policy_vla/pretrained_model \
  --task "Pick up the hair serum and place it in the delivery zone" \
  --item "سيروم الشعر" --attempts 3