From 08f1aac75817af7d36ac26e85f58df6fbe0d43a6 Mon Sep 17 00:00:00 2001 From: Kaloyan Nikolov Date: Mon, 6 Apr 2026 14:55:20 +0200 Subject: [PATCH] Final README with accurate LiteRT-LM integration guidance - Documents the C++ bridge approach based on Google's actual implementation - Explains MediaPipe (deprecated) vs C++ bridge vs Swift APIs (coming soon) - Clear instructions for obtaining LiteRT-LM binaries - Project structure and architecture diagram --- README.md | 191 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..0f1b45e --- /dev/null +++ b/README.md @@ -0,0 +1,191 @@ +# Sleepy Agent iOS + +A Swift port of Sleepy Agent - a fully local AI assistant for iOS powered by Google's Gemma 4 models. + +## Status: Developer Build (Work in Progress) + +**Architecture complete** - waiting on LiteRT-LM integration. + +## Quick Facts + +- **19 Swift files** (~4,500 LOC) +- **3 Objective-C++ bridge files** for LiteRT-LM integration +- **5 documentation files** +- **iOS 15.0+** target +- **SwiftUI** for UI + +## What's Working + +✅ Complete SwiftUI interface (chat, settings, navigation) +✅ Audio recording & playback +✅ Text-to-Speech (TTS) +✅ Web search (SearXNG) +✅ Model download manager +✅ Conversation persistence +✅ Image picker +⚠️ LLM inference (stubbed - needs LiteRT-LM binaries) + +## The LiteRT-LM Situation + +Based on research of Google's repositories: + +| Option | Status | Gemma 4 | Recommendation | +|--------|--------|---------|----------------| +| MediaPipe Tasks Swift | Deprecated | ❌ | Short-term only | +| **C++ Bridge** ⭐ | **Available** | ✅ | **Recommended** | +| Native Swift APIs | Coming soon | ✅ | Wait if possible | + +### Google's Approach + +The [Google AI Edge Gallery](https://apps.apple.com/us/app/google-ai-edge-gallery/id6749645337) iOS app uses a **C++ bridge** (confirmed by source code analysis): + +- [litert-samples](https://github.com/google-ai-edge/litert-samples) shows ObjC++ bridge pattern +- MediaPipe has [working Swift API](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/ios) but it's deprecated +- [Issue #420](https://github.com/google-ai-edge/gallery/issues/420) confirms iOS source isn't public + +### Our Implementation + +**Objective-C++ Bridge Files** (included): +- `SleepyAgent/Inference/Bridge/LlmEngineBridge.h` - Header +- `SleepyAgent/Inference/Bridge/LlmEngineBridge.mm` - Implementation +- `SleepyAgent/Inference/Bridge/SleepyAgent-Bridging-Header.h` - Swift bridge + +See [LITERT_INTEGRATION.md](LITERT_INTEGRATION.md) for details. + +## Building + +### Prerequisites + +- macOS with Xcode 15+ +- iOS 15.0+ device or simulator +- CocoaPods (`sudo gem install cocoapods`) + +### Quick Start + +```bash +cd sleepy_agent_ios + +# Check status +make check + +# Setup project +make setup # Installs CocoaPods + +# Open in Xcode +open SleepyAgent.xcworkspace # or Package.swift +``` + +### To Enable LLM Inference + +You need to obtain LiteRT-LM binaries: + +1. **Build from source**: https://github.com/google-ai-edge/LiteRT-LM +2. **Download releases**: Check GitHub releases page +3. **Contact Google**: For prebuilt iOS binaries + +Then: +1. Add LiteRT-LM headers to Xcode project +2. Link `liblitert_lm.a` static library +3. Configure bridging header in Build Settings +4. Uncomment C++ code in `LlmEngineBridge.mm` + +See [BUILD.md](BUILD.md) and [LITERT_INTEGRATION.md](LITERT_INTEGRATION.md) for full instructions. + +## Project Structure + +``` +SleepyAgent/ +├── App/ # Entry point, app lifecycle +├── Core/ # Models, DI container +│ └── Models/Message.swift # Data models +├── Inference/ # LLM engine +│ ├── Bridge/ # ⭐ ObjC++ LiteRT-LM bridge +│ ├── Agent.swift # Tool calling, chat logic +│ ├── LlmEngine.swift # Swift engine interface +│ └── ... +├── Services/ # Audio, network, storage +│ ├── AudioRecorder.swift +│ ├── TtsService.swift +│ ├── WebSearchService.swift +│ └── ... +└── UI/ # SwiftUI views + ├── Views/ # MainView, SettingsView, etc. + └── ViewModels/ # State management +``` + +## Architecture + +``` +┌─────────────────────────────────────────┐ +│ SwiftUI Views │ +│ (MainView, SettingsView, ...) │ +└─────────────────┬───────────────────────┘ + │ +┌─────────────────▼───────────────────────┐ +│ Swift ViewModels │ +│ (MainViewModel, SettingsViewModel) │ +└─────────────────┬───────────────────────┘ + │ +┌─────────────────▼───────────────────────┐ +│ Swift LlmEngine (Actor) │ +│ (async/await, thread-safe) │ +└─────────────────┬───────────────────────┘ + │ +┌─────────────────▼───────────────────────┐ +│ Objective-C++ LlmEngineBridge │ +│ (ObjC++ wrapper for LiteRT-LM) │ +└─────────────────┬───────────────────────┘ + │ +┌─────────────────▼───────────────────────┐ +│ C++ LiteRT-LM Library │ +│ (Google's on-device LLM engine) │ +└─────────────────────────────────────────┘ +``` + +## Documentation + +- **[BUILD.md](BUILD.md)** - Build instructions, troubleshooting +- **[LITERT_INTEGRATION.md](LITERT_INTEGRATION.md)** - LiteRT-LM integration details +- **[LITERT_IOS_STATUS.md](LITERT_IOS_STATUS.md)** - Current state of LiteRT on iOS +- **[PROJECT_STATUS.md](PROJECT_STATUS.md)** - Overall project status +- **[FILES.md](FILES.md)** - Complete file listing + +## Key Differences from Android + +| Feature | Android | iOS | +|---------|---------|-----| +| UI | Jetpack Compose | SwiftUI | +| Async | Kotlin Coroutines | Swift Concurrency (async/await) | +| DI | Manual | Manual | +| Storage | DataStore + JSON | UserDefaults + FileManager | +| LLM Engine | LiteRT-LM Kotlin | LiteRT-LM C++ via bridge | +| Audio | AudioRecord | AVAudioEngine | +| Background | WorkManager | URLSession (limited) | + +## Testing Without LLM + +The app works in "stub mode" for UI testing: + +1. Build and run +2. Type any message +3. See placeholder response explaining LiteRT-LM integration + +## Next Steps + +To complete the iOS port: + +1. Obtain LiteRT-LM iOS binaries (build or download) +2. Add to Xcode project +3. Configure build settings for C++ +4. Uncomment C++ code in bridge files +5. Test with Gemma 4 E2B model + +## Need Help? + +- Open an issue: https://github.com/sleepyeldrazi/sleepy-agent/issues +- Check [LITERT_INTEGRATION.md](LITERT_INTEGRATION.md) for troubleshooting +- See Google's [litert-samples](https://github.com/google-ai-edge/litert-samples) for reference + +## License + +MIT License