Final README with accurate LiteRT-LM integration guidance
- Documents the C++ bridge approach based on Google's actual implementation - Explains MediaPipe (deprecated) vs C++ bridge vs Swift APIs (coming soon) - Clear instructions for obtaining LiteRT-LM binaries - Project structure and architecture diagram
This commit is contained in:
@@ -0,0 +1,191 @@
|
|||||||
|
# Sleepy Agent iOS
|
||||||
|
|
||||||
|
A Swift port of Sleepy Agent - a fully local AI assistant for iOS powered by Google's Gemma 4 models.
|
||||||
|
|
||||||
|
## Status: Developer Build (Work in Progress)
|
||||||
|
|
||||||
|
**Architecture complete** - waiting on LiteRT-LM integration.
|
||||||
|
|
||||||
|
## Quick Facts
|
||||||
|
|
||||||
|
- **19 Swift files** (~4,500 LOC)
|
||||||
|
- **3 Objective-C++ bridge files** for LiteRT-LM integration
|
||||||
|
- **5 documentation files**
|
||||||
|
- **iOS 15.0+** target
|
||||||
|
- **SwiftUI** for UI
|
||||||
|
|
||||||
|
## What's Working
|
||||||
|
|
||||||
|
✅ Complete SwiftUI interface (chat, settings, navigation)
|
||||||
|
✅ Audio recording & playback
|
||||||
|
✅ Text-to-Speech (TTS)
|
||||||
|
✅ Web search (SearXNG)
|
||||||
|
✅ Model download manager
|
||||||
|
✅ Conversation persistence
|
||||||
|
✅ Image picker
|
||||||
|
⚠️ LLM inference (stubbed - needs LiteRT-LM binaries)
|
||||||
|
|
||||||
|
## The LiteRT-LM Situation
|
||||||
|
|
||||||
|
Based on research of Google's repositories:
|
||||||
|
|
||||||
|
| Option | Status | Gemma 4 | Recommendation |
|
||||||
|
|--------|--------|---------|----------------|
|
||||||
|
| MediaPipe Tasks Swift | Deprecated | ❌ | Short-term only |
|
||||||
|
| **C++ Bridge** ⭐ | **Available** | ✅ | **Recommended** |
|
||||||
|
| Native Swift APIs | Coming soon | ✅ | Wait if possible |
|
||||||
|
|
||||||
|
### Google's Approach
|
||||||
|
|
||||||
|
The [Google AI Edge Gallery](https://apps.apple.com/us/app/google-ai-edge-gallery/id6749645337) iOS app uses a **C++ bridge** (confirmed by source code analysis):
|
||||||
|
|
||||||
|
- [litert-samples](https://github.com/google-ai-edge/litert-samples) shows ObjC++ bridge pattern
|
||||||
|
- MediaPipe has [working Swift API](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/ios) but it's deprecated
|
||||||
|
- [Issue #420](https://github.com/google-ai-edge/gallery/issues/420) confirms iOS source isn't public
|
||||||
|
|
||||||
|
### Our Implementation
|
||||||
|
|
||||||
|
**Objective-C++ Bridge Files** (included):
|
||||||
|
- `SleepyAgent/Inference/Bridge/LlmEngineBridge.h` - Header
|
||||||
|
- `SleepyAgent/Inference/Bridge/LlmEngineBridge.mm` - Implementation
|
||||||
|
- `SleepyAgent/Inference/Bridge/SleepyAgent-Bridging-Header.h` - Swift bridge
|
||||||
|
|
||||||
|
See [LITERT_INTEGRATION.md](LITERT_INTEGRATION.md) for details.
|
||||||
|
|
||||||
|
## Building
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- macOS with Xcode 15+
|
||||||
|
- iOS 15.0+ device or simulator
|
||||||
|
- CocoaPods (`sudo gem install cocoapods`)
|
||||||
|
|
||||||
|
### Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd sleepy_agent_ios
|
||||||
|
|
||||||
|
# Check status
|
||||||
|
make check
|
||||||
|
|
||||||
|
# Setup project
|
||||||
|
make setup # Installs CocoaPods
|
||||||
|
|
||||||
|
# Open in Xcode
|
||||||
|
open SleepyAgent.xcworkspace # or Package.swift
|
||||||
|
```
|
||||||
|
|
||||||
|
### To Enable LLM Inference
|
||||||
|
|
||||||
|
You need to obtain LiteRT-LM binaries:
|
||||||
|
|
||||||
|
1. **Build from source**: https://github.com/google-ai-edge/LiteRT-LM
|
||||||
|
2. **Download releases**: Check GitHub releases page
|
||||||
|
3. **Contact Google**: For prebuilt iOS binaries
|
||||||
|
|
||||||
|
Then:
|
||||||
|
1. Add LiteRT-LM headers to Xcode project
|
||||||
|
2. Link `liblitert_lm.a` static library
|
||||||
|
3. Configure bridging header in Build Settings
|
||||||
|
4. Uncomment C++ code in `LlmEngineBridge.mm`
|
||||||
|
|
||||||
|
See [BUILD.md](BUILD.md) and [LITERT_INTEGRATION.md](LITERT_INTEGRATION.md) for full instructions.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
SleepyAgent/
|
||||||
|
├── App/ # Entry point, app lifecycle
|
||||||
|
├── Core/ # Models, DI container
|
||||||
|
│ └── Models/Message.swift # Data models
|
||||||
|
├── Inference/ # LLM engine
|
||||||
|
│ ├── Bridge/ # ⭐ ObjC++ LiteRT-LM bridge
|
||||||
|
│ ├── Agent.swift # Tool calling, chat logic
|
||||||
|
│ ├── LlmEngine.swift # Swift engine interface
|
||||||
|
│ └── ...
|
||||||
|
├── Services/ # Audio, network, storage
|
||||||
|
│ ├── AudioRecorder.swift
|
||||||
|
│ ├── TtsService.swift
|
||||||
|
│ ├── WebSearchService.swift
|
||||||
|
│ └── ...
|
||||||
|
└── UI/ # SwiftUI views
|
||||||
|
├── Views/ # MainView, SettingsView, etc.
|
||||||
|
└── ViewModels/ # State management
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────┐
|
||||||
|
│ SwiftUI Views │
|
||||||
|
│ (MainView, SettingsView, ...) │
|
||||||
|
└─────────────────┬───────────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────────▼───────────────────────┐
|
||||||
|
│ Swift ViewModels │
|
||||||
|
│ (MainViewModel, SettingsViewModel) │
|
||||||
|
└─────────────────┬───────────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────────▼───────────────────────┐
|
||||||
|
│ Swift LlmEngine (Actor) │
|
||||||
|
│ (async/await, thread-safe) │
|
||||||
|
└─────────────────┬───────────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────────▼───────────────────────┐
|
||||||
|
│ Objective-C++ LlmEngineBridge │
|
||||||
|
│ (ObjC++ wrapper for LiteRT-LM) │
|
||||||
|
└─────────────────┬───────────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────────▼───────────────────────┐
|
||||||
|
│ C++ LiteRT-LM Library │
|
||||||
|
│ (Google's on-device LLM engine) │
|
||||||
|
└─────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- **[BUILD.md](BUILD.md)** - Build instructions, troubleshooting
|
||||||
|
- **[LITERT_INTEGRATION.md](LITERT_INTEGRATION.md)** - LiteRT-LM integration details
|
||||||
|
- **[LITERT_IOS_STATUS.md](LITERT_IOS_STATUS.md)** - Current state of LiteRT on iOS
|
||||||
|
- **[PROJECT_STATUS.md](PROJECT_STATUS.md)** - Overall project status
|
||||||
|
- **[FILES.md](FILES.md)** - Complete file listing
|
||||||
|
|
||||||
|
## Key Differences from Android
|
||||||
|
|
||||||
|
| Feature | Android | iOS |
|
||||||
|
|---------|---------|-----|
|
||||||
|
| UI | Jetpack Compose | SwiftUI |
|
||||||
|
| Async | Kotlin Coroutines | Swift Concurrency (async/await) |
|
||||||
|
| DI | Manual | Manual |
|
||||||
|
| Storage | DataStore + JSON | UserDefaults + FileManager |
|
||||||
|
| LLM Engine | LiteRT-LM Kotlin | LiteRT-LM C++ via bridge |
|
||||||
|
| Audio | AudioRecord | AVAudioEngine |
|
||||||
|
| Background | WorkManager | URLSession (limited) |
|
||||||
|
|
||||||
|
## Testing Without LLM
|
||||||
|
|
||||||
|
The app works in "stub mode" for UI testing:
|
||||||
|
|
||||||
|
1. Build and run
|
||||||
|
2. Type any message
|
||||||
|
3. See placeholder response explaining LiteRT-LM integration
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
To complete the iOS port:
|
||||||
|
|
||||||
|
1. Obtain LiteRT-LM iOS binaries (build or download)
|
||||||
|
2. Add to Xcode project
|
||||||
|
3. Configure build settings for C++
|
||||||
|
4. Uncomment C++ code in bridge files
|
||||||
|
5. Test with Gemma 4 E2B model
|
||||||
|
|
||||||
|
## Need Help?
|
||||||
|
|
||||||
|
- Open an issue: https://github.com/sleepyeldrazi/sleepy-agent/issues
|
||||||
|
- Check [LITERT_INTEGRATION.md](LITERT_INTEGRATION.md) for troubleshooting
|
||||||
|
- See Google's [litert-samples](https://github.com/google-ai-edge/litert-samples) for reference
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT License
|
||||||
Reference in New Issue
Block a user