sleepy_agent_ios/LITERT_INTEGRATION.md

LiteRT-LM iOS Integration - Accurate Approach

What Google Actually Uses

Based on analysis of Google's official samples:

1. Google AI Edge Gallery App

• iOS app exists on App Store: https://apps.apple.com/us/app/google-ai-edge-gallery/id6749645337
• Source code: NOT in the gallery repo (Android only)
• Implementation: Uses LiteRT-LM via C++ bridge (confirmed by GitHub issue #420 asking for iOS source)

2. MediaPipe LLM Inference (DEPRECATED but working)

• Has a working Swift API via CocoaPods
• Source: https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/ios

pod 'MediaPipeTasksGenAI'
pod 'MediaPipeTasksGenAIC'

import MediaPipeTasksGenAI

let options = LlmInference.Options(modelPath: path)
let inference = try LlmInference(options: options)
let result = try inference.generateResponse(inputText: prompt)

Status: Google deprecated this in favor of LiteRT-LM, but it's the only working Swift API currently.

3. LiteRT Compiled Model API (Vision Models)

• Uses Objective-C++ bridge pattern
• Source: https://github.com/google-ai-edge/litert-samples/tree/main/compiled_model_api/image_segmentation/ios

Pattern:

// LiteRTSegmenter.h - Objective-C header
@interface LiteRTSegmenter : NSObject
- (instancetype)initWithModelPath:(NSString *)path error:(NSError **)error;
@end

// LiteRTSegmenter.mm - Objective-C++ implementation
#import "LiteRTSegmenter.h"
#include "litert/cc/litert_compiled_model.h"

@implementation LiteRTSegmenter {
    std::optional<litert::CompiledModel> _model;
}
@end

Recommended Integration for Sleepy Agent

Option 1: Use MediaPipe Tasks (Immediate, but deprecated)

Podfile:

pod 'MediaPipeTasksGenAI', '~> 0.10.0'
pod 'MediaPipeTasksGenAIC'

Note: Limited to older model formats (.bin, not .litertlm), no Gemma 4 support likely.

Option 2: LiteRT-LM C++ Bridge (Recommended)

Based on Google's actual implementation pattern:

Files to create:

1. LlmEngineBridge.h (Objective-C header)

#import <Foundation/Foundation.h>

NS_ASSUME_NONNULL_BEGIN

@interface LlmEngineBridge : NSObject
- (nullable instancetype)initWithModelPath:(NSString *)path
                                     error:(NSError **)error;
- (NSString *)generateResponse:(NSString *)prompt
                         error:(NSError **)error;
- (void)close;
@end

NS_ASSUME_NONNULL_END

2. LlmEngineBridge.mm (Objective-C++ implementation)

#import "LlmEngineBridge.h"
#include "litert_lm/engine.h"
#include "litert_lm/conversation.h"

@interface LlmEngineBridge () {
    std::unique_ptr<litert::lm::Engine> engine;
    std::unique_ptr<litert::lm::Conversation> conversation;
}
@end

@implementation LlmEngineBridge

- (instancetype)initWithModelPath:(NSString *)path error:(NSError **)error {
    self = [super init];
    if (self) {
        auto config = litert::lm::EngineConfig{
            .model_path = [path UTF8String]
        };
        auto result = litert::lm::Engine::Create(config);
        if (!result.ok()) {
            // Set error
            return nil;
        }
        engine = std::move(*result);
        
        // Create conversation for KV cache
        auto conv_result = engine->CreateConversation({});
        if (conv_result.ok()) {
            conversation = std::move(*conv_result);
        }
    }
    return self;
}

- (NSString *)generateResponse:(NSString *)prompt error:(NSError **)error {
    if (!conversation) {
        return nil;
    }
    
    auto contents = litert::lm::Contents::FromText([prompt UTF8String]);
    auto response = conversation->SendMessage(contents);
    
    if (response.ok()) {
        return [NSString stringWithUTF8String:response->text().c_str()];
    }
    return nil;
}

- (void)close {
    conversation.reset();
    engine.reset();
}

@end

3. Bridging-Header.h

#import "LlmEngineBridge.h"

4. Swift wrapper (update existing LlmEngine.swift)

import Foundation

actor LiteRtLlmEngine: LlmEngine {
    static let shared = LiteRtLlmEngine()
    
    private var bridge: LlmEngineBridge?
    
    func loadModel(path: String) async throws {
        var error: NSError?
        bridge = LlmEngineBridge(modelPath: path, error: &error)
        if let error = error {
            throw error
        }
    }
    
    func generate(prompt: String) async throws -> String {
        guard let bridge = bridge else {
            throw LlmEngineError.modelNotLoaded
        }
        
        var error: NSError?
        let response = bridge.generateResponse(prompt, error: &error)
        
        if let error = error {
            throw error
        }
        return response ?? ""
    }
}

Build Configuration

Podfile:

# Use LiteRT C++ library
pod 'TensorFlowLiteSwift', '~> 2.16.0'

# Or manual integration with prebuilt LiteRT-LM binaries

Build Settings:

• Set "Compile Sources As" to "Objective-C++" for .mm files
• Add header search paths for LiteRT-LM includes
• Link C++ standard library

Where to Get LiteRT-LM Binaries

1. Build from source: https://github.com/google-ai-edge/LiteRT-LM
2. Releases page: Check https://github.com/google-ai-edge/LiteRT-LM/releases
3. CocoaPods: May be available in future

Current Status Summary

Approach	Availability	Gemma 4	Swift API	Recommendation
MediaPipe Tasks	✅ Now	❌ No	✅ Yes	Short-term only
LiteRT-LM C++	✅ Now	✅ Yes	❌ No	Recommended
LiteRT-LM Swift	⏳ Coming	✅ Yes	✅ Yes	Wait if possible

Key Insight

The Google AI Edge Gallery iOS app likely uses the C++ bridge approach since:

1. No Swift source code is published
2. The pattern matches their litert-samples
3. LiteRT-LM's Swift APIs are still marked "coming soon"

Next Steps

To complete Sleepy Agent iOS:

1. Download/build LiteRT-LM iOS binaries
2. Create the Objective-C++ bridge files (LlmEngineBridge.h/.mm)
3. Update the Swift LlmEngine to use the bridge
4. Configure Xcode build settings for C++
5. Test with Gemma 4 E2B model

References

• LiteRT-LM GitHub: https://github.com/google-ai-edge/LiteRT-LM
• Google's C++ Bridge Example: https://github.com/google-ai-edge/litert-samples/tree/main/compiled_model_api/image_segmentation/ios
• MediaPipe iOS Sample: https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/ios
• Gallery Issue #420: https://github.com/google-ai-edge/gallery/issues/420 (asking for iOS source)