LKSampleHandler is only available on iOS, not on visionOS

[PaulAddy]

Is your feature request related to a problem? Please describe.
Yes, we are trying to use LiveKit in a visionOS app as we would for a standard basic iOS liveKit implementation. But, the LKSampleHandler is only available on iOS.

Describe the solution you'd like
Essentially, as currently possible on iOS, we would like to use a broadcast upload extension with LiveKit on visionOS to broadcast the entire passthrough of the user: main camera + any virtual element including Window, Volume, and Immersive space containers and their contents.

Describe alternatives you've considered
Stil considering alternatives, haven't yet found a workaround.

Additional context
n/a

[pblazej]

Hi, here's a buildable (experimental) branch enabling this feature: #856

I don't think there are many technical limitations here, but it would be great to test it on devices.

[PaulAddy]

Hi @pblazej ,

We are currently testing the branch to see if we can get a broadcast working using ReplayKit, Braodcast Upload Extension, and LiveKit in a visionOS test app. We will keep you posted.

Appreciate the quick response with a branch to test 🙏

[pblazej]

I haven't found any concrete example for visionOS in particular, so unfortunately we need to check it.

Any feedback is appreciated 🤝

[PaulAddy]

Hi @pblazej nad @hiroshihorie ,

Just an update.

We are having a lot of difficulties getting the screen share to occur. Our Broadcast Upload Extension appears to either fail immediately after starting, or never starts at all.

My tester is testing two different branches, two ways I am trying to use LiveKit:

1. The simple most basic way, minimal working example. This one is using the blaze branch you provided. See this approaches code below. With this branch, the Broadcast Upload Extension never starts, we see 0 logs in our Console.app.

2. More involved. A workaround that I am trying. It is pointing to your default package branch (not the blaze branch). We instantiate a LocalVideoTrack using its createBufferTrack. We then instantiate a BufferCapturer to which we provide it the sampleBuffer in processSampleBuffer. With this branch, the Broadcast Upload Extension tries to start, but fails immediately with an error akin to "replayd error: “Plugin … com.example.rklkworkaround.broadcast … must have pid! Extension request will fail / Failed to acquire assertion … pid: 0".

Both branches have (note: [rklkworkaround | rklkblaze] depending on which branch we are testing):

• Entitlements in all 3 targets
• • Main app:
• • • key: main-camera-access.allow
• • • key: screen-capture.include-passthrough
• • • App Groups/Item 0: group.com.example.[rklkworkaround | rklkblaze]
• • Broadcast Upload Extension:
• • • key: main-camera-access.allow
• • • key: screen-capture.include-passthrough
• • • App Groups/Item 0: group.com.example.[rklkworkaround | rklkblaze]
• • Broadcast Upload Extension UI:
• • • key: main-camera-access.allow
• • • key: screen-capture.include-passthrough
• • • App Groups/Item 0: group.com.example.[rklkworkaround | rklkblaze]
• PLIST keys
• • Main app:
• • • Privacy - Camera Usage Description
• • • NSMainCameraUsageDescription
• • • Privacy - Microphone Usage Description
• • • RTCAppGroupIdentifier: group.com.example.[rklkworkaround | rklkblaze]
• • • RTCScreenSharingExtension: com.example.[rklkworkaround | rklkblaze].broadcast
• • Broadcast Upload Extension:
• • • NSMainCameraUsageDescription
• • • Privacy - Microphone Usage Description
• • • RTCAppGroupIdentifier: group.com.example.[rklkworkaround | rklkblaze]
• • • NSExtension/NSExtensionPointIdentifier: com.apple.broadcast-services-upload
• • • NSExtension/NSExtensionPointIdentifier: $(PRODUCT_MODULE_NAME).SampleHandler
• • • NSExtension/NSExtensionPointIdentifier: RPBroadcastProcessModeSampleBuffer
• • Broadcast Upload Extension UI:
• • • NSMainCameraUsageDescription
• • • Privacy - Microphone Usage Description
• • • RTCAppGroupIdentifier: group.com.example.[rklkworkaround | rklkblaze]
• • • NSExtension/NSExtensionAttributes/NSExtensionActivationRule/NSExtensionActivationSupportsReplayKitStreaming: yes
• • • NSExtension/NSExtensionPointIdentifier: com.apple.broadcast-services-setupui
• • • NSExtension/NSExtensionPrincipalClass: $(PRODUCT_MODULE_NAME).BroadcastSetupViewController

Still wip.
Any tips will be appreciated 🙏.
If approach 2 will definitely not work, please also let me know.

Approach 1:

import OSLog
import ReplayKit
import LiveKit

let SUBSYSTEM = "com.example.rklkblaze.broadcast"

/// Custom SampleHandler that wraps LKSampleHandler and provides detailed logging
/// Since LKSampleHandler doesn't expose its lifecycle methods as `open`, we use
/// a composition approach with detailed init-time logging and periodic status updates
final class SampleHandler: LKSampleHandler, @unchecked Sendable {
    let logger = Logger(subsystem: SUBSYSTEM, category: "SampleHandler")

    // MARK: - LKSampleHandler Configuration

    /// Enable verbose logging from LKSampleHandler to see internal LiveKit activity
    /// This will log: broadcast lifecycle, sample processing, connection events, etc.
    override var enableLogging: Bool { true }
    override var verboseLogging: Bool { true }

    // MARK: - Initialization

    override init() {
        super.init()

        logger.info("\(#function) \(#line) 🚀 [SampleHandler] ========================================")
        logger.info("\(#function) \(#line) 🚀 [SampleHandler] Extension initialized with LKSampleHandler")
        logger.info("\(#function) \(#line) 🚀 [SampleHandler] ========================================")

        // Log app group credentials for debugging
        if let defaults = UserDefaults(suiteName: BroadcastStorage.suiteName) {
            let url = defaults.string(forKey: "livekit_url")
            let token = defaults.string(forKey: "livekit_token")
            logger.info("\(#function) \(#line) 🔑 [SampleHandler] AppGroup URL: \(url ?? "nil")")
            logger.info("\(#function) \(#line) 🔑 [SampleHandler] Token exists: \(token != nil)")
            if let token = token {
                logger.info("\(#function) \(#line) 🔑 [SampleHandler] Token length: \(token.count) chars")
            }
        } else {
            logger.error("\(#function) \(#line) ❌ [SampleHandler] Failed to access app group UserDefaults with suite: \(BroadcastStorage.suiteName)")
        }

        logger.info("\(#function) \(#line) ℹ️  [SampleHandler] LKSampleHandler will now handle:")
        logger.info("\(#function) \(#line) ℹ️  [SampleHandler]   • Broadcast lifecycle (start/pause/resume/finish)")
        logger.info("\(#function) \(#line) ℹ️  [SampleHandler]   • Sample buffer processing (video/audio)")
        logger.info("\(#function) \(#line) ℹ️  [SampleHandler]   • LiveKit server connection")
        logger.info("\(#function) \(#line) ℹ️  [SampleHandler]   • Stream publishing to LiveKit room")
        logger.info("\(#function) \(#line) ℹ️  [SampleHandler] Watch for [LiveKit] logs to track activity")
    }

    // MARK: - Connection Lifecycle

    /// This is the only lifecycle hook that LKSampleHandler exposes as `open`
    /// Use it to detect connection issues with LiveKit server
    override func connectionDidClose(error: Error?) {
        if let error = error {
            logger.error("\(#function) \(#line) 💥 [SampleHandler] ========================================")
            logger.error("\(#function) \(#line) 💥 [SampleHandler] LiveKit connection CLOSED with ERROR")
            logger.error("\(#function) \(#line) 💥 [SampleHandler] Error: \(error.localizedDescription)")
            logger.error("\(#function) \(#line) 💥 [SampleHandler] ========================================")
        } else {
            logger.info("\(#function) \(#line) 🔌 [SampleHandler] ========================================")
            logger.info("\(#function) \(#line) 🔌 [SampleHandler] LiveKit connection closed NORMALLY")
            logger.info("\(#function) \(#line) 🔌 [SampleHandler] ========================================")
        }

        super.connectionDidClose(error: error)
    }
}

// MARK: - Shared Storage
enum BroadcastStorage {
    static let suiteName = "group.com.example.rklkblaze"
}

Approach 2's SampleHandler:

import OSLog
import ReplayKit
import LiveKit

let SUBSYSTEM = "com.example.rklkworkaround.broadcast"

final class SampleHandler: RPBroadcastSampleHandler {
    let logger = Logger(subsystem: SUBSYSTEM, category: "SampleHandler")
    private var frameCount = 0

    // LiveKit components
    private var room: Room?
    private var videoTrack: LocalVideoTrack?
    private var capturer: BufferCapturer?

    // Connection state
    private var isConnected = false

    // MARK: - Lifecycle Logging

    override init() {
        super.init()
        logger.info("\(#function) \(#line) 🚀 [SampleHandler] Extension initialized")
        logger.info("\(#function) \(#line) 📱 [SampleHandler] Process: \(ProcessInfo.processInfo.processName)")
        logger.info("\(#function) \(#line) 🔢 [SampleHandler] PID: \(ProcessInfo.processInfo.processIdentifier)")
    }

    override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
        logger.info("\(#function) \(#line) 🎬 [SampleHandler] broadcastStarted called")
        logger.info("\(#function) \(#line) 📦 [SampleHandler] setupInfo: \(String(describing: setupInfo))")

        // Get LiveKit credentials from app group
        logger.info("\(#function) \(#line) 🔍 [SampleHandler] Attempting to access app group: \(BroadcastStorage.suiteName)")

        guard let defaults = UserDefaults(suiteName: BroadcastStorage.suiteName) else {
            logger.error("\(#function) \(#line) ❌ [SampleHandler] Failed to create UserDefaults with app group suite")
            finishBroadcastWithError(NSError(
                domain: RPRecordingErrorDomain,
                code: RPRecordingErrorCode.failed.rawValue,
                userInfo: [NSLocalizedDescriptionKey: "Cannot access app group storage."]
            ))
            return
        }

        logger.info("\(#function) \(#line) ✅ [SampleHandler] UserDefaults accessed successfully")

        let urlString = defaults.string(forKey: "livekit_url")
        let token = defaults.string(forKey: "livekit_token")

        logger.info("\(#function) \(#line) 🔍 [SampleHandler] Retrieved URL: \(urlString ?? "nil"), hasToken: \(token != nil)")

        guard let urlString = urlString, let token = token else {
            logger.error("\(#function) \(#line) ❌ [SampleHandler] LiveKit credentials not found in app group")
            finishBroadcastWithError(NSError(
                domain: RPRecordingErrorDomain,
                code: RPRecordingErrorCode.failed.rawValue,
                userInfo: [NSLocalizedDescriptionKey: "LiveKit credentials not found. Please configure in the main app."]
            ))
            return
        }

        logger.info("\(#function) \(#line) 🔑 [SampleHandler] Got credentials - URL: \(urlString), token length: \(token.count)")

        // Connect to LiveKit asynchronously
        Task {
            await connectToLiveKit(url: urlString, token: token)
        }

        super.broadcastStarted(withSetupInfo: setupInfo)
        logger.info("\(#function) \(#line) ✅ [SampleHandler] super.broadcastStarted completed")
    }

    private func connectToLiveKit(url: String, token: String) async {
        do {
            logger.info("\(#function) \(#line) 🔌 [SampleHandler] Connecting to LiveKit...")
            logger.info("\(#function) \(#line) 🌐 [SampleHandler] URL: \(url)")

            // Create room
            logger.info("\(#function) \(#line) 🏠 [SampleHandler] Creating Room instance...")
            let room = Room()
            self.room = room
            logger.info("\(#function) \(#line) ✅ [SampleHandler] Room instance created")

            // Create buffer track for ReplayKit frames
            logger.info("\(#function) \(#line) 🎥 [SampleHandler] Creating buffer track...")
            let options = BufferCaptureOptions(
                dimensions: .h1080_169,  // 1920x1080
                fps: 30
            )

            let videoTrack = LocalVideoTrack.createBufferTrack(
                name: "visionos-broadcast",
                source: .screenShareVideo,
                options: options
            )

            self.videoTrack = videoTrack
            logger.info("\(#function) \(#line) ✅ [SampleHandler] Buffer track created: \(videoTrack.name)")

            // Get the capturer so we can push frames to it
            if let capturer = videoTrack.capturer as? BufferCapturer {
                self.capturer = capturer
                logger.info("\(#function) \(#line) ✅ [SampleHandler] BufferCapturer obtained")
            } else {
                logger.error("\(#function) \(#line) ❌ [SampleHandler] Failed to get BufferCapturer")
            }

            // Connect to room
            logger.info("\(#function) \(#line) 🔗 [SampleHandler] Attempting room.connect()...")
            try await room.connect(url: url, token: token)
            logger.info("\(#function) \(#line) ✅ [SampleHandler] Connected to LiveKit room")
            logger.info("\(#function) \(#line) 📊 [SampleHandler] Room SID: \(room.sid ?? "nil")")
            logger.info("\(#function) \(#line) 👤 [SampleHandler] Participant: \(room.localParticipant.identity ?? "nil")")

            // Publish video track
            logger.info("\(#function) \(#line) 📤 [SampleHandler] Publishing video track...")
            try await room.localParticipant.publish(videoTrack: videoTrack)
            logger.info("\(#function) \(#line) ✅ [SampleHandler] Published video track successfully")

            self.isConnected = true
            logger.info("\(#function) \(#line) 🎉 [SampleHandler] LiveKit setup complete - ready to broadcast frames")

        } catch {
            logger.error("\(#function) \(#line) ❌ [SampleHandler] Failed to connect to LiveKit: \(error.localizedDescription)")
            logger.error("\(#function) \(#line) 🔍 [SampleHandler] Error details: \(String(describing: error))")
            finishBroadcastWithError(NSError(
                domain: RPRecordingErrorDomain,
                code: RPRecordingErrorCode.failed.rawValue,
                userInfo: [NSLocalizedDescriptionKey: "Failed to connect to LiveKit: \(error.localizedDescription)"]
            ))
        }
    }

    override func broadcastPaused() {
        logger.warning("\(#function) \(#line) ⏸️ [SampleHandler] Broadcast paused")
        super.broadcastPaused()
    }

    override func broadcastResumed() {
        logger.info("\(#function) \(#line) ▶️ [SampleHandler] Broadcast resumed")
        super.broadcastResumed()
    }

    override func broadcastFinished() {
        logger.info("\(#function) \(#line) 🛑 [SampleHandler] Broadcast finished")

        // Disconnect from LiveKit
        Task {
            // Unpublish all tracks
            await self.room?.localParticipant.unpublishAll()

            // Disconnect from room
            await self.room?.disconnect()
            logger.info("\(#function) \(#line) ✅ [SampleHandler] Disconnected from LiveKit")
        }

        super.broadcastFinished()
    }

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
        self.frameCount += 1

        // Log first few frames with detailed info
        if self.frameCount <= 5 {
            logger.info("\(#function) \(#line) 🎞️ [SampleHandler] Frame #\(self.frameCount): type=\(sampleBufferType.rawValue), isConnected=\(isConnected), hasCapturer=\(self.capturer != nil)")
        }

        // Send video frames to LiveKit
        if sampleBufferType == .video {
            if isConnected, let capturer = self.capturer {
                capturer.capture(sampleBuffer)

                // Log every 30th frame to monitor progress (about once per second at 30fps)
                if self.frameCount % 30 == 0 {
                    logger.info("\(#function) \(#line) 📹 [SampleHandler] Sent frame \(self.frameCount) to LiveKit")
                }
            } else {
                // Log why we're not capturing
                if self.frameCount <= 10 {
                    logger.warning("\(#function) \(#line) ⚠️ [SampleHandler] Not capturing frame \(self.frameCount): isConnected=\(isConnected), hasCapturer=\(self.capturer != nil)")
                }
            }
        }

        super.processSampleBuffer(sampleBuffer, with: sampleBufferType)
    }

    override func finishBroadcastWithError(_ error: Error) {
        logger.error("\(#function) \(#line) 💥 [SampleHandler] Broadcast finished with error: \(error.localizedDescription)")

        // Cleanup
        Task {
            await self.room?.disconnect()
        }

        super.finishBroadcastWithError(error)
    }
}

// MARK: - Shared Storage
enum BroadcastStorage {
    static let suiteName = "group.com.example.rklkworkaround"
}

[pblazej]

@PaulAddy let's focus on the 1st approach - avoid connect/publish code in the sample handler whatsoever.

Some tips for debugging:

1. Remove the bundle ID overrides from the plist, and just use .broadcast suffix similarly to this example
2. Make sure the debugger is attached to the extension, not to the app:

3. You should be able to step into the extension code like this (and get the logs):

4. If it fails silently, this part is the most vulnerable:

       guard let socketPath = BroadcastBundleInfo.socketPath else {
            os_log("Bundle settings improperly configured for screen capture", log: log, type: .error)
            return
        }

If that's the case, go back to 1.

I hope this is some trivial misconfiguration rather than an actual issue atm.

[pblazej]

@PaulAddy you can also use iOS for debugging, then move to visionOS.

[PaulAddy]

Hi @pblazej

you can also use iOS for debugging, then move to visionOS.

Yep, proved it works on iOS, works nicely, with barely anything in the SampleHanlder, like this:

final class SampleHandler: LKSampleHandler, @unchecked Sendable {

    // Optional but useful for debugging while you get it working:
    override var enableLogging: Bool { true }

}

I hope this is some trivial misconfiguration rather than an actual issue atm.

Thanks for the tips 🙏
Starting anew in a fresh project to make sure I stick to the bare minimum.
Will keep you posted..