Skip to content

Conversation

@critt
Copy link
Owner

@critt critt commented Mar 6, 2025

Background

This pull request introduces audio recording and streaming capabilities to the application, enabling real-time translation of spoken language. This feature allows users to speak into their device's microphone, have their speech transcribed and translated, and see the translated text displayed in the app.

Approach

The approach taken involves the following key components:

  1. Audio Recording: Implemented AudioSource to handle audio recording from the device's microphone.
  2. Real-time Translation: Integrated TranslationSource to manage the connection to the translation service via Socket.IO.
  3. State Management: Utilized MainViewModel to manage the application's state, including audio streaming, language selection, and translation results.
  4. UI Updates: Updated the UI in MainActivity to display the translated text and provide controls for language selection and toggling the audio stream.
  5. Speaker Management: Implemented a mechanism to switch between the user (SUBJECT) and the person they are talking to (OBJECT) using the updateSpeaker function in MainViewModel.
  6. Error Handling: Added error handling to the audio recording and streaming process.

What's in the PR?

  • Complete redesign of AudioSource to handle audio recording in a much more robust way. It now includes self-sufficient cleanup features and uses a coroutine scope provided by the caller to prevent leaks
  • Moved a lot of stuff out of TranslationRepository into TranslationSource, as that class isn't really a repository right now
  • Updated MainViewModel to use lambdas for reacting to audio data callbacks for each speaker
  • Did some much-needed state hoisting in MainActivity
  • Bug Fixes:
    • Fixed a bug where the app would crash if the user did not have permission to use the microphone
  • Testing:
    • The new features have been tested manually. Unit testing and PR test runner to come in next few PRs

Other changes

  • Dependency Management: Removed the lifecycle-livedata-ktx dependency, as it is no longer needed
  • Code Refactoring: Refactored the code to improve readability and maintainability
  • Added Javadocs to the MainViewModel , TranslationSource and AudioStreamingState classes

@critt critt marked this pull request as ready for review March 7, 2025 18:15
@critt
Copy link
Owner Author

critt commented Mar 7, 2025

mobsf still appears to be bugged. ill fix this soon.

@critt critt merged commit c2f7c40 into develop Mar 7, 2025
1 check failed
@critt critt deleted the feature/audio branch March 7, 2025 18:16
@critt critt linked an issue Mar 10, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

State Hoisting on Composables

2 participants