Created the basic structure for OGG-FLAC streaming sink with:
- OggFlacStreamHandle for HTTP client subscriptions
- OggFlacClientStream implementing AsyncRead
- StreamingOggFlacSinkLogic with metadata tracking
- TrackBoundary detection for OGG chaining (TODO)
Structure follows StreamingFlacSink pattern but designed for:
1. OGG container wrapping around FLAC frames
2. OGG chaining on TrackBoundary markers
3. Vorbis Comments metadata updates per track
LIMITATION: Cannot compile/test due to missing system dependencies
(alsa-sys, libsoxr-sys). The code structure is complete but encoding
logic needs to be implemented and tested on a machine with proper deps.
Next steps:
- Implement chunk_to_pcm_bytes conversion
- Implement OGG page wrapper task
- Implement OGG chaining logic
- Test on system with alsa-dev installed
Created initial structure for OGG-FLAC streaming encoder:
- OGG page writer with CRC32 calculation
- Vorbis Comment metadata support
- 100% streaming architecture (no seek operations)
This is work-in-progress. The OGG wrapping task needs to be
implemented to actually wrap FLAC frames in OGG pages.
Related to the need for streaming FLAC with embedded metadata.
Note: Vorbis Comments in OGG are static once written. For dynamic
metadata updates, use JSON endpoint or implement OGG chaining.
VLC cannot decode FLAC streams with embedded ICY metadata because
the ICY blocks break the FLAC decoder. Split into two endpoints:
- /test/stream: Pure FLAC (for VLC and standard FLAC players)
- /test/stream-icy: FLAC + ICY metadata (for ICY-aware clients)
This allows:
- VLC to play audio correctly using pure FLAC
- ICY-aware clients to receive metadata updates
- Metadata endpoint remains available for JSON queries
Fixes the "no audio" issue where VLC would connect, receive the
FLAC header with ICY metadata blocks, fail to decode, and disconnect.
Changed stream_handler to always serve ICY-wrapped FLAC instead of
checking for the Icy-MetaData header. This ensures all clients
(including VLC) receive metadata updates.
Changes:
- Removed conditional ICY/pure FLAC logic
- Always use subscribe_icy() for all connections
- Added standard ICY headers (icy-genre, icy-pub)
- Updated documentation to reflect default ICY mode
This allows clients to see "Now Playing" information without needing
to send specific HTTP headers.
Removed references to non-existent VLC options:
- --icy-metadata (doesn't exist)
- --http-continuous (not needed)
VLC automatically sends the "Icy-MetaData: 1" HTTP header when
connecting to HTTP audio streams, and the server responds with
ICY metadata blocks. No special VLC flags are needed.
Also fixed the stream URL in help text (/stream → /test/stream).
Changed from 4096 to 128 messages (~10s buffer instead of ~5min).
The large buffer was causing metadata drift: clients could be hearing
audio 5 minutes behind the metadata endpoint and ICY metadata updates.
With TimerNode pacing the stream to real-time, we only need a small
buffer for network jitter. This keeps metadata properly synchronized
with the actual audio being played.
The streaming FLAC implementation was experiencing severe lag warnings
(clients skipping 700-2200 messages) because:
1. The broadcast channel capacity (512) was too small for network backpressure
2. The pipeline had no rate limiting, sending data faster than real-time
Changes:
- Increased BROADCAST_CAPACITY from 512 to 4096 (~5min buffer)
- Added TimerNode (3s lead time) to stream_block example pipeline
- Pipeline now: RadioParadiseStreamSource → TimerNode → StreamingFlacSink
This ensures data flows at real-time playback speed with sufficient
buffering for network jitter, eliminating client lag warnings.
The broadcast channel capacity was too small (512) causing clients
to lag behind the encoder and drop thousands of messages, resulting
in choppy playback. Increased to 4096 to provide ~5 minutes of
buffer at typical FLAC streaming rates (~12-15 chunks/sec at 8KB each).
This resolves the "FLAC client lagged, skipped N messages" warnings.
Clone header value before modifying self to avoid holding RwLockReadGuard
while mutating buffer and state fields in both FlacClientStream and
IcyClientStream poll_read() implementations.
- Cache first FLAC chunk containing 'fLaC' magic bytes in StreamHandle
- Send cached header to each new subscriber before streaming data
- Add FlacStreamState enum to track header vs streaming state
- Increase BROADCAST_CAPACITY from 64 to 512 to reduce lag warnings
- Fixes 'this doesn't look like a flac stream' error in VLC
- Fix import paths to use public pmoaudio API instead of private modules
- Change AudioError::ConfigurationError to ProcessingError
- Use Node::new_with_input() instead of non-existent Node::new()
- Fix borrow checker issues in IcyClientStream::poll_read()
- Use flatten() on metadata getters to unwrap Result<Option<T>>
- Fix chunk.get_sample_rate() to chunk.sample_rate()
- Remove get_album_artist() call (not in TrackMetadata trait)
- Update pmoparadise Cargo.toml to enable pmoserver feature for axum
- Simplify example init_logging() call to match new pmoserver API
Enhances ICY metadata streaming to include cover artwork URLs, enabling
media players to display album art while streaming.
Changes:
- Add cover_pk field to MetadataSnapshot (cache primary key)
- Extract cover_pk in update_metadata() alongside cover_url
- Format ICY metadata with StreamUrl field pointing to cover image:
* If cover_pk exists: /covers/image/{pk}/256 (local cache, 256px)
* Fallback to cover_url if no local cache (external URL)
- Use relative URLs for compatibility with same-origin streaming
ICY format example:
StreamTitle='AC/DC - Highway to Hell';StreamUrl='/covers/image/abc123/256';
This works seamlessly with pmocovers which serves images at:
GET /covers/image/{pk} - Original WebP
GET /covers/image/{pk}/256 - 256px variant (used in ICY)
Relative URLs are resolved correctly by VLC and other ICY-compatible players
when streaming from the same server that serves covers.
Creates a new example demonstrating StreamingFlacSink usage with pmoserver
for real-world HTTP streaming testing with media players like VLC.
Features:
- Uses pmoserver instead of raw Axum for realistic testing
- Streams a single Radio Paradise block over HTTP
- Supports both pure FLAC and ICY metadata modes
- Provides /test/stream endpoint for streaming
- Provides /test/metadata endpoint for JSON metadata queries
- Includes health check endpoint
Usage:
cargo run --example stream_block --features full -- <channel_id>
Testing with VLC:
# Pure FLAC mode
vlc http://localhost:8080/test/stream
# ICY metadata mode (Now Playing)
vlc --http-continuous --icy-metadata http://localhost:8080/test/stream
Dependencies:
- Requires pmoserver for HTTP server
- Requires StreamingFlacSink from pmoaudio-ext (http-stream feature)
- Integrated with full feature set (pmoaudio + pmoaudio-ext + pmoserver)
Changed is_valid_pk() from sync to async to properly wait for file
creation without blocking. This is a breaking change but we're in
active development.
Changes:
- is_valid_pk() signature: fn -> async fn
- Replaced std:🧵:sleep with tokio::time::sleep
- Updated all 6 call sites in pmoplaylist to add .await:
- WriteHandle::push()
- WriteHandle::push_set()
- ReadHandle::pop()
- ReadHandle::peek()
- ReadHandle::remaining()
- ReadHandle::get_all()
Benefits:
- Non-blocking wait for file creation during ingestion
- More idiomatic async Rust code
- Better integration with tokio runtime
When add_from_reader() returns after prebuffering, the file may not
exist on disk yet due to tokio::spawn() scheduling. This caused
"Cache entry not found" errors when playlist tried to validate the pk.
Solution:
- If DB entry exists but file doesn't, wait up to 1 second for file creation
- This handles the race condition between prebuffer completion and
File::create() in the background task
- Deterministic and robust: either file exists or we timeout with error
The fix preserves the progressive caching design while ensuring
validation is deterministic.
Test: Verified no "Cache entry not found" errors with clean cache.
This commit fixes two critical issues that prevented covers from being
cached for tracks beyond the first one in Radio Paradise blocks:
1. FlacCacheSink Phase 3 metadata loss:
- When TrackBoundary for track N+1 was received during Phase 3
of track N, the metadata was discarded
- Main loop would then wait for a NEW TrackBoundary that never came
- Solution: Store metadata in next_track_metadata variable and reuse
it in next iteration
- Added wait_for_first_audio_chunk() for when metadata is pre-loaded
2. RadioParadiseStreamSource not sending subsequent TrackBoundaries:
- Code was only checking elapsed_ms >= song.elapsed in loop
- Added debug logging to track TrackBoundary sending
- Improved comments explaining first song special handling
Test results:
- Successfully cached covers for 4 consecutive tracks
- Verified with test showing "Successfully cached cover" for each track
- Cover cache directory contains 4 .webp files with complete markers
Files modified:
- pmoaudio-ext/src/sinks/flac_cache_sink.rs
- pmoparadise/src/radio_paradise_stream_source.rs
Corrige le bug critique qui empêchait la mise en cache des covers pour
les fichiers courts (jingles, etc.) :
Le problème :
- Quand EndOfStream arrivait AVANT la fin du prebuffer, le code retournait
immédiatement sans copier les métadonnées ni cacher les covers
- Cela affectait particulièrement les fichiers courts (jingles) où le
prebuffer de 512KB n'était pas atteint avant la fin du fichier
La solution :
- Lorsque EndOfStream est reçu pendant le prebuffer, on ferme le pump mais
on CONTINUE à attendre que cache_future se termine pour obtenir le pk
- Une fois le pk obtenu, on copie les métadonnées et on cache les covers
normalement avant de retourner
- Utilise un flag end_of_stream_received et une Option<track_tx> pour gérer
le cas où track_tx est déjà fermé
Test validé :
✓ Les covers sont bien cachées même pour les fichiers courts
✓ Fichier de cover présent : 36e3e134b8de74e6c16f202e3b3b543d.orig.webp (38K)
✓ Logs montrent : "Successfully cached cover for pk ... with cover pk ..."
Corrige un bug critique qui empêchait la mise en cache des covers :
- RadioParadiseStreamSource envoie maintenant un TrackBoundary pour la première
song IMMÉDIATEMENT après le TopZeroSync, AVANT le premier chunk audio
- Cela garantit que FlacCacheSink reçoit les métadonnées (incluant cover_url)
dès le début du traitement
Le problème :
- Avant, le TrackBoundary n'était envoyé que quand elapsed_ms >= song.elapsed
- Pour la première song avec elapsed > 0, le TrackBoundary arrivait APRÈS
plusieurs chunks audio
- FlacCacheSink recevait le premier chunk SANS métadonnées
- Quand le prebuffer se terminait, track_metadata était None
- Les métadonnées (incluant cover_url) n'étaient jamais copiées dans le cache
- Résultat : aucune cover n'était mise en cache
La solution :
- Envoyer explicitement un TrackBoundary pour la première song avant de
commencer la boucle de chunks
- Les songs suivantes continuent d'être gérées par la logique existante
Test validé :
✓ RadioParadiseStreamSource configure cover_url correctement
✓ FlacCacheSink reçoit cover_url
✓ Les covers sont téléchargées et mises en cache
✓ Les logs montrent : "Successfully cached cover for pk ... with cover pk ..."
Corrige un bug critique de race condition dans RadioParadiseStreamSource :
- Rend song_to_metadata() async et attend que toutes les métadonnées soient configurées
- Supprime le tokio::spawn() qui causait un retour prématuré des métadonnées
- Garantit que cover_url est disponible quand FlacCacheSink lit les métadonnées
- Ajoute des logs de debug pour tracer la configuration des métadonnées
- Remplace eprintln! par tracing::warn! pour une meilleure cohérence
Corrige également un warning de compilation :
- Retire le `mut` inutile sur la variable `writer` dans play_and_cache.rs
Le problème : song_to_metadata() retournait les métadonnées avant que
la task asynchrone ne finisse de les configurer, ce qui causait un
cover_url manquant quand FlacCacheSink essayait de cacher les covers.
Améliore la gestion du cache des covers dans FlacCacheSink :
- Remplace les avertissements génériques par des logs détaillés (debug/info/warn)
- Corrige la gestion des erreurs en retirant le `let _ =` qui ignorait les résultats
- Ajoute des logs de debug pour tracer le processus de mise en cache des covers
- Améliore la gestion des erreurs avec des messages plus informatifs
Corrige la playlist de l'exemple play_and_cache :
- Remplace create_persistent_playlist par get_write_handle pour créer une playlist éphémère
- Une playlist persistante n'est pas nécessaire pour cet exemple de démonstration
This script calculates the Maximum delta / Mean delta ratio to detect
clicks in FLAC files. A ratio > 10 indicates audio discontinuities caused
by buffer underruns during encoding.
Usage: ./check_audio_quality.sh [cache_directory]
The script helps verify that the PCM buffer fix (256 instead of 8) has
eliminated the clicks.
- detect_clicks.sh: Batch analysis of all cached FLAC files
- analyze_flac.sh: Detailed analysis of a single FLAC file
These tools help verify audio quality and detect encoding issues
like clicks caused by buffer underruns.
The small buffer (8) was causing the pump to block frequently when the FLAC
encoder was slow to consume data. This created micro-pauses in the PCM stream
that resulted in audible clicks in the encoded FLAC files.
With a larger buffer (256), the pump can continue sending data without blocking,
ensuring continuous audio flow to the encoder and eliminating the clicks.
Previous fix consumed the TrackBoundary with drain_until_track_boundary(),
preventing the next track from being processed correctly. This caused
audio to stop after the first cached track.
Solution: Use a pump_closed flag instead of draining. When the pump
closes early (cache hit), set the flag and ignore subsequent chunks
until TrackBoundary. The TrackBoundary is then handled normally by
the existing code, allowing proper continuation to the next track.
This preserves the block structure and allows all tracks in a block
to be processed correctly, whether cached or not.
When a file was already in cache, add_from_reader() would return
immediately after reading only 1024 bytes to compute the pk. This
closed the flac_stream and pcm_tx, causing the pump to terminate
normally. However, the dispatcher treated track_tx.send() failure
as a fatal error, even though the pump had completed successfully.
Changes:
- In phase 3 post-prebuffer, when track_tx.send() fails, wait for
pump to complete and check its result
- If pump returned Ok(), drain remaining segments until TrackBoundary
- If pump returned Err(), propagate the error
- This allows graceful handling of cache hits while preserving error
detection for genuine pump failures
Fixes the "Pump task died" error when relaunching play_and_cache
with existing cached files.
Apply same architecture as FlacCacheSink to prevent file truncation
when external readers access files during encoding.
Changes:
1. Add pump_track_segments_from_channel() for parallel pump tasks
2. Refactor process() to use dispatcher + tokio::select! pattern
3. Create .complete marker after flush/wait to signal file is ready
4. Allow multiple tracks to encode in parallel (pump continues in background)
This ensures FlacFileSink is cache progressive compliant, meaning external
code can safely read output files while they're being written without
risk of truncation.
Problem: The dispatcher was placed AFTER the prebuffer await, causing a deadlock:
- Pump waits for data on track_rx
- Cache waits for pump to produce PCM data
- Code awaits cache completion before reaching dispatcher
- Dispatcher never runs → pump never receives data → deadlock
Solution: Use tokio::select! to dispatch segments in parallel with prebuffer wait
Architecture now has 3 phases:
1. Phase 1: Dispatch chunks + await prebuffer (in parallel via select!)
2. Phase 2: Copy metadata + push to playlist (after prebuffer complete)
3. Phase 3: Continue dispatching until TrackBoundary
This fixes the "sans musique" blocking issue where the system would freeze
waiting for prebuffer that could never complete.
Problem: When TrackBoundary arrived, the pump was awaited before continuing,
causing file truncation when pcm_tx was dropped while data was still buffering.
Solution: Allow multiple pump tasks to run in parallel:
- Create dedicated channel (track_tx/track_rx) for each track's pump
- Main loop reads from rx and dispatches segments to current pump via track_tx
- When TrackBoundary arrives: drop track_tx (signals pump to finish) and immediately start new pump
- Old pump continues writing in background until all data is flushed
This prevents truncation in progressive cache scenario (radio streaming).
Changes in flac_cache_sink.rs:
- Replace pump_track_segments_owned() with pump_track_segments_from_channel()
- Remove rx ownership passing - each pump gets its own channel
- Dispatcher loop reads rx and forwards to active pump
- No await on pump completion - let it finish in background
Changes:
- Add TimerNode (pmoaudio/src/nodes/timer_node.rs): Rate-limits audio chunk flow based on timestamps with configurable max_lead_time
- Integrate TimerNode into play_and_cache.rs pipeline: PlaylistSource → TimerNode (3s pacing) → AudioSink
- Improve EOF retry in playlist_source.rs: Wait for prebuffer (512KB) before decoding, retry on temporary EOF with 200ms delay
- Export TimerNode in pmoaudio lib.rs and nodes/mod.rs
Known issue: Cache files may still be truncated when TrackBoundary arrives before pump completes flushing.
This requires allowing parallel write tasks as suggested.
Problem:
- PlaylistSource reads cached files faster than FlacCacheSink writes them
- FLAC decoder encounters EOF and stops playback prematurely
- First track doesn't play completely (stops at prebuffer point ~600ms)
- Needed to differentiate:
* Temporary EOF: file still being written (wait and retry)
* Real EOF: file completely written (stop decoding)
Solution:
1. Added Cache::is_download_complete() method (pmocache/src/cache.rs:735)
- Checks for existence of completion marker (.complete file)
- Marker created only when file is fully written and closed
- Fast synchronous check (no async overhead)
2. Modified decode_and_emit_track() (playlist_source.rs:337)
- On EOF: check if completion marker exists
- If no marker: file still being written → wait 50ms and retry read
- If marker exists: file complete → finish decoding
- Reduced wait from 100ms to 50ms for better responsiveness
Benefits:
✅ First track now plays completely (not just prebuffer portion)
✅ Progressive caching still works (playback starts at ~600ms)
✅ Proper EOF handling (no premature stops)
✅ Efficient polling (50ms retry interval)
✅ Works for both fresh downloads and cached files
Tested:
- Fresh download: EOF retries visible in logs every ~50ms
- File plays until completion marker created
- No premature track termination
Related to previous optimization (commit d8594e7) that made
prebuffer→playlist push immediate (76ms instead of 19s).
Problem:
- tokio::join!() waited for both cache_future AND pump_future to complete
- cache_future returned after prebuffer (~530ms)
- pump_future read entire first track (~19s)
- Track only pushed to playlist after both finished → 19s delay
Solution (Solution A from OPTIMIZATION_PREBUFFER_TO_PLAYLIST.md):
- Created pump_track_segments_owned() that takes ownership of rx and returns it
- Spawned pump in tokio::spawn to run independently
- Wait for cache_future alone → push to playlist immediately
- Wait for pump_handle later to recover rx for next track
Results (tested with play_and_cache --null-audio):
Before:
- Prebuffer → playlist: ~19s
- Prebuffer → playback: ~19.5s
After:
- Prebuffer → playlist: ~24ms
- Prebuffer → playback: ~76ms
- Improvement: 99.6% (250x faster!)
Target was <1s, achieved 76ms (13x better than target!)
Changes:
- Added pump_track_segments_owned() in flac_cache_sink.rs:516
- Modified FlacCacheSinkLogic::process() to use tokio::spawn pattern
- Added timing logs (INFO level) for prebuffer and playlist push
- rx ownership properly managed: moved to pump, returned, recovered
Tests passed:
✅ Prebuffer completes in ~530ms (512KB downloaded)
✅ Track pushed to playlist in ~24ms after prebuffer
✅ Playback starts in ~76ms after prebuffer
✅ rx properly recovered for next tracks
✅ No panics or deadlocks
Document détaillé pour réduire le délai de 19s à 1s en pushant à la
playlist immédiatement après le prebuffer, sans attendre pump_future.
Contient:
- Analyse du problème actuel (tokio::join! bloquant)
- 3 solutions possibles avec avantages/inconvénients
- Plan d'implémentation détaillé avec code complet
- Guide de test et validation
- Debugging tips et tests de régression
Ce document permet de reprendre l'optimisation dans une nouvelle session
avec tout le contexte nécessaire.
Added comprehensive logging to FlacCacheSink::process():
- Process start
- Waiting for/receiving first audio chunk
- FLAC encoder creation
- Cache ingestion and pump parallel execution
- tokio::join! completion
- Track added to cache confirmation
Testing results show PROGRESSIVE CACHING WORKS:
✅ Prebuffer reached in 0.6 seconds
✅ Track added to cache with pk
✅ Download pipeline completes successfully
✅ Playlist receives track
✅ Playback starts
Current timing:
- t=0.6s: Prebuffer complete (512KB)
- t=3.6s: Track added to playlist (after pump completes)
- t=4.5s: Playback starts
The 3s delay is because tokio::join! waits for BOTH futures:
- cache_future (returns after prebuffer ~0.6s)
- pump_future (pumps entire first track ~3s)
For true 1-2s startup, would need to refactor to push to playlist
immediately after prebuffer, without waiting for pump to complete.
Changes:
1. pmocache/cache_trait.rs - Fixed is_valid_pk() logic:
- Accept files WITH completion markers (complete downloads)
- Accept files WITHOUT markers but recent (< 60s) (downloads in progress)
- Reject files WITHOUT markers and old (>= 60s) (failed downloads)
This preserves progressive caching: files are valid as soon as prebuffer
completes, without waiting for completion marker.
2. pmoupnp/cache_registry.rs - Added compatibility layer:
- Re-exports get_audio_cache/get_cover_cache from singletons
- Provides build_audio_url/build_cover_url for pmosource
- Uses PMO_SERVER_URL env var for base URL
3. pmoupnp/lib.rs - Added cache_registry module to public API
This fixes "Cache entry not found" errors while maintaining progressive
caching functionality for play_and_cache example.
Added heuristic to accept files modified within last 60 seconds,
which should catch files currently being downloaded.
Also added debug logging to diagnose why validation fails.
Still debugging - need to test with logs to see what's happening.
Critical Bug Fixed:
Files between 512 and 1024 bytes (e.g., small images) were incorrectly
handled. The condition `header.len() > 512` would skip the first 512
bytes even for small files, using only a tiny portion for pk calculation.
Example Bug:
- PNG image of 700 bytes
- header.len() = 700
- 700 > 512 = TRUE
- Used &header[512..] = only 188 bytes (octets 512-700)
- SKIPPED important PNG header and image data!
Solution:
Changed condition from `> 512` to `>= 1024`:
- Files < 1024 bytes → use ALL content (correct for images)
- Files >= 1024 bytes → skip first 512 bytes (correct for FLAC)
Impact:
- pmocovers cache now works correctly with small images
- No more data loss for files between 512-1024 bytes
- FLAC behavior unchanged (still skips header correctly)
Problem Analysis:
- All FLAC files had the same pk (071c5713d5cf485ca688832207bef0f9)
- Root cause: read() can return < 1024 bytes on first call
- If read returned only 400 bytes:
* header.len() = 400
* 400 > 512 = false
* Used header[..] (first 400 bytes = FLAC header)
* All FLAC files have identical headers → same pk!
Solution:
- Added read_exact_or_eof() that loops until 1024 bytes read (or EOF)
- Guarantees we skip FLAC header and use actual audio content
- Works for small files (< 512 bytes) and large files (>= 1024 bytes)
Additional Feature:
- Added AudioSink::with_null_output() for testing without audio device
- Added --null-audio flag to play_and_cache example
- Allows testing in containerized environments
Changes:
1. pmocache/src/download.rs: Added read_exact_or_eof()
2. pmocache/src/cache.rs: Use read_exact_or_eof() for pk calculation
3. pmoaudio/src/nodes/audio_sink.rs: Added null output mode
4. pmoparadise/examples/play_and_cache.rs: Added --null-audio flag
Test Results:
- New pk: 83702c1cbca72074ebf7c123336786ea (was 071c...)
- Null audio output works correctly
- Ready for full testing
