SpotMicroESP32-Leika/FILESYSTEM_CHUNKED_TRANSFER.md

Chunked Filesystem Transfer System

This system enables chunked file uploads and downloads between the client (web browser) and ESP32 over WebSocket, overcoming the 1KB per stream limitation.

Architecture

Protocol Messages (Protobuf)

The system uses Protocol Buffers for efficient binary messaging with the following operations:

1. Delete: Remove files or directories
2. Mkdir: Create directories
3. List: List files and directories
4. Download: Transfer files from ESP32 to client (chunked)
5. Upload: Transfer files from client to ESP32 (chunked)
6. Cancel: Cancel ongoing transfers

Chunked Transfer Flow

Download (ESP32 → Client)

1. Client sends FSDownloadStartRequest with file path
2. ESP32 responds with FSDownloadStartResponse containing:
   • Transfer ID
   • File size
   • Chunk size (1024 bytes max)
   • Total chunks
3. Client requests chunks sequentially using FSDownloadChunkRequest
4. ESP32 sends each chunk via FSDownloadChunkResponse
5. Transfer completes when last chunk is received

Upload (Client → ESP32)

1. Client sends FSUploadStartRequest with destination path and file size
2. ESP32 responds with FSUploadStartResponse containing:
   • Transfer ID
   • Max chunk size (1024 bytes)
3. Client sends chunks sequentially using FSUploadChunkRequest
4. ESP32 responds with FSUploadChunkResponse after each chunk
5. Transfer completes when last chunk is written

Implementation Details

ESP32 Side

Files:

• esp32/include/filesystem_ws.h - Header file with handler class definition
• esp32/src/filesystem_ws.cpp - Implementation of filesystem operations
• esp32/include/communication/proto_helpers.h - Message traits for protobuf

Key Features:

• Transfer state management with automatic cleanup
• Timeout handling (30 seconds of inactivity)
• Recursive directory deletion
• File integrity verification

Integration: You need to integrate the filesystem handlers into your WebSocket message handling. In your main WebSocket handler, add:

#include <filesystem_ws.h>

// In your correlation request handler:
void handleCorrelationRequest(const socket_message_CorrelationRequest& request, int clientId) {
    socket_message_CorrelationResponse response;

    // ... existing handlers ...

    if (request.which_request == socket_message_CorrelationRequest_fs_delete_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_delete_response_tag;
        response.response.fs_delete_response =
            FileSystemWS::fsHandler.handleDelete(request.request.fs_delete_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_mkdir_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_mkdir_response_tag;
        response.response.fs_mkdir_response =
            FileSystemWS::fsHandler.handleMkdir(request.request.fs_mkdir_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_list_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_list_response_tag;
        response.response.fs_list_response =
            FileSystemWS::fsHandler.handleList(request.request.fs_list_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_download_start_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_download_start_response_tag;
        response.response.fs_download_start_response =
            FileSystemWS::fsHandler.handleDownloadStart(request.request.fs_download_start_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_download_chunk_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_download_chunk_response_tag;
        response.response.fs_download_chunk_response =
            FileSystemWS::fsHandler.handleDownloadChunk(request.request.fs_download_chunk_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_upload_start_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_upload_start_response_tag;
        response.response.fs_upload_start_response =
            FileSystemWS::fsHandler.handleUploadStart(request.request.fs_upload_start_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_upload_chunk_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_upload_chunk_response_tag;
        response.response.fs_upload_chunk_response =
            FileSystemWS::fsHandler.handleUploadChunk(request.request.fs_upload_chunk_request);
    }
    else if (request.which_request == socket_message_CorrelationRequest_fs_cancel_transfer_request_tag) {
        response.which_response = socket_message_CorrelationResponse_fs_cancel_transfer_response_tag;
        response.response.fs_cancel_transfer_response =
            FileSystemWS::fsHandler.handleCancelTransfer(request.request.fs_cancel_transfer_request);
    }

    // Send response back to client
    sendCorrelationResponse(response, clientId);
}

// Optionally, in your main loop or timer:
void loop() {
    // Clean up expired transfers periodically
    FileSystemWS::fsHandler.cleanupExpiredTransfers();
}

Client Side (TypeScript/Svelte)

Files:

• app/src/lib/filesystem/chunkedTransfer.ts - Client library for file transfers
• app/src/lib/components/filesystem/FileManager.svelte - Example UI component

Usage Example:

import { fileSystemClient } from '$lib/filesystem/chunkedTransfer'

// Upload a file
const file = new File(['Hello World'], 'test.txt')
const result = await fileSystemClient.uploadFileFromBrowser('/test.txt', file, (progress) => {
    console.log(`Upload: ${progress.percentage}%`)
})

// Download a file
const download = await fileSystemClient.downloadFileAndSave(
    '/test.txt',
    'test.txt',
    (progress) => {
        console.log(`Download: ${progress.percentage}%`)
    }
)

// List directory
const listing = await fileSystemClient.listDirectory('/')
console.log('Files:', listing.files)
console.log('Directories:', listing.directories)

// Create directory
await fileSystemClient.createDirectory('/new_folder')

// Delete file
await fileSystemClient.deleteFile('/old_file.txt')

Build Integration

Protobuf Compilation

After modifying platform_shared/message.proto, you must regenerate the protobuf code:

ESP32:

python esp32/scripts/compile_protos.py

Client (TypeScript):

cd app && pnpm proto

Or build the app (which automatically compiles protos):

cd app && pnpm build

PlatformIO Build

The ESP32 implementation files are automatically included in the build:

• esp32/include/filesystem_ws.h
• esp32/src/filesystem_ws.cpp

Make sure these files are in your source paths in platformio.ini.

Configuration

Maximum Chunk Size

Currently set to 1024 bytes (FS_MAX_CHUNK_SIZE) to work within ESP32 WebSocket frame limitations. Adjust if your setup allows larger frames.

Transfer Timeout

Transfers inactive for 30 seconds (FS_TRANSFER_TIMEOUT) are automatically cleaned up. Increase for slower connections.

Error Handling

• Network errors: Transfers are automatically cancelled
• Timeouts: Inactive transfers are cleaned up on ESP32
• File errors: Detailed error messages returned to client
• Partial uploads: Cancelled uploads delete the partial file on ESP32

Performance Considerations

• Sequential Chunks: Chunks are sent sequentially to ensure order and reliability
• Memory Usage: ESP32 keeps one File handle open per active transfer
• Browser Memory: Downloads buffer entire file in memory before saving
• Network: ~1KB per message overhead due to protobuf encoding

Security Notes

• No authentication/authorization implemented - add as needed
• Path traversal: Validate paths to prevent access outside allowed directories
• File size limits: Consider adding max file size restrictions
• Rate limiting: Consider limiting concurrent transfers per client

Testing

1. Build and flash the ESP32 firmware
2. Run the web application
3. Navigate to the FileManager component
4. Test upload/download with files of various sizes

Troubleshooting

Transfer fails midway:

• Check WebSocket connection stability
• Verify ESP32 has sufficient filesystem space
• Check for timeout issues

Upload creates corrupted files:

• Verify chunk order is preserved
• Check for protobuf encoding/decoding errors

ESP32 runs out of memory:

• Reduce number of concurrent transfers
• Close File handles properly after transfers
• Run cleanup more frequently