SpotMicroESP32-Leika/FILESYSTEM_INTEGRATION_SUMMARY.md

Filesystem Chunked Transfer - Integration Summary

✅ Completed Implementation

1. Protocol Definition (platform_shared/message.proto)

• ✅ Added 8 filesystem message pairs (Request/Response)
• ✅ Integrated into CorrelationRequest and CorrelationResponse
• ✅ Compiled for both ESP32 (nanopb) and client (ts-proto)

Messages Added:

• FSDeleteRequest / FSDeleteResponse
• FSMkdirRequest / FSMkdirResponse
• FSListRequest / FSListResponse
• FSDownloadStartRequest / FSDownloadStartResponse
• FSDownloadChunkRequest / FSDownloadChunkResponse
• FSUploadStartRequest / FSUploadStartResponse
• FSUploadChunkRequest / FSUploadChunkResponse
• FSCancelTransferRequest / FSCancelTransferResponse

2. ESP32 Implementation

Files Created:

• ✅ esp32/include/filesystem_ws.h - Handler class definition
• ✅ esp32/src/filesystem_ws.cpp - Complete implementation

Files Modified:

• ✅ esp32/include/communication/proto_helpers.h - Added message traits
• ✅ esp32/src/main.cpp - Added include for filesystem_ws.h
• ✅ esp32/src/main.cpp - Added 8 correlation handlers
• ✅ esp32/src/main.cpp - Added periodic cleanup task (every 60s)

Features:

• Transfer state management with unique IDs
• 1024-byte chunk size (configurable via FS_MAX_CHUNK_SIZE)
• Automatic timeout cleanup (30s, configurable via FS_TRANSFER_TIMEOUT)
• Recursive directory deletion
• Sequential chunk processing for reliability
• Error handling with detailed error messages

3. Client Implementation (TypeScript/Svelte)

Files Created:

• ✅ app/src/lib/filesystem/chunkedTransfer.ts - Client library
• ✅ app/src/lib/components/filesystem/FileManager.svelte - Example UI

API:

const client = new FileSystemClient()

// List directory
await client.listDirectory('/')

// Upload file
await client.uploadFile('/path.txt', data, progressCallback)

// Download file
await client.downloadFile('/path.txt', progressCallback)

// Create directory
await client.createDirectory('/folder')

// Delete file/directory
await client.deleteFile('/path.txt')

// Cancel transfer
await client.cancelTransfer(transferId)

Helper Methods:

• uploadFileFromBrowser(path, File, callback) - Upload from browser File object
• downloadFileAndSave(path, filename, callback) - Download and trigger browser download

4. Documentation

Files Created:

• ✅ FILESYSTEM_CHUNKED_TRANSFER.md - Complete documentation
• ✅ FILESYSTEM_INTEGRATION_SUMMARY.md - This file

🧪 Testing Checklist

ESP32 Side

• Build Test: Compile ESP32 firmware

  pio run -e seeed-xiao-esp32s3
  # or your target environment
  

• Upload Test: Flash to ESP32

  pio run -e seeed-xiao-esp32s3 -t upload
  

• Monitor Test: Check for any errors in serial output

  pio device monitor
  

Client Side

• Build Test: Build the web application

  cd app && pnpm build
  

• Dev Test: Run development server

  cd app && pnpm dev
  

Integration Testing

• List Directory: Verify directory listing works
• Create Directory: Test directory creation
• Upload Small File: Upload file < 1KB (single chunk)
• Upload Large File: Upload file > 10KB (multiple chunks)
• Download Small File: Download file < 1KB
• Download Large File: Download file > 10KB
• Delete File: Test file deletion
• Delete Directory: Test directory deletion
• Progress Tracking: Verify progress callbacks work
• Error Handling: Test with invalid paths, full filesystem, etc.
• Transfer Cancellation: Test cancelling mid-transfer
• Timeout Handling: Verify transfers timeout after inactivity
• Concurrent Transfers: Test multiple simultaneous transfers

🔧 Configuration Options

ESP32 (esp32/include/filesystem_ws.h)

#define FS_MAX_CHUNK_SIZE 1024      // Maximum bytes per chunk
#define FS_TRANSFER_TIMEOUT 30000   // Transfer timeout in milliseconds

Client (app/src/lib/filesystem/chunkedTransfer.ts)

const MAX_CHUNK_SIZE = 1024  // Must match ESP32 setting

📝 Usage Example

Using the FileManager Component

<script>
  import FileManager from '$lib/components/filesystem/FileManager.svelte'
</script>

<FileManager />

Using the API Directly

import { fileSystemClient } from '$lib/filesystem/chunkedTransfer'

// Upload a file with progress tracking
const file = new File(['Hello World'], 'test.txt')
const result = await fileSystemClient.uploadFileFromBrowser(
  '/test.txt',
  file,
  (progress) => {
    console.log(`Upload: ${progress.percentage.toFixed(1)}%`)
    console.log(`Bytes: ${progress.bytesTransferred} / ${progress.totalBytes}`)
    console.log(`Chunks: ${progress.chunksCompleted} / ${progress.totalChunks}`)
  }
)

if (result.success) {
  console.log('Upload complete!')
} else {
  console.error('Upload failed:', result.error)
}

🐛 Troubleshooting

ESP32 Build Errors

Issue: Undefined references to filesystem_ws functions

• Solution: Ensure esp32/src/filesystem_ws.cpp is in the build
• Check: Verify src_dir = esp32/src in platformio.ini

Issue: Protobuf message type errors

• Solution: Regenerate protobuf files:

  python esp32/scripts/compile_protos.py
  

Client Build Errors

Issue: TypeScript errors about missing message types

• Solution: Regenerate TypeScript protobuf files:

  cd app && pnpm proto
  

Issue: Module not found for chunkedTransfer

• Check: Verify file exists at app/src/lib/filesystem/chunkedTransfer.ts

Runtime Errors

Issue: "Invalid transfer ID" errors

• Cause: Transfer timed out or was already completed
• Solution: Start a new transfer

Issue: "Failed to open file for writing"

• Cause: Parent directory doesn't exist or filesystem is full
• Solution: Create parent directory first or free up space

Issue: Upload creates corrupted files

• Check: Ensure chunk size matches between client and ESP32
• Check: Verify chunks are sent in order without gaps

🚀 Next Steps

1. Test the implementation using the checklist above
2. Adjust chunk size if needed based on your network conditions
3. Add authentication if required for your use case
4. Monitor performance and adjust timeout values
5. Add rate limiting for production use

📊 Performance Characteristics

• Chunk Size: 1024 bytes
• Protobuf Overhead: ~20-50 bytes per message
• Effective Throughput: ~970 bytes/chunk payload
• Transfer Speed: Depends on network latency (sequential chunks)
• Memory Usage:
  • ESP32: One File handle per active transfer
  • Client: Entire file buffered in memory for downloads

🔐 Security Considerations

• Path Traversal: Validate paths on ESP32 to prevent directory traversal attacks
• File Size Limits: Consider adding maximum file size restrictions
• Authentication: Add user authentication to filesystem operations
• Rate Limiting: Implement rate limiting to prevent abuse
• Allowed Paths: Restrict operations to specific directories only

✨ Features Implemented

• ✅ Chunked upload (Client → ESP32)
• ✅ Chunked download (ESP32 → Client)
• ✅ Directory listing
• ✅ Directory creation
• ✅ File/directory deletion
• ✅ Progress tracking with callbacks
• ✅ Transfer cancellation
• ✅ Automatic timeout cleanup
• ✅ Error handling and reporting
• ✅ Browser File API integration
• ✅ Example UI component

📚 Additional Resources

• FILESYSTEM_CHUNKED_TRANSFER.md - Detailed documentation
• platform_shared/message.proto - Protocol definition
• esp32/src/filesystem_ws.cpp - ESP32 implementation
• app/src/lib/filesystem/chunkedTransfer.ts - Client library