Claude: Fixing esp side (and stupid amount of .md)
This commit is contained in:
Niklas Jensen
committed by
Rune Harlyk
Rune Harlyk
parent
f440fa3973
commit
0435605e18
@@ -0,0 +1,262 @@
|
||||
# Filesystem Chunked Transfer - Integration Summary
|
||||
|
||||
## ✅ Completed Implementation
|
||||
|
||||
### 1. Protocol Definition ([platform_shared/message.proto](platform_shared/message.proto))
|
||||
- ✅ Added 8 filesystem message pairs (Request/Response)
|
||||
- ✅ Integrated into `CorrelationRequest` and `CorrelationResponse`
|
||||
- ✅ Compiled for both ESP32 (nanopb) and client (ts-proto)
|
||||
|
||||
**Messages Added:**
|
||||
- `FSDeleteRequest` / `FSDeleteResponse`
|
||||
- `FSMkdirRequest` / `FSMkdirResponse`
|
||||
- `FSListRequest` / `FSListResponse`
|
||||
- `FSDownloadStartRequest` / `FSDownloadStartResponse`
|
||||
- `FSDownloadChunkRequest` / `FSDownloadChunkResponse`
|
||||
- `FSUploadStartRequest` / `FSUploadStartResponse`
|
||||
- `FSUploadChunkRequest` / `FSUploadChunkResponse`
|
||||
- `FSCancelTransferRequest` / `FSCancelTransferResponse`
|
||||
|
||||
### 2. ESP32 Implementation
|
||||
|
||||
**Files Created:**
|
||||
- ✅ [esp32/include/filesystem_ws.h](esp32/include/filesystem_ws.h) - Handler class definition
|
||||
- ✅ [esp32/src/filesystem_ws.cpp](esp32/src/filesystem_ws.cpp) - Complete implementation
|
||||
|
||||
**Files Modified:**
|
||||
- ✅ [esp32/include/communication/proto_helpers.h](esp32/include/communication/proto_helpers.h:44-59) - Added message traits
|
||||
- ✅ [esp32/src/main.cpp](esp32/src/main.cpp:9) - Added include for filesystem_ws.h
|
||||
- ✅ [esp32/src/main.cpp](esp32/src/main.cpp:191-238) - Added 8 correlation handlers
|
||||
- ✅ [esp32/src/main.cpp](esp32/src/main.cpp:316-319) - Added periodic cleanup task (every 60s)
|
||||
|
||||
**Features:**
|
||||
- Transfer state management with unique IDs
|
||||
- 1024-byte chunk size (configurable via `FS_MAX_CHUNK_SIZE`)
|
||||
- Automatic timeout cleanup (30s, configurable via `FS_TRANSFER_TIMEOUT`)
|
||||
- Recursive directory deletion
|
||||
- Sequential chunk processing for reliability
|
||||
- Error handling with detailed error messages
|
||||
|
||||
### 3. Client Implementation (TypeScript/Svelte)
|
||||
|
||||
**Files Created:**
|
||||
- ✅ [app/src/lib/filesystem/chunkedTransfer.ts](app/src/lib/filesystem/chunkedTransfer.ts) - Client library
|
||||
- ✅ [app/src/lib/components/filesystem/FileManager.svelte](app/src/lib/components/filesystem/FileManager.svelte) - Example UI
|
||||
|
||||
**API:**
|
||||
```typescript
|
||||
const client = new FileSystemClient()
|
||||
|
||||
// List directory
|
||||
await client.listDirectory('/')
|
||||
|
||||
// Upload file
|
||||
await client.uploadFile('/path.txt', data, progressCallback)
|
||||
|
||||
// Download file
|
||||
await client.downloadFile('/path.txt', progressCallback)
|
||||
|
||||
// Create directory
|
||||
await client.createDirectory('/folder')
|
||||
|
||||
// Delete file/directory
|
||||
await client.deleteFile('/path.txt')
|
||||
|
||||
// Cancel transfer
|
||||
await client.cancelTransfer(transferId)
|
||||
```
|
||||
|
||||
**Helper Methods:**
|
||||
- `uploadFileFromBrowser(path, File, callback)` - Upload from browser File object
|
||||
- `downloadFileAndSave(path, filename, callback)` - Download and trigger browser download
|
||||
|
||||
### 4. Documentation
|
||||
|
||||
**Files Created:**
|
||||
- ✅ [FILESYSTEM_CHUNKED_TRANSFER.md](FILESYSTEM_CHUNKED_TRANSFER.md) - Complete documentation
|
||||
- ✅ [FILESYSTEM_INTEGRATION_SUMMARY.md](FILESYSTEM_INTEGRATION_SUMMARY.md) - This file
|
||||
|
||||
## 🧪 Testing Checklist
|
||||
|
||||
### ESP32 Side
|
||||
|
||||
- [ ] **Build Test**: Compile ESP32 firmware
|
||||
```bash
|
||||
pio run -e seeed-xiao-esp32s3
|
||||
# or your target environment
|
||||
```
|
||||
|
||||
- [ ] **Upload Test**: Flash to ESP32
|
||||
```bash
|
||||
pio run -e seeed-xiao-esp32s3 -t upload
|
||||
```
|
||||
|
||||
- [ ] **Monitor Test**: Check for any errors in serial output
|
||||
```bash
|
||||
pio device monitor
|
||||
```
|
||||
|
||||
### Client Side
|
||||
|
||||
- [ ] **Build Test**: Build the web application
|
||||
```bash
|
||||
cd app && pnpm build
|
||||
```
|
||||
|
||||
- [ ] **Dev Test**: Run development server
|
||||
```bash
|
||||
cd app && pnpm dev
|
||||
```
|
||||
|
||||
### Integration Testing
|
||||
|
||||
- [ ] **List Directory**: Verify directory listing works
|
||||
- [ ] **Create Directory**: Test directory creation
|
||||
- [ ] **Upload Small File**: Upload file < 1KB (single chunk)
|
||||
- [ ] **Upload Large File**: Upload file > 10KB (multiple chunks)
|
||||
- [ ] **Download Small File**: Download file < 1KB
|
||||
- [ ] **Download Large File**: Download file > 10KB
|
||||
- [ ] **Delete File**: Test file deletion
|
||||
- [ ] **Delete Directory**: Test directory deletion
|
||||
- [ ] **Progress Tracking**: Verify progress callbacks work
|
||||
- [ ] **Error Handling**: Test with invalid paths, full filesystem, etc.
|
||||
- [ ] **Transfer Cancellation**: Test cancelling mid-transfer
|
||||
- [ ] **Timeout Handling**: Verify transfers timeout after inactivity
|
||||
- [ ] **Concurrent Transfers**: Test multiple simultaneous transfers
|
||||
|
||||
## 🔧 Configuration Options
|
||||
|
||||
### ESP32 ([esp32/include/filesystem_ws.h](esp32/include/filesystem_ws.h))
|
||||
|
||||
```cpp
|
||||
#define FS_MAX_CHUNK_SIZE 1024 // Maximum bytes per chunk
|
||||
#define FS_TRANSFER_TIMEOUT 30000 // Transfer timeout in milliseconds
|
||||
```
|
||||
|
||||
### Client ([app/src/lib/filesystem/chunkedTransfer.ts](app/src/lib/filesystem/chunkedTransfer.ts))
|
||||
|
||||
```typescript
|
||||
const MAX_CHUNK_SIZE = 1024 // Must match ESP32 setting
|
||||
```
|
||||
|
||||
## 📝 Usage Example
|
||||
|
||||
### Using the FileManager Component
|
||||
|
||||
```svelte
|
||||
<script>
|
||||
import FileManager from '$lib/components/filesystem/FileManager.svelte'
|
||||
</script>
|
||||
|
||||
<FileManager />
|
||||
```
|
||||
|
||||
### Using the API Directly
|
||||
|
||||
```typescript
|
||||
import { fileSystemClient } from '$lib/filesystem/chunkedTransfer'
|
||||
|
||||
// Upload a file with progress tracking
|
||||
const file = new File(['Hello World'], 'test.txt')
|
||||
const result = await fileSystemClient.uploadFileFromBrowser(
|
||||
'/test.txt',
|
||||
file,
|
||||
(progress) => {
|
||||
console.log(`Upload: ${progress.percentage.toFixed(1)}%`)
|
||||
console.log(`Bytes: ${progress.bytesTransferred} / ${progress.totalBytes}`)
|
||||
console.log(`Chunks: ${progress.chunksCompleted} / ${progress.totalChunks}`)
|
||||
}
|
||||
)
|
||||
|
||||
if (result.success) {
|
||||
console.log('Upload complete!')
|
||||
} else {
|
||||
console.error('Upload failed:', result.error)
|
||||
}
|
||||
```
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### ESP32 Build Errors
|
||||
|
||||
**Issue**: Undefined references to filesystem_ws functions
|
||||
- **Solution**: Ensure [esp32/src/filesystem_ws.cpp](esp32/src/filesystem_ws.cpp) is in the build
|
||||
- **Check**: Verify `src_dir = esp32/src` in platformio.ini
|
||||
|
||||
**Issue**: Protobuf message type errors
|
||||
- **Solution**: Regenerate protobuf files:
|
||||
```bash
|
||||
python esp32/scripts/compile_protos.py
|
||||
```
|
||||
|
||||
### Client Build Errors
|
||||
|
||||
**Issue**: TypeScript errors about missing message types
|
||||
- **Solution**: Regenerate TypeScript protobuf files:
|
||||
```bash
|
||||
cd app && pnpm proto
|
||||
```
|
||||
|
||||
**Issue**: Module not found for chunkedTransfer
|
||||
- **Check**: Verify file exists at [app/src/lib/filesystem/chunkedTransfer.ts](app/src/lib/filesystem/chunkedTransfer.ts)
|
||||
|
||||
### Runtime Errors
|
||||
|
||||
**Issue**: "Invalid transfer ID" errors
|
||||
- **Cause**: Transfer timed out or was already completed
|
||||
- **Solution**: Start a new transfer
|
||||
|
||||
**Issue**: "Failed to open file for writing"
|
||||
- **Cause**: Parent directory doesn't exist or filesystem is full
|
||||
- **Solution**: Create parent directory first or free up space
|
||||
|
||||
**Issue**: Upload creates corrupted files
|
||||
- **Check**: Ensure chunk size matches between client and ESP32
|
||||
- **Check**: Verify chunks are sent in order without gaps
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
1. **Test the implementation** using the checklist above
|
||||
2. **Adjust chunk size** if needed based on your network conditions
|
||||
3. **Add authentication** if required for your use case
|
||||
4. **Monitor performance** and adjust timeout values
|
||||
5. **Add rate limiting** for production use
|
||||
|
||||
## 📊 Performance Characteristics
|
||||
|
||||
- **Chunk Size**: 1024 bytes
|
||||
- **Protobuf Overhead**: ~20-50 bytes per message
|
||||
- **Effective Throughput**: ~970 bytes/chunk payload
|
||||
- **Transfer Speed**: Depends on network latency (sequential chunks)
|
||||
- **Memory Usage**:
|
||||
- ESP32: One File handle per active transfer
|
||||
- Client: Entire file buffered in memory for downloads
|
||||
|
||||
## 🔐 Security Considerations
|
||||
|
||||
- **Path Traversal**: Validate paths on ESP32 to prevent directory traversal attacks
|
||||
- **File Size Limits**: Consider adding maximum file size restrictions
|
||||
- **Authentication**: Add user authentication to filesystem operations
|
||||
- **Rate Limiting**: Implement rate limiting to prevent abuse
|
||||
- **Allowed Paths**: Restrict operations to specific directories only
|
||||
|
||||
## ✨ Features Implemented
|
||||
|
||||
- ✅ Chunked upload (Client → ESP32)
|
||||
- ✅ Chunked download (ESP32 → Client)
|
||||
- ✅ Directory listing
|
||||
- ✅ Directory creation
|
||||
- ✅ File/directory deletion
|
||||
- ✅ Progress tracking with callbacks
|
||||
- ✅ Transfer cancellation
|
||||
- ✅ Automatic timeout cleanup
|
||||
- ✅ Error handling and reporting
|
||||
- ✅ Browser File API integration
|
||||
- ✅ Example UI component
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [FILESYSTEM_CHUNKED_TRANSFER.md](FILESYSTEM_CHUNKED_TRANSFER.md) - Detailed documentation
|
||||
- [platform_shared/message.proto](platform_shared/message.proto) - Protocol definition
|
||||
- [esp32/src/filesystem_ws.cpp](esp32/src/filesystem_ws.cpp) - ESP32 implementation
|
||||
- [app/src/lib/filesystem/chunkedTransfer.ts](app/src/lib/filesystem/chunkedTransfer.ts) - Client library
|
||||
Reference in New Issue
Block a user