Efficiently Handling Large Excel and CSV File Uploads in Spring Boot

Uploading large Excel or CSV files efficiently in a Spring Boot application is crucial for performance and scalability. This comprehensive guide covers strategies to manage large file uploads, including increasing file size limits, asynchronous processing, and tracking upload progress.

Why Large File Uploads Matter

When dealing with large datasets in enterprise applications, efficient file uploads are essential to prevent memory overload and ensure seamless user experience. This guide will help you optimize your Spring Boot application for large file uploads.

---

Key Considerations for Large File Uploads

Before implementing a solution, consider the following factors:

• Multipart Upload Limits: Configure the server to allow large files.
• Streaming Processing: Read files in chunks to optimize memory usage.
• Asynchronous Processing: Prevent request timeouts by processing files in the background.
• Progress Tracking: Provide real-time feedback to users.
• Database Optimization: Use batch inserts to enhance performance.

---

Configuring File Upload Limits in Spring Boot

To support large file uploads, increase the file size limits by configuring application.properties or application.yml:

application.properties

spring.servlet.multipart.max-file-size=500MB
spring.servlet.multipart.max-request-size=500MB
server.tomcat.max-swallow-size=500MB

application.yml

spring:
  servlet:
    multipart:
      max-file-size: 500MB
      max-request-size: 500MB
server:
  tomcat:
    max-swallow-size: 500MB

---

Asynchronous File Processing for Large Uploads

Using Spring Boot's @Async annotation enables non-blocking file processing, ensuring a responsive user experience.

Service to Process Files Asynchronously

import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import java.io.BufferedReader;
import java.io.InputStreamReader;

@Service
public class AsyncFileProcessingService {

    @Async
    public void processFile(MultipartFile file) {
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(file.getInputStream()))) {
            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println("Processing: " + line);
            }
            System.out.println("File processing completed.");
        } catch (Exception e) {
            System.err.println("Error processing file: " + e.getMessage());
        }
    }
}

---

Tracking File Upload Progress

Tracking file uploads helps provide real-time feedback to users. There are three popular methods:

1. Polling API for Status Check

Create a status tracking service:

import org.springframework.stereotype.Service;
import java.util.concurrent.ConcurrentHashMap;

@Service
public class FileUploadStatusService {
    private final ConcurrentHashMap<String, String> uploadStatus = new ConcurrentHashMap<>();

    public void updateStatus(String fileId, String status) {
        uploadStatus.put(fileId, status);
    }

    public String getStatus(String fileId) {
        return uploadStatus.getOrDefault(fileId, "Processing");
    }
}

Controller to check upload status:

@RestController
@RequestMapping("/api/files")
public class FileStatusController {

    private final FileUploadStatusService statusService;

    public FileStatusController(FileUploadStatusService statusService) {
        this.statusService = statusService;
    }

    @GetMapping("/status/{fileId}")
    public ResponseEntity<String> getFileStatus(@PathVariable String fileId) {
        String status = statusService.getStatus(fileId);
        return ResponseEntity.ok("File ID " + fileId + " status: " + status);
    }
}

2. WebSocket for Real-Time Notifications

Spring Boot WebSockets allow real-time progress updates.

@Configuration
@EnableWebSocket
public class WebSocketConfig implements WebSocketConfigurer {
    @Override
    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
        registry.addHandler(new FileUploadWebSocketHandler(), "/upload-status").setAllowedOrigins("*");
    }
}

Client-side WebSocket example:

const socket = new WebSocket("ws://localhost:8080/upload-status");

socket.onopen = () => {
  socket.send("your-file-id");
};

socket.onmessage = (event) => {
  console.log("Upload Status:", event.data);
};

3. Email Notification

Send email notifications upon file processing completion.

@Service
public class EmailNotificationService {
    private final JavaMailSender mailSender;

    public EmailNotificationService(JavaMailSender mailSender) {
        this.mailSender = mailSender;
    }

    public void sendCompletionEmail(String toEmail, String fileId) {
        SimpleMailMessage message = new SimpleMailMessage();
        message.setTo(toEmail);
        message.setSubject("File Upload Completed");
        message.setText("Your file with ID " + fileId + " has been processed.");
        mailSender.send(message);
    }
}

---

Efficient File Processing with Streaming

Use Apache POI's streaming API to process large Excel files efficiently.

import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.xssf.streaming.SXSSFWorkbook;
import org.springframework.web.multipart.MultipartFile;
import java.io.InputStream;

public void processExcelFile(MultipartFile file) throws Exception {
    try (InputStream inputStream = file.getInputStream();
         SXSSFWorkbook workbook = new SXSSFWorkbook()) {
        Sheet sheet = workbook.getSheetAt(0);
        for (Row row : sheet) {
            String data = row.getCell(0).getStringCellValue();
            System.out.println("Processing: " + data);
        }
    }
}

---

Conclusion

Handling large file uploads in Spring Boot requires optimized configurations and best practices. This guide provides strategies such as:

1. Increasing multipart file upload limits.
2. Implementing asynchronous file processing.
3. Tracking upload progress via REST API, WebSockets, or email notifications.
4. Optimizing file reading using streaming techniques.

By implementing these techniques, your Spring Boot application will efficiently handle large Excel and CSV file uploads.