Anyone who has worked out of a coffee shop and has had to upload large files to a remote system has probably experienced the initial burst speed of the upload followed by a disheartening throttling down of bandwidth to a crawl.
As I write a lot of software that uses the JVM I produce some rather large JAR files which need to get copied to a remote Puppet host, and I experience this bandwidth throttling often - both on public wifi and when I use my cellular data plan. Fortunately there's an easily solution if you're working on a Unix platform - split a file and have multiple concurrent uploads where each upload can take advantage of the initial bandwidth burst.
The below script isn't perfect but it gets the job done. I've limited the concurrent uploads to four and new batches of uploads are only started once the current batch has completed. I based this script off an article about running multiple processes in bash.
#!/bin/bash FILE=$1 HOSTNAME=$2 USER=cfeduke SPLIT_SIZE=3145728 # bytes, 3 MiB (1024*1024*3) SCP\_MAX\_COUNT=4 # no. of maximum scps at once cp $FILE /tmp cd /tmp FILE=$(basename $FILE) split -b$SPLIT_SIZE $FILE file-part- scp_count=0 for part in file-part-*; do scp -B "$part" $HOSTNAME:~ & scp_count=$(($scp_count+1)) if [ "$scp_count" -gt $SCP_MAX_COUNT ]; then wait scp_count=0 fi done if [ "$scp_count" -gt 0 ]; then wait fi ssh -o "BatchMode yes" -t $HOSTNAME "cat /home/$USER/file-part-* >> /home/$USER/$FILE && rm /home/$USER/file-part-*" rm file-part-*