Abe Massry

Web Development and Building Businesses

Building Wsend - a Command Line Tool to Easily Send Files

Building wsend - A Command Line Tool to Easily Send Files. I wanted to write a post about the technical aspect of building wsend, after Jon Gottfried suggested I write one. Thanks Jon

Motivation

As any technical project I undertake, I like to explain the motivation behind the project. What is the end state of the project? What is the final product supposed to look like? Most of the time I have an answer to these questions before I start. With wsend the answer was clear, a command line program that gave you a URL for a file up to 10GB in size. It should be simple to install and quick to use and not require any type of complicated setup. There are other command line tools for sending files but they either require me to put in additional information about where I am sending the file like scp, or I have to set this sort of thing up before I start. So I had this idea in mind to scratch my own itch and hopefully you find it useful too. (More motivation also came from: http://xkcd.com/949/).

Server Side

Any time I’m working on a service that uploads a file, I like to start out on the back end. The reason for this is because we have straightforward tools for sending files like HTML upload forms and curl; but many tools for accepting an upload on the server.

My first attempt for a different website was to use PHP.

  • I had found that PHP ran into a 2GB file upload limit, even when running on a 64bit machine.
  • So my next option was Perl and nginx.
  • Perl could handle a large file and so could nginx, but in order to link the two together I needed FastCGI.
  • I ran into a similar 2GB file upload limit with FastCGI.
  • So the next option was node.js

I was able to upload a file up to 10GB and it worked using node. This worked out really well because the application is the server, there’s no layer in between.

The rest of the backend is a standard express app that exposes a simple API using POST for all communication. And all this can be set up using curl on the client side for testing.

And here is an example of the express route used to handle the upload.

Express Upload Example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
// upload file from command line
app.post('/upload_cli', function(req, res) {
  var userID = req.param('uid');
  var now = new Date();
  if (userID) {
    models.users.findOne({user_id: userID}, function (err, doc) {
      if (err) throw err;
      if (doc.user_id) {
        var filesize = req.files.filehandle.size;
        var filesizeInt = parseInt(filesize);
        var filename=req.files.filehandle.name;

        // newpath needs a unique folder,
        var uniqueDir = getUniqueDir();
        var newDir = __dirname + "/uploads"+'/'+ uniqueDir + '/';

        fs.mkdir(newDir, 0744, function (err) {
          if(err) { throw err; }
          var newPath = __dirname + "/uploads"+'/'+ uniqueDir + '/' +filename;

          //add a check to make sure newpath is writable
          fs.rename(req.files.filehandle.path, newPath, function(err) {
            if (err) throw err;
            var permalink = "https://wsend.net/"+ uniqueDir + '/' + filename;
            doc.files.push({timestamp: now,
                            filename: filename,
                            type: '',
                            size: filesizeInt,
                            dir: newPath,
                            permalink: permalink,
                            permissions: ''
                          });
            // save file location to db
            doc.save(function(err) {
              if (err) throw err;
            });
            // send permalink as response
            res.send(permalink);
          });
        });
      }
    });
  }
});

Express makes it really easy to handle the file upload. The properties of the file are stored in req.files.filehandle and you save the file to the file system with fs.rename which comes from the fs module of nodejs and you get it with a var fs = require('fs');.

The files are stored on the same server as everything else, while this is not optimal and a service like S3 or a private fileserver should be used, this is small scale right now and when the usage grows an alternative file storage system can be explored. All transfers happen using https and you can encrypt your file before it leaves your computer with a handy script called wsend-gpg which I describe later.

Client Side

Now that we have a server setup with an API that accepts uploads and gives you a URL for a file we can do a lot of fun things on the client. So the first thing to do was write a command line script to handle all of these API calls using curl.

Which language to chose for a command line script?

  • Perl is widely deployed on most Unix / Linux / Mac systems.
  • In order to install a progress bar CPAN was needed.
  • CPAN has to be set up on the client’s machine
  • Bash is the default shell on many Unix installations
  • Bash is a scripting language.
  • cURL has its own built in progress bar.
  • Use bash and curl.

So I started working on the CLI. After all the setup that the script does automatically it comes down to one command.

wsend command
1
wsend file.txt

And it returns you a URL.

Here is the function inside the script that sets everything up and actually sends the file.

wsend upload function
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
sendFile() {
  if [[ -e "$fileOrDirToSend" ]]; then
    if [ -d "$fileOrDirToSend" ]; then
      #we want to send a directory, so make a compressed archive
      fileOrDirToSend=${fileOrDirToSend%/}
      tar cfj "$fileOrDirToSend.tar.bz2" "$fileOrDirToSend"
      fileToSend="$fileOrDirToSend.tar.bz2"
    elif [ -e "$fileOrDirToSend" ]; then
      fileToSend=$fileOrDirToSend
    fi

    if [ "$clientOS" == "Darwin" ]; then
      fileToSendSize=$(stat -f %z "$fileToSend")
    else
      fileToSendSize=$(stat -c%s "$fileToSend")
    fi

    getAccountSpace
    if [ "$accountSizeAvailable" == "not enough space in your account for this transfer" ]; then
      notEnoughSpaceErr
    elif [ "$accountSizeAvailable" == "file is too big for your account size" ]; then
      filesizeTooLarge
    else
      if [[ $link ]]; then
        #link was provided, so update target link with file
        curlReturn=$(curl -F "uid=$id" -F "link=$link" -F "filehandle=@$fileToSend" $host/update_cli)
      else
        #simply create a new one
        curlReturn=$(curl -F "uid=$id" -F "filehandle=@$fileToSend" $host/upload_cli)
        echo "$curlReturn|$(make_absolute "$fileOrDirToSend")" >> "$wsend_base/.list"
      fi
      echo $curlReturn
    fi

    if [ -d "$fileOrDirToSend" ]; then
      #remove our temporary file
      rm "$fileToSend"
    fi
  elif [ "$fileSendBool" == "true" ]; then
    #want to send file, but source doesn't exist
    echoUsage="true"
  fi
}

Inside the script this is the main command that actually sends your file.

curl the file
1
curl -F "uid=$id" -F "filehandle=@$fileToSend" $host/upload_cli

Where $host is https://wsend.net. The return of that command is your URL. Its really simple, but sometimes simple works out the best.

Other Client Possibilities

Now that we have an API and a command line utility we can do some interesting stuff:

Future projects like these could include:

  • Writing clients in Ruby, Python, Perl, C …
  • Distributing with rubygems, pip, CPAN, apt, yum, pacman …
  • Or anything else you can thing of and do.

Outcome

It’s been a really good learning experience and it has been my first opportunity to work with others in open source on my projects, usually I’m a contributor to someone else’s project but when people are submitting me code to review and incorporate into my open source project I really get a feeling that the community is working together as a whole. Its really amazing that people believe in my ideas and like them enough to contribute and I’m really grateful for the help. It was really exciting to get my first pull request and I hope to work with others on this in the future.

Thanks Jon, for encouraging me to write this post.

Check out the site at https://wsend.net

Follow me on twitter @abemassry

Comments