Building wsend - A Command Line Tool to Easily Send Files. I wanted to write a post about the technical aspect of building wsend, after Jon Gottfried suggested I write one. Thanks Jon
As any technical project I undertake, I like to explain the motivation behind the project. What is the end state of the project? What is the final product supposed to look like? Most of the time I have an answer to these questions before I start. With wsend the answer was clear, a command line program that gave you a URL for a file up to 10GB in size. It should be simple to install and quick to use and not require any type of complicated setup. There are other command line tools for sending files but they either require me to put in additional information about where I am sending the file like scp, or I have to set this sort of thing up before I start. So I had this idea in mind to scratch my own itch and hopefully you find it useful too. (More motivation also came from: http://xkcd.com/949/).
Any time I’m working on a service that uploads a file, I like to start out on the back end. The reason for this is because we have straightforward tools for sending files like HTML upload forms and curl; but many tools for accepting an upload on the server.
My first attempt for a different website was to use PHP.
- I had found that PHP ran into a 2GB file upload limit, even when running on a 64bit machine.
- So my next option was Perl and nginx.
- Perl could handle a large file and so could nginx, but in order to link the two together I needed FastCGI.
- I ran into a similar 2GB file upload limit with FastCGI.
- So the next option was node.js
I was able to upload a file up to 10GB and it worked using node. This worked out really well because the application is the server, there’s no layer in between.
And here is an example of the express route used to handle the upload.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
Express makes it really easy to handle the file upload. The properties of the file are stored in
req.files.filehandle and you save the file to the file system with
fs.rename which comes from the fs module of nodejs and you get it with a
var fs = require('fs');.
The files are stored on the same server as everything else, while this is not optimal and a service like S3 or a private fileserver should be used, this is small scale right now and when the usage grows an alternative file storage system can be explored. All transfers happen using https and you can encrypt your file before it leaves your computer with a handy script called wsend-gpg which I describe later.
Now that we have a server setup with an API that accepts uploads and gives you a URL for a file we can do a lot of fun things on the client. So the first thing to do was write a command line script to handle all of these API calls using curl.
Which language to chose for a command line script?
- Perl is widely deployed on most Unix / Linux / Mac systems.
- In order to install a progress bar CPAN was needed.
- CPAN has to be set up on the client’s machine
- Bash is the default shell on many Unix installations
- Bash is a scripting language.
- cURL has its own built in progress bar.
- Use bash and curl.
So I started working on the CLI. After all the setup that the script does automatically it comes down to one command.
And it returns you a URL.
Here is the function inside the script that sets everything up and actually sends the file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
Inside the script this is the main command that actually sends your file.
https://wsend.net. The return of that command is your URL. Its really simple, but sometimes simple works out the best.
Other Client Possibilities
Now that we have an API and a command line utility we can do some interesting stuff:
- We can write a script to auto upload an image and get a link that shows up as a Twitter Card.
- We can write a program to provide end to end encryption
- We can write a new client in nodejs as well.
- We can make a Drag and Drop GUI website interface.
- We can upload an html file and have it render in the browser, even a slideshow using reveal.js.
- Even a static site generator (still under development [in pre-alpha] though)
Future projects like these could include:
- Writing clients in Ruby, Python, Perl, C …
- Distributing with rubygems, pip, CPAN, apt, yum, pacman …
- Or anything else you can thing of and do.
It’s been a really good learning experience and it has been my first opportunity to work with others in open source on my projects, usually I’m a contributor to someone else’s project but when people are submitting me code to review and incorporate into my open source project I really get a feeling that the community is working together as a whole. Its really amazing that people believe in my ideas and like them enough to contribute and I’m really grateful for the help. It was really exciting to get my first pull request and I hope to work with others on this in the future.
Thanks Jon, for encouraging me to write this post.
Check out the site at https://wsend.net
Follow me on twitter @abemassry