While BitTorrent might be all the rage, I’m a Usenet guy.

I am honestly not very patient when it comes to obtaining things I want, so waiting hours or days for downloads to complete has never been my strong suit. Aside from the original Napster, BitTorrent and similar P2P apps have never been something I have enjoyed using. Usenet however, is a completely different experience – one which I highly recommend.

You may or may not have heard of Usenet before, but it has likely been around longer than you have. In fact, the first spam message ever delivered was done so via Usenet. It’s the Internet’s oldest and best-kept “secret”. This is partly because Usenet is kind of like “Fight Club” in that the only real, yet unofficial rule is, “You do not talk about Usenet.”

I on the other hand do tell people about Usenet. However, I only do so provided I have deemed them technically capable enough to handle the task. This is really one of the main reasons why I think Usenet remains a little more obscure than most other online technology. While Usenet is not as complex as say, compiling your own Linux distro from scratch, there is definitely a learning curve that stretches beyond the patience and understanding of most people. I find that the benefits of Usenet far outweigh the cost and effort associated with using it.

The major highlight of Usenet is that there is virtually no waiting. You don’t have to wait hours for a slow download to complete. You don’t have to sit and hope that there are enough peers available to grab that old file you are looking for. When something is posted to Usenet, it’s there for good (almost). When you select a file that you want, you are able to download it as fast as your broadband connection can handle, provided you selected a good Usenet provider. Everything is “just there” – no waiting, no fuss…you just grab and go.

That said, if you are contemplating leaving the world of trackers and leechers behind, I will be more than happy to lead you to the path of enlightenment…

What is Usenet?

I won’t go into a thorough history of Usenet, a brief explanation will suffice.

In 1980, the first Usenet servers came to life, linking users at UNC Chapel Hill and Duke University. Much like BBS systems which were common at the time, the these servers allowed users to chat with one another by submitting posts, known as articles. These posts were organized into hierarchical “news groups” by subject, and read using special clients known as news readers. Posts consisted of threaded conversations that somewhat resemble commenting systems used by the majority of web sites today. The difference between Usenet and a simple BBS is that the servers would replicate messages back and forth, enabling users at both locations to connect to a local server while reading messages posted to any other server in the network.

Fast forward some time, and Usenet has evolved from a simple communications mechanism to over 100,000 newsgroups replicated to thousands of servers across the globe. These servers house and transfer terabytes of data daily, though the type of data has changed significantly since the Usenet was first established. While Usenet was intended as a means of basic, communication it was not long before people began encoding binary data into 8-bit ASCII and proliferating files across the globe.

Newsgroups

As I mentioned earlier, Usenet is hierarchical in structure, divided into nine major branches, each with many subgroups beneath them. “The Big Eight” refers to the primary discussion hierarchies of Usenet, which are all operated under certain guidelines, as to promote useful, structured discussion. The Big Eight consist of the following [courtesy of Wikipedia]:

• comp.* – computer-related discussions (comp.software, comp.sys.amiga)
• humanities.* – fine arts, literature, and philosophy (humanities.classics, humanities.design.misc)
• misc.* – miscellaneous topics (misc.education, misc.forsale, misc.kids)
• news.* – discussions and announcements about news (meaning Usenet, not current events) (news.groups, news.admin)
• rec.* – recreation and entertainment (rec.music, rec.arts.movies)
• sci.* – science related discussions (sci.psychology, sci.research)
• soc.* – social discussions (soc.college.org, soc.culture.african)
• talk.* – talk about various controversial topics (talk.religion, talk.politics, talk.origins)

The “everything else” hierarchy on Usenet all falls under the alt.* heading. The most populated hierarchy under alt.* is alt.binaries.*, which contains many subgroups separated into very specific subjects. The vast majority of file sharing goes on under the alt.binaries.* hierarchy. If there is something that interests you, there is likely a newsgroup dedicated to it.

Files on Usenet

While there is plenty of discussion to be had, people turn to Usenet for the files. As I mentioned before, downloading binary files is quite painless and fast, but you need to know a few things before you begin.

Most everything posted to Usenet is split apart in one way or another. Even a simple .jpg file is likely split into multiple articles before being posted.

Let’s consider an ISO image of the latest Ubuntu release for a moment. First, the poster will split the ISO image into RAR files, usually around 50 MB in size. Then, the poster will create parity data archives, called PAR files based on the content within the newly created RAR archives. The PAR files are posted to Usenet along with the RAR files, as they can be used to reconstruct any of the archives damaged during the upload or download process. Once the files have been created, the poster uploads them to his provider’s servers using software specifically written for this purpose. As the files are being transferred, they are yEnc encoded and then broken up into multiple posts so that they can be properly placed on the server. The post is then replicated to any servers that communicate with the poster’s provider.

When you load the header data for the group where the file was posted (which I will explain later), your newsreader looks at the articles and combines any multi-part postings into a single item. That way, each RAR file does not show up as hundreds of posts, rather it is represented as a single file. If you select all of the RAR files that make up the ISO you want, your newsreader will proceed to download them all. Then, you can extract the ISO using the proper application, or depending on which newsreader you use, it may assemble them into the ISO automatically. If there are issues with the files, you can manually check the RAR files for completeness using an application that handles PAR files. However, your newsreader may check for completion and download PAR files as needed automatically, again depending on which program you use.

All of this might sound a bit complicated, though it’s really not. This is just a general technical overview of the file upload and download process – I will go into more detail as to how you actually select and download files in a later piece.

Required Applications

Obtaining files from Usenet takes a handful of tools, some of which you might already have installed on your computer. Here are the the basic requirements:

Newsreader

There are many newsreaders available online. In fact, you probably already have one installed as most email applications have some news reading abilities built-in. The only problem is that email-based newsreaders such as Outlook or Thunderbird are built for text-based discussions, not binary downloads. You really need a purpose-built application to do the job.

Personally, I use Newsbin. I have taken cursory looks at other clients, but I instantly liked the interface and the support structure behind the application. I, like many people, pirated the software initially before purchasing it. They do give you a 30-day free trial, but I “borrowed” the application for a few months before taking the plunge. Honestly, the application is simply so good, I wanted to pay for it. $35 buys you a lifetime license that allows you to install it on several computers while also giving you access to the support forums. The forums are great – more often than not your questions are answered immediately by a seasoned Newsbin veteran or the main developer himself. The license also allows you to access all of the beta builds as they are released. It really is a hard deal to beat.

Additionally, Newsbin is chock full of features that really streamline your Usenet experience. Currently, Newsbin will try to repair damaged groups of archives as well as extract all of your downloads for you. It allows for internet-based file searches via a $5 per month service offered by Newsbin. However, it does allow you to search through all of the headers you have downloaded to your computer, all with the press of a button. Newsbin also gives you the ability to integrate the application with certain IRC channels for hassle-free downloading as well as use it in conjunction with my application, MailBin, to remotely queue up file downloads while away from home.

Other alternatives I have tried briefly are Grabit and Newsleecher. I really can’t comment too much on either, as I stuck with Newsbin almost immediately.

Archive Software

More likely than not, you already have an application on your computer that handles archives. I use both WinRAR and 7-Zip, depending on my needs. Both applications handle all of the same file formats, but they each have some features the other does not.

In very rare instances, a set of RAR files can be damaged before PAR files have been created. When these files are posted on Usenet, they will appear to look fine, though the files are really damaged. Unfortunately, PAR files are useless in this instance, however you should be OK since WinRAR has a feature which allows you to repair RAR archives. The application automatically packs some parity data into the archives it creates, so there is a chance you can repair them. This feature comes in handy every once in awhile, and is exclusive to WinRAR.

7-Zip’s unique feature is that it allows you to extract a set of RAR files even if you have not yet downloaded the entire set. WinRAR will usually report an error and quit without extracting the files, but 7-Zip does its best to get as much data from the incomplete archives as it can. This is quite a useful feature when downloading video files. You can easily preview the quality of the video before downloading the entire set since 7-Zip will extract as much of the video as it can from what you already have.

QuickPar

As I mentioned, PAR files are an important part of the Usenet puzzle. They contain data that allows you to rebuild incomplete, damaged, or missing files. They can be used to add repair and recovery data to any sort of file type, whether it be RAR, Zip, ISO, etc. QuickPar is THE app to use for file repairs. It has not been updated since 2004, but don’t let that deter you from downloading it. The program does its job well, and there really has not been any need for updates. Sure, a multi-threaded version of the application would be nice, but beggars can’t be choosers when you are talking about free software.

Usenet Providers

The last thing you need to get started is a Usenet Provider. There are many providers out there, and you may be paying for some sort of Usenet access already.

There was a time when almost every ISP bundled some sort of Usenet access into your broadband plan for free. Usually you were restricted to a 2 gigabyte per month download limit, which was not great, but then again it was free. Pressure from copyright groups has all but eliminated ISP-based Usenet access, so if you want to give Usenet a shot, you will likely have to look elsewhere.

There are many Usenet providers out there, so the question becomes, “How do I choose the best provider?” The answer to that question depends on what is important to you. The main categories I considered when I last went looking for a host were price and monthly data allotment, speed, security, retention and completion rates, uptime, as well as free add-on features. For new users, ease of use is also a big factor, and some providers abandon the client-server model to instead provide in-browser access to Usenet content.

Price and Data Allotment

You are probably wondering why I think Usenet is so great if I have paid for the software and also pay a monthly access fee. I honestly believe it is worth the cost due to the quality of the connections, the speed, and the security of the content delivery. It is yet another bill to pay, but you have some flexibility in what you spend.

Most providers offer tiered pricing based on the monthly data allotment. Some plans start as small as 10 GB per month and others scale to allow for unlimited data usage. One important factor when considering metered plans is whether or not header downloads count towards your overall quota. You can download several gigabytes of headers per month, depending on how many newsgroups you monitor, and this can easily push you over your limit if you are not careful. I recommend investing in an unlimited plan, but I download a lot, so that is just my opinion.

Speed

One of the big reasons for trying Usenet is the speed. If your provider can’t keep up with the speed of your Internet connection, it’s just not worth it. The good news is that most providers offer a free trial period, so you can test out their servers before you commit to a monthly plan.

Header download speeds are another thing to consider. Some providers such as Giganews offer the ability to download compressed headers, which increase your download speed ten-fold at times. Certain newsreaders such as Newsbin have the ability to download compressed headers built-in, which is a nice bonus feature to look out for.

Security

While this could technically reside under add-on features, it really deserves a section of its own. When talking about Usenet, security refers to encrypting your communications with your Usenet provider. Comcast is well-known for having specifically targeted BitTorrent traffic for throttling, based upon deep examination of customer’s data packets. The only way to prevent this sort of deep packet inspection and tampering is by encrypting your data. Some providers allow the use of SSL tunnels between their servers and the client’s newsreader in order to keep prying eyes from looking at the data. This ensures that no one can see what you are downloading, thus preventing any data or packet tampering/throttling/etc.

Unfortunately most providers require you to purchase a higher-tier plan in order to take advantage of SSL tunneling, but I personally think it is well worth the cost.

Retention and Completion

Retention used to be very important when choosing a Usenet provider. There is somewhat of a retention arms race going on in the world of Usenet, and most providers offer at least 500 days of binary retention. This means that you can pretty much rely on being able to download content that was posted to the server up to 500 days ago, with any of the major providers. Some providers however have far surpassed that number, and are well into the 700+ day range.

Completion was and always will be an important consideration when choosing a provider. Completion refers to how much, on average of the total content posted on Usenet is captured by your provider for your use. You might see people on Usenet exclaiming “Get a better provider” when people complain that they cannot download something due to incomplete posts. That is why completion is important – if your provider cannot provide complete content, then what is the point in paying for a service?

The only problem is that nearly every provider claims 99.99% completion. Are they being truthful? In all honesty, you won’t find out until you actually use them for awhile.

Uptime

This one is such a no-brainer that I will only give it a brief mention. Uptime is extremely important. There is no sense in paying for a provider if you can’t access the content reliably. Much like completion stats, most providers will claim 99% or 100% uptime. Only real-world usage can be the judge of that claim.

Add-on Features

Everyone likes something for free, there is no arguing that. Some Usenet providers offer small perks to go along with your membership in order to sweeten the deal. Some offer free SSL encryption, others additional connections to their servers. Giganews even happens to offer free high-speed VPN connectivity to their top tier users. Whatever the gimmick, keep your eyes peeled for the freebies various providers offer!

Picking a Provider

The process of searching for a Usenet provider can be daunting, even if you know what you are looking for. If you hadn’t gleaned the fact that I am a Giganews customer from the previous few sections, I am. I gladly pay $30 per month for the Diamond membership.

It might sound crazy to pay that much, but I love Giganews for many reasons. The retention time is well over 700 days and growing, the completion rate is out of this world, and I have never, ever experienced downtime with Giganews. I use the compressed headers with Newsbin to streamline my download process and I use the SSL tunneling to protect my traffic at all times. I can consistently max out my 18 Mb/s connection at home without a hiccup. Additionally, I use the free VyprVPN service to protect my web browsing from prying eyes while at hotels and public hot spots. I seriously cannot recommend them enough – I have been a customer for nearly 3 years and recommend them to all of my friends. I will disclose the fact that we do publish affiliate links, but only because I believe in their service so strongly.

I have also used Newshosting in the past. There was a decent amount of repeated downtime due to router issues, which eventually caused me to move to Giganews. I did however spend several years as a Newshosting customer and the experience was mostly good overall. I am pretty sure they have significantly beefed up their infrastructure since I last used them, so I would definitely give them a chance if you are trying to save a few dollars.

Jared is a fan of Usenetserver. I have not personally used them, but he may have something more to say about them if you stop by the Chat Room.

Finally, I will mention Easynews. As with Usenetserver, I have not personally used the service, but I have friends that do. Most of them have done the server/client Usenet setup, and find that they are downloading less data as time passes, so the web-based search and download features of Easynews more than covers their needs. Check it out if you want to try something a little easier.

That concludes part one in a series of articles on Usenet. I hope you hung in with me the whole way through, it is definitely worth it – I have convinced even the most die hard BitTorrent fans to give it up completely. In the next few weeks I will present more articles, which will go into detail about downloading and reading headers, downloading files, NZB files, IRC integration, and more!

More on this topic

Giganews – The Best Usenet Provider, Hands Down
MailBin – Remotely Queue up NZB Files Using Gmail

One Response to “An Introduction to Usenet”