Posts filed under ‘Programming’

I recently had the need to install Nagios on my network, and being new to the application, was stumped by an issue I encountered when installing the “standard” Nagios plugins pack. Upon running the make command, I received the following error (among several other related errors):

make[2]: *** [check_http.o] Error 1
make[1]: *** [all-recursive] Error 1

After a bit of digging around I saw that the errors were SSL-related, and upon doing a bit of research, I found that my server was missing libssl-dev. After running apt-get install libssl-dev, the install continued on without further issue.

Pi (π) has always been a source of interest and frustration to mathematicians. The duality of this constant lies in the simplistic beauty of its definition paired with the frustration of its mathematically proven transcendental nature. For centuries mathematicians have been devising ever more clever and accurate approximations for pi, but did you know that you can approximate the value for pi without the need for complex mathematical equations? By using a random sampling of numbers coupled with elementary statistics you can execute a Monte Carlo simulation to approximate the value of pi with a few short lines of programming.

Read the rest of this entry »

Part 3: Further OverDrive Media Console Analysis and DRM Removal Automation

In the previous article we identified a weakness within the OverDrive Media Console which allows direct access to EPUB data. However, application of public tools for the removal of the ADEPT DRM layer proved to be fruitless due to the specification noncompliant nature of the EPUB data. In this article we will further investigate the inner workings of OverDrive Media Console and outline several vectors which will allow the removal of the ADEPT DRM layer in an automated fashion.

From inspection of the ineptepub tool we can see that the encrypted version of the RSA certificate is stored in an XML tag titled encryptedKey. So lets use our trusty grep tool again and see what we can turn up within OverDrive Media Console’s program directory:

Very interesting. Apparently OverDrive Media Console stores some information in a SQLite database. Using SFTP lets copy this file to our host system and see what goodies are contained within its binary walled garden.

While most of the information contained within the database is not applicable to our efforts the ZMEDIA and ZLICENSE tables contain some choice information. Examining the contents of these tables we find some columns of data that have the information relevant to our interests:

Table       Column                 Description
ZMEDIA    ZCONTENTID   Unique identifier for the fulfilled EPUB request.
ZMEDIA    ZTITLE           Title of the EPUB.
ZMEDIA    ZBASEURL       Download URL for the fulfilled EPUB request.
ZLICENSE  ZLICENSESTR  A mangled version of the ADEPT rights.xml info.

Jackpot! Although there’s much more data to be investigated most of it has to do with business logic associated with the OverDrive Media Console GUI itself. In theory with this data we should be able to generate a rights.xml file from the ZLICENSESTR column data using an XSLT or manual transformation, package that file with the target EPUB file, and use the ineptepub tool to decrypt the archive’s contents.

For purposes of testing I have created just such an EPUB by extracting the corresponding ZLICENSESTR data, converting it to a proper rights.xml file, and archiving it with the test.epub file as test.rights.epub. Now, when executing the ineptepub tool against the file we get the following result:

Success! If we extract the contents of this archive we find that the AES-128-CBC encryption has been removed from the EPUB contents and the result is a completely open standards compliant EPUB that can be loaded into any compatible reader.

When viewing the EPUB even though OverDrive Media Console will decrypt several pages at once every 10 pages or so and cache the results to improve the interaction with the document on the iPad or iPhone there is a pause of up to a full minute while the next batch of pages are decrypted. Now with this encryption layer removed and the EPUB loaded into iBooks the response time is at least several order of magnitudes faster and much more in line with the experience that a publisher would want a customer to have with their product.

So to recap we are able to strip the ADEPT security layer from the titles downloaded through OverDrive Media Console by the following method:

1. Copy the unarchived EPUB data from the device.
2. Copy the OverDrive.sqlite database from the device.
3. Extract the ZLICENSESTR data for the corresponding EPUB title.
4. Convert the ZLICENSESTR to the appropriate rights.xml format and create the file in the extracted EPUB’s META-INF directory.
5. Zip the extracted archive’s data into a new EPUB.
6. Execute the ineptepub tool on the newly created EPUB.

While this method works there should be a much easier way to utilize the data from the OverDrive.sqlite database to automate the process.

Developing Automated DRM Removal Tool

To begin our development it should be noted that every EPUB that is downloaded to OverDrive Media Console is identified with a unique 32 character hexadecimal Content ID of the form:

01234567-9ABC-DEF0-1234-567890ABCDEF

As this ID is present in both the ZMEDIA and corresponsing ZLICENSE entries of the database we should be able to consolidate this information into a single data structure containing all necessary information about the entries. Furthermore recall that there was a ZBASEURL column in the ZMEDIA table that was described as containing a download URL for the fulfilled EPUB request. The URL exists in the form:

http://acs.contentreserve.com/ACSStore1/
0123-4/567/89A/BC/{01234567-9ABC-DEF0-1234-567890ABCDEF}Fmt410.epub

Note that this URL exists per title within the Content Reserve system as the data contained within the EPUB is encrypted with the same AES cipher key for all requests. Also, from testing it can be seen that there is no known user-agent checking or authentication associated with this URL so any method may be used to retrieve the data including but not limited to: regular browser request, cURL, download accelerator, or even a custom download library. By utilizing this field we are negated from having to transfer the uncompressed EPUB archive from our device. Therefore the only transfer to the host system that should be necessary is the OverDrive.sqlite file.

However, there is still one more catch. As it exists the ineptepub tool requires a rights.xml file to be present in the EPUB archive or it will throw an invalid ADEPT EPUB exception. Therefore the proposed solution will automatically modify the specification noncompliant ADEPT EPUB by adding this rights.xml file effectively making a specification compliant ADEPT EPUB and pass execution off to the ineptepub tool to finish the processing.

Finally, since Python is currently being used for execution of the ineptepub tool the solution may as well also execute within the same Python environment. With all these requirements in place we present to you:

OverDrive Media Console DRM Stripper v1.1 (omcstrip.py)

Simply execute this tool passing in the path to your OverDrive.sqlite and adeptkey.der files and the tool will print out a list of EPUBs ready to be downloaded and stripped of the ADEPT DRM layer. Execute the tool along with a desired Content ID and the EPUB will be automatically downloaded from the server, insert the necessary rights.xml file, and delegate the actual removal of the DRM to the ineptepub tool.

OMCStrip Usage Information

python ./omcstrip.py --help

OverDrive Media Console DRM Stripper

Usage: omcstrip.py [OPTION]...

Startup:
-h, --help    print this help
-l, --list    list available files for processing and exit
-i, --id      content id of the asset to be processed.
-a, --all     process all available assets
-d, --debug   print debug information
-v, --version print script version and exit
Input:
-s, --sqlfile  OverDrive SQLite database (Default: OverDrive.sqlite)
-k, --keyfile  Extracted Abode ADEPT DER key file (Default: adeptkey.der)

python ./omcstrip.py --list

Content ID                        Title
----------                        ------------
01234567-9ABC-DEF0-1234-567890ABCDEF    Test EPUB

Example OMCStrip Excution

python ./omcstrip.py --id 01234567-9ABC-DEF0-1234-567890ABCDEF

Content ID                        Title
----------                        ------------
01234567-9ABC-DEF0-1234-567890ABCDEF    Test EPUB

Downloading 01234567-9ABC-DEF0-1234-567890ABCDEF.epub (12345678 bytes)
Downloading complete.
Injecting rights.xml
Stripping Adobe ADAPT DRM from 01234567-9ABC-DEF0-1234-567890ABCDEF.epub
Saving ePub as 01234567-9ABC-DEF0-1234-567890ABCDEF.decoded.epub

Conclusion

While the ADEPT DRM scheme provides excellent content protection in accordance with the EPUB specification its use as a rights management tool is architecturally flawed. Furthermore, OverDrive Media Console’s unencrypted and trivially obfuscated implementation of its own standard can be easily leveraged to repackage the content into a specification compliant format which can be used in conjunction with existing tools for easy decryption.

[Thanks to Armin Tamzarian]

OverDrive Media Console Analysis

In the previous article we investigated the background and technologies surrounding the DRM process used by the OverDrive Media Console. In this article we will turn our attention to the OverDrive Media Console itself in order to locate any mechanism which we can manipulate for our advantage.

Before we begin with the analysis of the OverDrive Media Console application it should be noted that there are some prerequisites to enable proper analysis:

• A jailbroken iDevice (For demonstration purposes an iPad and iPhone will be used). I have not had a chance to look into Android or Blackberry installations of OverDrive Media Console, but with file system access similar result may be achieved.
• An Adobe ID
• An installation of Adobe Digital Editions
• A system capable of running Python
• inpetkey
• ineptepub

Firstly SSH or otherwise gain command line access to your device and locate the installation of OverDrive Media Console. As an example, on the iPad the program was found at following location:

/private/var/mobile/Applications/EB126631-4B6C-40E5-8430-4F6B9ABC403C

If you’re having trouble the execute from the root (/ not /root) directory:

find * | grep OverDrive

Inspecting the program directory it can be seen that the EPUBs that have been downloaded are stored in an uncompressed format under:

[OverDrive Media Console Root]
- /Documents
-- /eBooks
--- /ePubs
---- /{EPUBID}.epub
----- ...
---- /{EPUBID}.epub
----- ...
---- /{EPUBID}.epub
----- ...

From this listing it can be seen that when one chooses to download an EPUB title through the OverDrive Media Console application the EPUB data is effectively fetched from the content server and stored locally. This functionality is what allows the program to operate in an offline mode. However, even though the content is stored locally it is still encrypted utilizing the AES-CBC-128 cipher as noted in each EPUB’s META-INF/encryption.xml file.

So now with all of this information one should be able to ZIP up the EPUB directory to create a specification-compliant EPUB file and use the available ineptkey and ineptepub tools to decrypt the file. However, let’s see how these tools work and verify their functionality with our newly created EPUBs.

First, since we have already installed completed the process of creating an Adobe ID, installing Adobe Digital Editions, and authenticating with our Adobe ID we will use the ineptkey tool to retrieve our private RSA cipher key.

The RSA cipher key was saved as adeptkey.der which includes the private cipher key in a x.509 certificate utilizing ASN.1 DER encoding. Now in theory all we have to do is use the ineptepub tool in conjunction with this certificate to extract the AES cipher key and decrypt the target EPUB content.

Strange. We get a failure, and the only message is that our file is not a valid ADEPT EPUB. So what situation can cause this error to be thrown? Examining the source we find the following logic statement:

if 'META-INF/rights.xml' not in namelist or \
'META-INF/encryption.xml' not in namelist:
raise ADEPTError('%s: not an ADEPT EPUB' % (inpath,))

So essentially if the EPUB does not contain a rights.xml or encryption.xml file in the archive’s META-INF subdirectory the document is assumed to not be a valid ADEPT archive. Indeed, upon further inspection of the archive this fact is confirmed insomuch that although there exists the required META-INF/encryption.xml document there is no META-INF/rights.xml document which contains the information needed to ultimately decrypt the document’s contents.

What does this mean? In essence, either Adobe (through their Adobe Content Server product) or OverDrive Inc (through their Content Reserve system) are apparently transmitting EPUBs in some sort of format that is not compliant with the ADEPT standard that they themselves created. To wit, does this mean that we’ve hit a brick wall in our analysis of the OverDrive Media Console Software? Hardly. Since the software can operate in an offline mode there must exist a method by which the information originally contained within the rights.xml is stored and retrieved locally.

In our next article we will delve even further into the inner workings of the OverDrive Media Console and bring to light several failures of the system and its architecture that allow for full automation of the DRM removal process with minimal effort.

[Thanks to Armin Tamzarian]

An Introduction to EPUBs, OverDrive Media Console, and Adobe ADEPT

Many libraries are finally entering the technology age and have started offering the ability to lend digital media to patrons including eBooks for use on compatible devices. While there are many companies which offer the services and infrastructure for your local library to enable this service one of the largest is OverDrive, Inc. with its Content Reserve system. Content Reserve is built on Adobe Content Server, and in order to enforce content protection where applicable OverDrive implements the ADEPT digital rights management scheme.

In order to view the protected content the Content Reserve system requires the use of OverDrive Media Console on a compatible device. This software allows the user to download a reserved title and view it for a library-defined loan period before revoking access and requiring the user to reserve the book again. However, how does one view these titles on an incompatible device or utilize otherwise non-approved software? In order to answer this question let’s break down the problem into its individual components and see if there is a way to work around the limits of the content protections.

EPUB Format

The EPUB (short for Electronic PUBlication) format is a free and open eBook standard managed by the International Digital Publishing Forum. The current 2.0.1 version of the EPUB standard defines three specifications:

• Open Publication Structure (OPS)
• Open Packaging Format (OPF)
• Open Container Format (OCF)

Although this separation of the standard may be confusing just know that any EPUB file that you may find is likely packaged in the OCF format which in turns uses the OPS and OPF specifications to define its internal data structure. In reality the OCF format is simply a ZIP archive of the documents adhering to an internal structure as defined by the OPS and OPF standards. Therefore if you’re a curious individual you can take most EPUB files, process them with an UnZIP utility, and view contents on the files.

Of note to us regarding the OCF specification is the optional support of a digital rights management (DRM) layer. When the DRM layer is present on an EPUB a rights.xml file must exist in the internal file structure within the META-INF directory. Additionally the OCF specification allows for the encryption of its content, but when present, an encryption.xml file must exist in the internal file structure also within the META-INF directory.

OverDrive Media Console

OverDrive Media Console is a freeware program distributed by OverDrive Inc. which enables end users to view digital media distributed through the Content Reserve system. While it is available for many platforms only the versions available for Android, Blackberry, and iOS support the EPUB formats.

For reasons which will become clear later in this article the workflow for checking out an EPUB document from the Content Reserve system and viewing it in the OverDrive Media Console is as follows:

1. Create an Adobe ID.
2. Install Adobe Digital Editions and activate the software with your Adobe ID.
3. Install OverDrive Media Console and activate the software with your Adobe ID.
4. Browse content at your library’s website and check out the desired EPUB title.
5. Download the EPUB with the provided URL that initiates a connection to the OverDrive Media Console to begin the download process.
6. Download the title to the OverDrive Media Console.
7. Open the title.

While quite the lengthy process every step has a very specific purpose. The reason for the tight coupling between the OverDrive software and Adobe is due to the fact that media downloaded through the Content Reserve system is protected using the ADEPT DRM scheme.

Of note to us is the fact that the OverDrive Media Console actually downloads the EPUB document to the device for viewing in offline mode. Additionally, a user is not require to authenticate with their Adobe ID ever time they open a title in the OverDrive Media Console.

ADEPT

ADEPT (Adobe Digital Experience Protection Technology) is a digital rights management scheme developed by Adobe and OverDrive Inc. While there is not much information publicly available concerning the technical implementation the author of the I♥CABBAGES blog was able to successfully reverse engineer the functionality of the system in early 2009.

ADEPT relies on the DRM and encryption layers as defined by the EPUB OCF standard. Specifically the EPUB content is encrypted using the Advanced Encryption Standard (AES) cipher operating in Cipher-Block Chaining (CBC) mode with a per-book 128-bit cipher key and a randomized Initialization Vector (IV). The stated AES cipher key is included in the distributed archive after being encrypted with a per-user RSA encryption key utilizing PKCS #1 v1.5 padding.

When an end user requests fulfillment of a title it is the job of the Adobe Content Server to retrieve the user’s encryption credentials from the ADEPT server and generate the rights.xml file to be included with the EPUB title according to the specification.

As noted by the author of the I♥CABBAGES blog the ADEPT system outlines a very good encryption scheme for content protection. By including the RSA encrypted AES cipher key as a part of the EPUB distributable itself the rights management of the scheme and more appropriately the entirety of the content protection scheme is effectively compromised.

In our next article we will begin our analysis of the OverDrive Media Console software itself and attempt to identify any available methods for accessing the EPUB content and a process for stripping any relevant DRM from the content.

[Thanks to Armin Tamzarian]

While all of the applications I write in .NET do not require installation, the restrictions that Microsoft places on using their built-in settings provider tend to make the programs less than portable.

By default, configuration files generated by .NET applications reside in a complicated path located under a user’s AppData folder, or Application Data folder for those of you still using Windows XP. For instance, the configuration file used for Suction on my system is:

C:\Users\mike\AppData\Local\MN\Suction.exe_Url_0k4nu2jvc3qbgln4l3jyokwo1uqvgcs5

This particular configuration path reflects a copy of Suction located on my desktop. If I were to move the executable to another directory, a separate configuration file would be created under a new path in my AppData directory. Clearly this is less than ideal, as well as incredibly flexible.

Developers have asked Microsoft to grant the ability to change the location of these configuration files using the .NET API through official feedback channels, but Microsoft has refused. The boys in Redmond responded saying that this behavior is by design and was created in order to prevent one application from overwriting another program’s configuration file. They suggest that users create their own settings provider if they want to choose where configuration files are saved. Their inflexibility is not only annoying, it is stifling.

I have had more than a few people who use my software ask me to localize the configuration files so that the applications are truly portable. I searched high and low for a easy solution to the problem, but I could not find one for the life of me. I found examples and pieces of code here and there, but nothing that would create the kind of configuration files I was looking for. What I was searching for was a portable settings provider that was 100% back-compatible with existing application configuration files. I finally decided that if I wanted that kind of functionality, I would have to build it myself. Continue reading to view and download my C# Portable Settings Provider.

Read the rest of this entry »