Tuesday, 9 June 2015

WebSphere Liberty, égalité, fraternité for Java EE 7 – in all clouds

When we created WebSphere Liberty we designed it to be easy to use and run well in any cloud, so that Liberty could always be the enterprise Java container regardless of the choice of cloud infrastructure. WebSphere Liberty - Égalité in all clouds for the Java EE 7 fraternité… OK, I’m still working on that after a pleasant weekend break in Paris. But we've just about finished working on adding Java EE 7 to Liberty and now you can run it in production in any cloud. Full posting with example on WASdev.net.

Wednesday, 25 March 2015

Whole disk encryption cloning and fun with Windows MBR

or How I am Paranoid but Not Paranoid Enough.

Working for a large IT company makes me a paranoid person in a paranoid organization - but we are paranoid about different things. The company is crazy about managing the data security of thousands of employees travelling the world with access to confidential corporate information. I'm bothered about my part in that too but also about how easy it is for me to keep my laptop completely backed-up so I can quickly resurrect my working environment if I ever lost or damaged my laptop or had it stolen. Data backup is easy and secure in the corporate cloud but that doesn't get close to satisfying my need to be back up and running within hours of losing my laptop - I know from bitter experience that it can take days to reinstall and reconfigure all the software I use if all I have is the data backed up in cloud.
I have 3 basic requirements for backing up my laptop:
  1. I want to have everything (data, software, configuration) on my HDD (or SSD) in a backed up image.
  2. I want to be able to easily use that backed-up image to re-create a new, identical bootable drive.
  3. I want this to be fast and cheap and not send/receive hundreds of gigabytes over the network.

This used to be an easy problem to solve. There are lots of very good open source and commercial software that make it easy to clone a hard drive - all you need is a second hard drive of the same or larger capacity than the source. An hour a week to clone the HDD and you're done. I use Acronis True Image but there are plenty of good alternatives. If I lose the laptop or the primary drive is fried I have an immediately-available bootable replica that is as up-to-date as I have chosen to keep it.

Then along came PGP whole disk encryption (WDE) and a corporate policy to use it.

To me, WDE is something best left to hardware. Encryption in hardware is fast and reliable and doesn't complicate disk cloning at all because the encryption is wholly self-contained within the drive hardware and independent of the actual content stored on the drive. My company's policy for specifically requiring PGP rather than hardware encryption is nothing to do with security or reliability but is all about centralized management of the encryption keys. No issue with that BUT it makes it much harder to achieve all 3 of my backup criteria. And while I can sleep soundly knowing that if I was ever dozy enough to forget the password from which the key is derived my company could recover it, I'm on my own for the second (and hardest) of my three recovery criteria.

Why is backup/recovery harder with software FDE?

I learnt this the hard way. By far the simplest way to satisfy my backup objectives is a disk clone - something I've done regularly for years. This is not possible with robust software-enabled FDE because, unlike hardware FDE, something has to be stored on the disk to enable the encryption software to decrypt it. The way PGP and other FDE software works is to use a custom boot manager to present the user with a pre-OS-boot screen to enter the encryption key which is validated against files stored on the disk. (As a detail, at the time this happens there is no OS file system available so the location of the required PGPWDE* files that are actually in the c:\ root directory is known to the boot manager by disk sector location alone). During the initial disk encryption, PGP creates a master boot record (MBR) to load that boot manager. While it is generally a very bad thing to ever lose or corrupt your MBR, there aren't enough superlatives beyond very very very bad to describe how catastrophic it is to lose the MBR on a software-encrypted disk. But how stupid or unlucky would you have to be for this to happen? Unfortunately I can't use the unlucky defence.

Its perhaps ironic that it was the act of backing up my disk that utterly corrupted it. And I was backing it up "just to be on the safe side" a week before our biggest customer conference because I couldn't afford to lose any time during the busiest period of my year. Its hard to accurately describe the sick feeling I got after the disk clone activity failed very quickly and left my primary drive unable to boot with just a sickly "MBR Boot Error" message.  This turns up a lot of hits in any popular web search engine, and Acronis and PGP feature in many of them. You can bet that most people doing that search are pretty desperate.

There are three obvious questions this raises:
  1. Why did this happen?
  2. What is the recovery from it?
  3. What should I do differently to back up my software-encrypted disk?

Why Did It Happen?

Acronis and software like it provide multiple strategies for backing disks up. For the reasons I described above, I used the disk clone approach. I connect a 2nd HDD to my laptop and just zap it every time with a clone of my primary drive using Acronis. I can then (if I need to) simply boot from the backup HDD (or switch it with the primary drive) and I'm done. Nothing could be simpler to get to a bootable backup.
In order to be able to copy all the partitions including the MBR, Acronis reboots during the process using its own boot manager to boot into the clone-operation part of the backup. It does this by temporarily replacing the MBR with its own, restoring the original MBR as part of the clone. This is a mistake I only made once.
The problem, of course, is that after the Acronis MBR replaced the original PGP one and the machine rebooted, the content on the C drive of the disk is encrypted garbage - no operating system, data or anything. And no way to decrypt. At this point, I needed to get the PGP MBR back in place or my disk is dead. And that is horribly difficult.

What Is The Recovery?

There is an answer to #2 but its not pretty. You certainly can't use any off-the-shelf Windows tools for recovering the MBR in this scenario and you need to use PGP tools. But this is not straightforward - you need to take the messed-up drive, get it fully decrypted using PGP tools and then restore the MBR using a standard Windows recovery disk. Getting it decrypted is the hard part. For all practical purposes, you need to find another machine (or disk) with the same level of PGP installed, connect your messed-up disk as a slave and then use the other machine's PGP installation to decrypt the messed-up disk. The end result of that is a still non-bootable but unencrypted disk. Recreating a default MBR after you've done this is easy - for example on Win7 just boot from a Win7 recovery CD you created earlier, open the command window and run bootrec /fixmbr. At this point, you're ready to use PGP to re-encrypt again and implicitly generate you a new MBR for BootGuard.

For more details on the recovery side of things, my colleague Olly Brand described his journey to a successful conclusion using this technique./

But better never to have to get to this point in the first place.

So what's the right way to back up a software-encrypted disk?

I have tried 3 approaches to try to preserve all the requirementss above - 2 approaches were successful and one I gave up on.
Successful Approach 1: Decrypt, clone, re-encrypt.

Easy to understand and always works but takes ages (6 hours to decrypt and 6 more to re-encrypt) so fails requirement #3 (fast).

Successful Approach 2: Use an archived backup/restore approach.
I resisted this approach for years because it slightly complicates the 'restore a bootable image' requirement but in the end this is the most practical approach with PGP encryption and its what I now do regularly. I'm still using Acronis TrueImage for backup/restore but through the more mainline incremental whole-back-up approach rather than the disk-clone utility. This creates an archive file that requires Acronis software to restore from, but it works without any PGP-related complications because there's no need to use any special Acronis boot manager or muck about with temporary MBRs.

In order to restore from such an archive you do need to have created bootable media (e.g. on a USB drive) that includes the recovery software to read and restore from the backup archive. Acronis (like other backup solutions) makes this easy so restoring from the archive is a simple 3-step process:
1) create the bootable media using whatever method the backup solution provides - in my case an image on a USB drive that delivers a bootable Acronis recovery manager. Also create a Windows recvoery CD for the last step below.
2) when a restore is required, boot from this media and restore from the backup. In my case I boot with the target to-be-primary disk, source backup archive disk and bootable USB available, booting from the USB. Then use Acronis to restore the disk (primary and boot partition) from the archive to the to-be-primary disk. There is a one further step required:
3) On reboot of the restored disk the text "bootguard..." is displayed and the boot will hang. This is because the restored MBR was created by PGP but the restored disk is not encrypted and the BootGuard boot manager cannot make sense of the files at the disk sectors it is pointing to (which it expects to be able to use to decrypt the contents of an encrypted disk). At this point the PGP MBR needs to be replaced by a default Windows MBR. This can easily be done by booting from the Windows recovery CD created in the first step, opening the command window and running bootrec /fixmbr. Finished.

All this is a lot quicker than Approach 1 but does require some preparation. It satisfies all the 3 requirements I listed above for 'simple' backup/restore although it's not as simple as the disk clone I could do before PGP.

Unsuccessful Approach 3: Create a sector-by-sector encrypted clone preserving the PGP MBR
I really wanted this approach to work but kept failing at the restore step. Software like Acronis can perform a sector-by-sector disk clone without messing up the MBR if the clone operation is initiated from outside Windows. Using the Acronis bootable media (and therefore bypassing PGP), I can initiate a disk clone at the raw sector level. As far as Acronis is concerned it's copying garbage but the end result is an encrypted clone. I can see that its a semi-successful clone by booting from the old primary source disk using the normal PGP boot manager, at which point the PGP software on the primary disk can see and interpret all the data on both the source disk and the new encrypted clone. However, when trying to actually boot from that clone I never got past:
There comes a point when enough is enough. Maybe there's something hardware-drive-specific in the BootGuard sequence which means no amount of faithful cloning can be successful but I lost interest in making that work once I had the good-enough approach #2.

Its good to be a little paranoid.

Friday, 1 November 2013

Pumpkins are Composable Modules Too

Ludwigsburgers (if thats a reasonable way to describe the people of Ludwigsburg) know how to celebrate Hallowe'en. They have the most brilliantly mad display of pumpkin sculpture I've ever seen. I don't know how this exhibition ever got the green light, but I'm glad it did. "I know, lets model all the events of the Olympic games as pumpkin sculptures!". And why not? Swimming, ski-jumping, discus, bob-sleigh, weight-lifting...its all there in the beautiful baroque gardens of Ludwigsgburg palace.

I was in Ludwigsburg for Eclipsecon Europe on Hallowe'en this year to talk about how OSGi helps complex software engineering projects manage their own success. If you want to see more about the highs and lows of WebSphere's use of OSGi, the mistakes we made and lessons we learned - and where I see OSGi heading - then here are (most of) the slides I used.

One aspect of this story that's going to be hard to improve on is the one at the bottom of slide 2. But if that longest of long-haul journeys ends up in Berlin in 2015, I'll be there...

Tuesday, 29 October 2013

Travelling Light for the Long Haul - at EclipseCon Europe

I've been travelling a lot recently and this week I'll be speaking at EclipseCon Europe, in Ludwigsburg Germany, on.....Travelling Light for the Long Haul. I quite like using long-distance travel, with all its potential discomforts, as an analogy for long-lived software projects. The more you do the travel, the more you plan ahead with strategies for making it smoother and more comfortable for yourself. Today, for example, I'm travelling economy (as always...) but writing this from a business lounge.

I'll be giving a keynote on the OSGi track and I'll start off by looking at how OSGi has enabled one of IBM's most successful software products to survive and thrive in a challenging market.

One of the attractive qualities of OSGi is its role in enabling technologies that adopt it to manage the cost of their own success. Anything that gains adoption - in technology or elsewhere - picks up baggage as a result and needs to figure out how to deal with current installations while expanding in new directions. The WebSphere platform has been around for almost as long as Java and knows a thing or two about baggage but still manages to travel to many places with just a carry-on allowance. We adopted OSGi internally 8 years ago and have gradually increased our exploitation with each passing release, most recently and deeply with the lightweight WAS Liberty Profile. It hasn't all been plain sailing and we learned from a number of mistakes made along the way. When WebSphere Application Server first adopted OSGi it had over 10 million lines of code in a modest number of huge JARs. The engineering effort to modularize that into a “sensible” number of OSGi bundles was fairly significant. We had a global development team spread across a dozen labs and nearly as many timezones, all learning OSGi principles at the same time. What could possibly go wrong? I’ll spend a little time reviewing the consequences of our bundles-first-services-later approach but our success was initially limited to having the equivalent of a well-organized and large container ship which could travel at speed but needed a pretty wide berth. Our initial investment in OSGi delivered on most of the internal benefits we wanted but failed on some of the external ones that matter to our customers.

Application Servers are used in different ways by Developers and IT Operations. Ops teams care about the overall cost, including performance and availability, of the platform and the applications it supports; Dev teams care about how quickly and easily they can create and deliver their applications and treat the server as a tool. Only some of them know or care about OSGi; multi-channel enablement and cloud deployment are the current pressures they are under. Today, WebSphere is a consumer of OSGi in two distinct fashions. Internally we learned from our earlier experiences and embraced an OSGi services model to enable us to run the same runtime just as fast but in a far more dynamic fashion: it’s how we can start/stop individual technologies of the Java EE Web Profile independently on the WAS Liberty profile, in a 50MB install with a 2-second startup while still support all our customers’ existing deployments. Externally we support both Enterprise OSGi and traditional Java EE as application programming models, on the same runtime and using the same Eclipse-based tools. Our customers who understand and care about OSGi can develop and deploy web application bundles and multi-bundle enterprise applications. Those who don’t care about OSGi benefit from it indirectly through the OSGi subsystem features that provide the server capabilities their applications use.

One of the challenges OSGi continues to face is over when to be “front of office” and when to be “back”. As the industry accelerates towards cloud, OSGi is an internal part of IBM’s strategy for high-density virtualized Platform-as-a-Service through WebSphere Liberty. Today’s cloud provisioning strategies, for example the buildpacks used by Heroku and CloudFoundry, are designed to be technology-agnostic. As a programming model for the cloud, OSGi is in a position of strength with its heavily service-oriented architecture. But in the spirit of agnosticism, one of the next steps OSGi needs to take is simply greater availability of the core OSGi framework in some of the more popular cloud platforms. Once there are more OSGi services running in those environments then the value and simplicity of autowiring OSGi services as cloud services becomes more apparent.

We’ve taken some steps in this direction already with a WebSphere Liberty buildpack for the Cloud Foundry PaaS which provides an OSGi container (using WebSphere and Eclipse Equinox) for enterprise OSGi applications – you just bring your bundles and push them to the cloud. Come and hear more at EclipseCon. I’ll be speaking on Hallowe’en…

Monday, 31 December 2012

Why isn't Software Install just 'Unzip'?

I've been a big fan of Spotify since it first launched in the UK in 2008. I like it because it does what you expect it to and has a pretty good history of providing those "why hasn't anyone done this before" features....before anyone else does. (Or at least, before anyone else does them well enough for me to want to use them). Like collaborative playlists that don't require any money to change hands to create, share and enjoy. Brilliant. And the Spotify Download store was an obvious next step and a nice poke in the eye of the increasingly distasteful iTunes Store hegemony. But this isn't an advert for Spotify....like many others I was annoyed before Christmas at the abrupt closure of the Spotify store which I was relying on for the annual update to the Robinson christmas playlist. After enjoying the Gig of the Year in July (last year's christmas present to my wife) when Springsteen was on fire in London and then had the plug unceremoniously pulled at 10:30pm (the promoter's public-spirited conscience not wanting to disturb all the hard-working families who live next door to Hyde Park), an obvious omission that needed adding was Santa Claus is Comin' to Town. So after the terse "We're currently not supporting new download purchases on Spotify", and burdened by my old fashioned belief in the moral imperative of paying for legal music downloads, I went to Amazon (turning a blind eye to Amazon's UK tax avoidance. A Guardian reader's life is full of compromises). And having just started my Christmas vacation from work, I was immediately reminded of what I had recently left behind....the question of "why isn't software install just unzip?".

Bruce's addition to my playlist was an 8Meg MP3. Having purchased it, it should have taken me a couple of seconds to download it to wherever I liked on my hard drive and then 1 or 2 more to add it to my festive playlist. But wait....I'd forgotten about the Amazon MP3 Downloader - a 2M software installer for an 8M MP3 file that surely requires no "installation". Having spent the 89p I was obviously committed beyond the point of no return and so installed the damned installer. And after a few more seconds, Bruce was nicely sandwiched between Greg Lake and Louis Armstrong.

If I were ever to look back at my last week of 2012's mail/meeting/slidedecks/wiki summaries, I know I would find an inordinate amount of discussion over how we evolve the two approaches we have of installing WebSphere AppServer software. This may address a different scale of software and consumer from an MP3 file but some of the end-user install considerations are the same - certainly for those who don't want to have to manage the lifecycle of an Installer in addition to the software it installs. For the smallest distribution of the WebSphere AppServer - the WAS Liberty Profile - we provide two quite different ways to install the AppServer software and manage its lifecycle:
  1. through IBM Installation Manager
  2. through unzip
Actually, "unzip" isn't quite this because the archive file is actually an executable JAR whose Main-Class is a simple embedded unpack-installer and so requires java.exe to initiate the archive install. (For the sake of completeness, there is a third approach to install the WAS Liberty Profile which is via the WebSphere Developer Tools for Eclipse (WDT)). The reason there are two such different approaches is because we have two fairly different types of user - one type who prefers greater control and flexibility (and favours the unzip approach) and another who prefers a more managed approached that hides complexity, for example around managing dependencies between interim fixes (iFixes). I'm often asked about the differences between these two approaches so this post attempts to describe both our intentions and some of the end-user considerations around them.

Archive Install

Archive install (a more accurate name than "unzip install") was introduced in WAS V8.5, specifically for the WAS Liberty Profile - a component of WAS V8.5. It was initially motivated by developer-centric scenarios but expanded in ambition during the development of WAS V8.5 so that by the time we shipped WAS V8.5 it was a supported means for production scenarios too. The archive file to be installed is an executable JAR whose Main-Class displays an End User License Agreement and then unpacks the JAR content to the target installation directory. Distributions that contain a production license are available from IBM Passport Advantage. Distributions that contain a no-fee developer license are also available from WASdev.net > Downloads. This is the usual starting point for both a free try-it-and-see experience and a free Developer Desktop environment because it requires no accounts to be created or contracts to be procured - its a straightforward download of a 47M jar that is then installed through running:
java -jar jar-you-just-downloaded

There is no independent installer that has to be set up first, although this route does assume you already have Java 6 SE (or later) installed. If you also need to install Java then the IBM Java Development Kits can be downloaded from here. Lets assume you downloaded and used 'java -jar' to install the WAS Liberty runtime in c:\wlp. There is a README.TXT file in the root install directory (c:\wlp) that describes the defaults for a number of directories relative to where you installed the WAS Liberty runtime. For example, wlp.install.dir is the install directory and the default user directory, wlp.user.dir, can be found at c:\wlp\usr. These are important to understand, especially from an install perspective: the user directory is where all user information, such as server configurations, resources and applications will be located. The manner in which the install is laid out on disk and in which the user directory information is isolated from the rest of the install is designed to simplify the lifecycle management of WAS Liberty and to make archive install a practical option for production environments to which service can be applied. Consider a scenario where WAS Liberty V8.5.0.0 is installed with all the defaults and two servers, server1 and server2 have been created, with application snoop.war installed on both and application creditApp.war also on server1. The default directory structure looks like this:
WAS Liberty directory structure

Actually, the etc\server.env properties file is not created by default and is only needed if you want to override an environment default for all the servers under wlp.user.dir, in which case you can create it with any text editor. To be explicit about the location of the user directory, we can create a server.env file which contains:


Lets now assume this is a production scenario (likely fronted by a Apache HTTP server with WAS plugin configured for these two servers): how would you now use archive install to move up to the next fix pack ( with minimal disruption of service? This is actually very easy. You obtain the fix pack jar wlp-base- from IBM Fix Central [http://www-933.ibm.com/support/fixcentral] and install it using java -jar to a new location e.g. c:\wlp8501\wlp. Next you override the default user directory of the new install to point to your existing wlp.user.dir to pick up all the server configurations, applications and other user-defined resources. This is as simple as copying the etc\server.env file from c:\wlp\etc to c:\wlp8501\wlp\etc. Then you just stop each running sever (e.g. 'c:\wlp\bin\server stop server1') and restart using the new fix pack level ('c:\wlp8501\wlp\bin\server start server1 --clean'). The '--clean' option is advisable on a first restart after updating binaries to flush any cached information from the previous run. Moving back to is simply the same process in reverse (stop, start This example didn't change any defaults in the WAS Liberty install. In reality, if you're planning for this sort of service methodology you'd almost certainly create your initial wlp.user.dir somewhere that wasn't a subdirectory of the wlp.install.dir.

As you can see, moving up or down fix pack levels with archive install is very straightforward. What about applying interim fixes (iFixes)? The fix pack install I just described is a side-by-side replacement of (all) the platform binaries. An iFix is a jar-cumulative update of one of more jars in the platform install. By "jar-cumulative" I mean that if the same jar is fixed for two different APARs and you want the second APAR, then it is cumulative with the first APAR - you can't take the second without the first. Each iFix is another executable JAR obtained from Fix Central, named using the scheme -WS-WASProd_WLPArchive-IF.jar , along with a ReadMe.txt containing instructions on how to apply the iFix and the list of files that should be manually backed up before applying the iFix. The result of executing the iFix JAR replaces files in the wlp.install.dir\lib directory.There are further details on this process in the WAS InfoCenter. Having applied an iFix, you'll notice a new directory wlp.install.dir\lib\fixes containing metadata about the applied fix. This is used by the WAS Liberty productInfo utility to help you identify whether specific iFixes have been applied to an install and/or to compare two different WAS Liberty installs to see how the list of applied iFixes differs between the two. This is particularly useful if you deploy applications as part of packaged servers using the WAS Liberty command 'server package' to produce a 'packaged server' archive from an existing WAS Liberty install, including the applications, server configuration and other contents of the wlp.user.dir relevant to the server. This archive can be unzip-deployed to multiple target hosts; over a period of time multiple iFixes might be applied to each instance of the unzip-deployed servers as well as to the original install. The productInfo utility can help you to manage keeping the unzip-deployed servers at the same fix level (if desired). For example, suppose you produced a packaged server1 from the scenario I described above:

C:\wlp8501\wlp>server package server1
Server server1 package complete in c:\wlp8501\wlp\usr\servers\server1\server1.zip.

This zip archive includes all of the WAS Liberty runtime and user content in wlp.user.dir\servers\server1 and shared.app.dir. This can be copied and unzipped to multiple hosts. Any future iFixes can be applied directly to each target install or could be applied to the original master install if you prefer to create a new packaged server image (now including the iFix) and redistribute/unzip on the target hosts. For example, against the WAS Liberty install:
C:\wlp8501>java -jar
Applying fix to Liberty install directory at C:\wlp8501\wlp now.
Replacing file at 'C:\wlp8501\wlp\lib\com.ibm.ws.kernel.boot_1.0.1.jar'.
Fix has been applied successfully.

You can use the WAS Liberty productInfo command to compare the contents of any two installations, including an installation in its packaged form resulting from a server package operation. For example, the following command starts with the WAS Liberty installation at c:\wlp8501\wlp (which now includes an iFix) and compares it against the earlier packaged server1.zip (without the iFix):

C:\wlp8501\wlp>productinfo compare --target=c:\wlp8501\wlp\usr\servers\server1\server1.zip
Some of the iFixes in the image at c:\wlp8501\wlp are missing in the image at c:\wlp8501\wlp\usr\servers\server1\server1.zip.

The following iFixes are in the image at c:\wlp8501\wlp but are missing in the image at c:\wlp8501\wlp\usr\servers\server1\server1.zip.
PM75505 in the iFix(es): []

All in all, the archive install procedure is a quick and easy way to get started, requires no external installer and provides a lot of flexibility around managing and maintaining WAS Liberty instances so installed. But it does not manage the install and maintenance of the Java platform used by WAS, it does not implicitly calculate any dependencies between iFixes, it doesn't provide automatically back up for files replaced when an iFix is applied and nor does it search Fix Central for available updates. For this sort of managed install capability you do need a discrete installer which brings us on to......

Installation Manager

Installation Manager (IM) is an IBM technology used to install many IBM products, including WAS since WAS V8.0. The WAS Liberty Profile was introduced into all editions of WAS V8.5 and can also be installed using IM. The initial result of installing WAS Liberty through IM is exactly the same as an archive install except that IM also lays down additional metadata that it uses to manage future feature and service updates. Installation Manager itself needs to be downloaded first, for example from the IBM Support Portal and unzipped before it can be used. The thing you download here is imaginatively referred to as the "Installation Manager installer". There was probably an IBM prize for coming up with that name. Once you've unzipped this you have the choice of using this directly as an "install kit" for scripted/silent installs of the target product (e.g WAS) or using it to fully install Installation Manager, including the IM user interface, and then using the result of that to install and service WAS. Details about the differences in these two approaches are in the IM InfoCenter but for the sake of simplicity in this article, I'll assume the latter.

IM installs and services product offerings like WAS from one or more installation repositories, the URLs for which needs to be configured in IM. Each edition of WAS (e.g. WAS Network Deployment edition) actually consists of a set of installable offerings, each with its own installation repository. The repositories themselves are available in a number of different forms depending on the product offering - for example the repository for the WAS ND offering is provided at three locations: on the WAS ND product media, downloadable from IBM Passport Advantage and in an IBM-hosted web-based repository. The complete list of all installable WAS V8.5 offerings (across all editions) can be seen in the WAS InfoCenter. You can also create custom enterprise repositories containing the service levels and iFixes desirable for an enterprise using the Packaging Utility as described here. IM installs WAS Liberty as part of one of the higher-level product offerings that include Liberty. You can see in the WAS InfoCenter that for V8.5, this includes:
  • com.ibm.websphere.BASE.v85
  • com.ibm.websphere.BASETRIAL.v85
  • com.ibm.websphere.EXPRESS.v85
  • com.ibm.websphere.EXPRESSTRIAL.v85
  • com.ibm.websphere.DEVELOPERS.v85
  • com.ibm.websphere.DEVELOPERSILAN.v85
  • com.ibm.websphere.zOS.v85
  • com.ibm.websphere.ND.v85
  • com.ibm.websphere.NDTRIAL.v85
To use IM to install WAS Liberty from a hosted web repository, you start IM and configure it, using the IM File->Preferences->Repositories menu, to point to the repository for the desired offering. The URLs use the following convention:
for example

Choosing the IM "Install" option will offer the choice to install, in this example, the base edition of WAS V8.5. As part of the install you choose whether to include the WAS full profile (with various options) and/or the WAS Liberty profile.
Installation Manager install of WAS V8.5 showing WAS full and Liberty profiles

By default, IBM Java 6 is also installed with WAS when using IM (unlike the archive install route). IBM Java 7 can be optionally installed as part of WAS if the following repository ID is added to IM:

While there is more overhead to installing WAS Liberty by this route rather than the archive install, the benefits of IM become more apparent once you need to apply service to the installed product. When you start IM and choose the "Update" option, IM searches the configured repositories (IBM hosted repositories or custom enterprise repositories) for all fixes applicable to the products installed by IM and presents a list of those that have not yet been applied - unlike the archive install route you don't have to know about these a priori.

Installation Manager simplifies the application of service to installed packages

If there are any pre-requisites to this iFix, IM will resolve these and by default it back up any updates so they can be undone - all capability beyond archive install. IM can only manage installations of WAS Liberty that have been installed by IM. If you use IM to install WAS Liberty, develop and test an application using that install, use the Liberty runtime server package command to create a packaged server with your application resource and server configuration, and then unzip deploy that on a remote host, you will not be able to use IM to apply service directly to the unzip deployed packaged server. For this scenario you can either:
  1. use IM to apply service to the original install and then repeating the server package and unzip deploy of the result to the remote host
  2. use the methods described in the archive install section above to apply an iFix directly to the unzip deployed server on the remote host.

In summary, "unzip" can be good enough for software install. But after a few service updates have been applied, the complexity of managing these manually may start to make a managed installer look more cost effective. Both approaches are valid for WAS Liberty in production - I hope this post helps explain the pros and cons of each.

- Ian.

Saturday, 15 October 2011

WAS V8.5 Alpha and the WebSphere Liberty Profile

Earlier this year I took on the role of Chief Architect of the WebSphere foundation and have been busy with the team since then working on the development of our next release. We launched the first fruits of our labour last week through our new WASdev community site: the WAS V8.5 Alpha. This is an early glimpse of what's coming next in WAS and it is heavily focussed on making the WAS runtime and tools the best environment possible for developing and testing web applications. At the heart of the Alpha is a new dynamic profile of the WebSphere runtime that we've called the WAS Liberty Profile. Rather than being a static profile of runtime features, Liberty adapts to the requirements of the application and ensures - at a really fine-grained level - that only the necessary application container functions are started. It gives you the freedom to deploy web applications with wide-ranging requirements and provides all the pieces they need (like security, transaction management, connection pooling, persistence through JPA or JDBC, and so on) without pulling in any more than they need. And it provides fidelity with the full-profile WAS runtime to which applications are deployed in production because the containers and services started by the Liberty profile are the same.

One of the key innovations in Liberty is a new kernel that processes the specific "features" required by an application to load the required container(s) and WebSphere platform services: WAS has been an OSGi-based runtime for many years (since WAS V6.1) but, with the Liberty profile in the WAS V8.5 Alpha, we've moved beyond simply modularity and gone to town on exploitation of dynamic OSGi services. This is what delivers the tiny runtime footprint and a WAS server that starts up in a couple of seconds. And this is only one aspect of what's new for developers - we've also dramatically simplified server runtime configuration so that a test/debug WAS server instance can be configured easily, either inside or outside an Eclipse environment, with a single XML file covering all aspects of the server, the applications and the resources required by the applications. If you've ever looked at a WAS configuration and wished you could treat it more like a development artefact - storing the configuration in an SCM system, versioning or diffing it, sharing it between developers - well now you can. While the server config can be as simple as a single XML file, there are also flexible ways to compose a configuration from fragments that are aggregated through include statements.

The WASdev home page lists "8.5 Reasons why the WAS Liberty Profile is awesome". For more details, including an overview of the new simplified server config, take a look at the Introduction to the Liberty Profile article I posted to WASdev.

And did I mention that its free? And so, now, are our most popular web development tools which we're making available as Eclipse features (in addition to being bundled inside RAD). You can get all this from WASdev downloads page.

But enough talk....lets see how quickly we can download the tools, the WAS server, configure a server instance, install a simple app and bring the whole thing up. I'll assume you already have the Eclipse IDE for Java EE but if not you can get that from the Eclipse site. The only other thing you need is an IBM ID for the downloads - if you don't have one then simply register for one here.

OK. Open up your Eclipse IDE (3.7 or 3.6.2), start your stopwatch and follow these steps:
  1. Go the WASdev download page to install the WAS Liberty Profile tools into Eclipse. This gives you a rich set of web development tools along with the ability to launch and control instances of the WAS Liberty Profile server. For Eclipse 3.7, its as easy as dragging the install button from that page onto your Eclipse IDE title bar.
  2. Accept the license terms (I'll take the liberty of assuming that, under these "beat the clock" conditions, you have already squared this with your friends in the legal department) for the install to proceed. The total size of the tools being added to your Eclipse environment here is less than 3M.
  3. Restart Eclipse. The tools are now installed - we can use these to now install the WAS server runtime itself for use testing and debugging applications. This is going to download and install a whole WAS Liberty profile server. Sounds scary? It isn't - the whole runtime is just another 25M and there's no additional installation program required.
  4. In the Eclipse workbench, open the Servers view. Right-click the Servers view and select New > Server. Under the server type list, expand IBM then select the WebSphere Application Server v8.5 Alpha Liberty Profile server type. Click Next.
  5. The WebSphere Runtime Environment page is displayed. In the Installation folder section, click download and install. Choose the WebSphere Application Server V8.5 Alpha site and enter your IBM ID and password and click Next. Accept the license terms for the runtime. (You'll have already mentioned this to the same lawyer you talked to about the tools license). Click Next.
  6. In the Download and Install page, specify the installation target folder for the WAS runtime to be installed to. (Just 25M, remember).
  7. Click Next. WAS is now installed. (Note that, if you just want to install the WAS Liberty profile runtime independently of the tools for test and debug puposes, then you can skip all the steps above and simply download the WAS Liberty Profile runtime directly from the WASdev downloads page and unzip it).
  8. There's a pre-configured defaultServer instance but lets go all the way and create our own server instance using the New button. Call it whatever you like e.g. firstserver. Click Finish. You now have a WAS server instance to which you can install and test web applications. On your filesystem, there will be a firstserver directory structure in the usr/servers directory of your new WAS install.
  9. Now we need to deploy a web app to this server and run it. You can download this simple StopWatch WAR file and import it into Eclipse.
  10. In Eclipse: File > Import WAR file. Click Next. Enter the location of the WAR file on your filesystem and make sure the Target runtime is set to WebSphere Application Server V8.5 Alpha Liberty Profile. Click Finish.
  11. In the Project Explorer, right click on the new StopWatch project and choose Run As > Run on Server.
  12. The WebSphere Application Server V8.5 Alpha Liberty Profile should already be highlighted. Click Finsh.
  13. Stop the Clock!
If you're looking for the fastest, cheapest and easiest way to develop web apps for WAS then this is it.

I've been busy spreading the message about Liberty since we started the WAS V8.5 Alpha - you can take a look at my slides from the WebSphere Technical Conference in Berlin, where we launched the Alpha:

And if you happen to be in Mumbai on Oct 19 or 20, come and hear me talk about Liberty at the IBM Software Universe.

Saturday, 18 June 2011

God Dad, You're Soooo Embarrassing

...the words every proud parent longs to hear, to know they've succeeded in keeping their offspring alive long enough for them to become a teenager. I heard this as I walked through the door yesterday and was doubly pleased when I found out why. The video I made a little while ago to kick off the new Enterprise OSGi YouTube channel has successfully attracted enough attention for my daughter to be teased at school, once again proving the value and reach of social media. Result.