Anyway, I turned on the new machine this morning, and selected the Restore from Time Machine option. It chugged away for an hour or whatever, but by about 9:30 AM I was sitting in front of a near replica of the other machine. Pretty cool.

I did have to install XCode and Dashcode again.

And I had to re-acquaint my bluetooth phone with the new Mac. And re-authorize my iTunes music.

And for some reason, I had to reset my USB drive as the Time Machine volume (and it choose a new backup database folder, so it’s going to start over with a full backup).

Technorati Tags: apple, leopard, mac

The previous problem occurred when I was running a build or other long, CPU-intensive operation. Sometimes, the machine would just reboot suddenly. And it was always when I was away. I had no luck with any of the usual fixes suggested by Apple or found on the web. I had figured out that it was probably related to the screen saver coming on, but it was impossible to reproduce this for anyone.

But something about Time Machine allowed me to reproduce this more repeatably: lots of CPU, generating lots of heat, coupled with intense graphics == reboot.

The folks at the FlatIrons Apple Store were really great, especially given the madhouse produced by the holidays. I described both problems and they just decided to replace the logic board. They ordered the part, and I was without my machine for only a day-and-a-half. It seems like this has done the trick. Hurray!

Technorati Tags: apple, mac, reboot

The comments last night made it clear what we already knew about Paul: while he did work hard to meet all the Eagle requirements, he did not have to work to become the kind of person you expect an Eagle to be. He just is that kind of person.
He worked hard for this, obviously, but all that work served only to validate who he already was.

Congratulations, Paul!

Technorati Tags: eagle, bsa

Hey! where’s the text? The rest of us (at least Firefox and Safari) were seeing something more like this:

Much better, but what’s going on?

I quote code in a <pre class="code">....</pre> block. And my style sheet contained:

.code {
    font-family: courier, monospace;
    margin: 10px 20px;
    padding: 3px 8px;
    border: 1px dashed #999999;
    overflow:auto;
}

It looked to me like IE was robbing space from either the text height or the padding to cram in a scrollbar. And since that made the resulting area for text too small, it also added a (way too small) vertical scrollbar, too. The result is non-functional in any real sense.

A bit of searching and I am drawn to this comment from the CSS 2.1 spec:

In the case of a scrollbar being placed on an edge of the element’s box, it should be inserted between the inner border edge and the outer padding edge. Any space taken up by the scrollbars should be subtracted from the computed width/height, thus preserving the inner border edge.

As I read this, the scrollbar needs to fit inside the element border, and thus subtracts from space that can be used for the content. This does preserve the overall size of the element no matter what type of scrolling might be used (or not). But in my context, it seems wrong. It also means that IE is probably right here (which also goes against my intuition :).

Wrong or not, it is certainly not what I want. So I did more hunting and experimenting, and I found only two solutions that I really thought worked.

First, I could simply increase the padding-bottom to make room for the scrollbar. This solution is nice because it keeps the browser-specific hacks out of the stylesheet. But I don’t like the way this looks when there is not a scrollbar, because there’s extra blank padding at the bottom.

So the only real choice for me was to add some browser-specific thing to the stylesheet. What I ended up with is based mainly on
this post, and looks like this:

.code {
    font-family: courier, monospace;
    margin: 10px 20px;
    padding: 3px 8px;
    border: 1px dashed #999999;
    overflow:auto;
    /* add padding in IE (which supports expressions) to make room for the scroll bar */
    padding-bottom: expression(this.scrollWidth > this.offsetWidth ? "19px" : "3px");
}

This seems to be working reasonably well, as it does not mangle the size of the element when there is no need for a scrollbar, and it bumps up the bottom padding to make way for IE’s scrollbar.

There are some downsides, like the expression is probably being over-evaluated, and it relies on a proprietary IE feature (expressions), so I may reevaluate this in the future. But for now at least my blog is (more) readable for IE users.

Technorati Tags: css, ie, scrollbar

But backups aren’t backups unless you test that you can recover files (and you know how to do it without having to mess around when the time comes). So today, with a day’s worth of backup history, I decided to play with Time Machine.

I flipped back and forth thru time-cheese, and after a few seconds the world disappeared. Oh wait, it was just my MacBook Pro rebooting. Bong. Apple. Spin wheel. Login.

Nothing in any of the logs (there wasn’t time) - just a hard, fast, shutdown.

So, I did what any self-respecting software engineer would do. I tried it again. Same result. I have now hard-rebooted my machine like 10 times via Time Machine (I don’t recommend it).

If recovering a file risks a hard shutdown, and I can’t resolve this - I will just have to go back to rsync.

Technorati Tags: apple, leopard, mac, reboot, time machine

Mail

Mail still doesn’t obey the Hide checkbox when it is started at login. Just what is so hard about this? I find this really annoying. My son did figure out a rather clever “cheat” - he starts Mail in a different Space, so it is not really hidden, but just looks that way (but I’m not using Spaces).

And Mail still doesn’t grok the age-old function “go to next unread message” (the shortcut for this should be “spacebar”).

On the plus side, Mail will finally display a proper count of unread messages in the Dock icon (rather than only counting messages in the inbox).

And in the bug arena, there is something seriously wrong. I occasionally see a popup from Mail saying it can’t identify the certificate for pop.gmail.com.

When I inspect the certificate, it is the one for my work email account. So something is seriously broken in Mail’s multiple-account settings or fetching.

Mail / RSS

I never really have got along with RSS in the browser - never seemed right to me. So I thought having it in Mail was a good idea - it’s really where I’ve wanted most of my feeds all along. But Apple’s implementation is just not “there” yet. It would probably work for someone with just a handful of feeds, but not for me.

First, there is just no way to import a set of feeds from anywhere but Safari. I found these instructions and got my OPML feeds imported (via Firefox to Safari to Mail - ugh), but they lost their folders. It took me a while to figure out that you can create folders for feeds (you just create a new mailbox), and then I was on the way to rebuilding my folders, and importing feeds into them. But along the way I figured out that you can’t sort or arrange anything. So I was stuck thinking I’d have to rename my folders so they’d be “aNews”, “bTech”, “cBlogs”, etc. Gack. And things were just getting crowded in the sidebar - what with my inboxes and all my mail folders, there just wasn’t room for a dozen more folders for feeds.

So I came to my senses and went back to using the most excellent NetNewsWire.

Safari

I think the best thing in the new Safari is actually the new eye candy in the Find feature. I was always having trouble seeing the default (subtile) blue highlight. Apple has now made it easy to see the find results on any web page (regardless of color scheme).

The history menu also now has two new options: Reopen Last Closed Window and Reopen All Windows from Last Session. Fantastic for those of us who sometimes accidentally close or quit.

The other two things I like come with the debug menu. First, they moved Open Page With and User Agent from the bottom to the top of the menu. And the new Web Inspector is great (kinda like Firebug, but more Apple-y). I especially like the way it rolls up CSS styles and shows Metrics (margins/borders/padding/size).

Spotlight

Spotlight just seems to work better. First, there’s new stuff contributing - most notibly Web History. Spotlight searches now include your Safari cache, so you can find that site you visited last week.

Also, they’ve made the “Top Hit” automatically selected (rather than messing around with the Command key). I can type, for example, C-Space, sys, Return and get System Preferences. It’s just better.

iCal

Well, finally you can set a default alarm for new events, although there’s no way that I can see to set the default sound to anything but Basso. And the default alarm doesn’t seem to apply to notifications created from Mail. So some progress here, but still a way to go.

Stacks

I actually like this. I used to keep folders in my Dock for Applications and Downloads, but this just works better. I don’t really like the Fan view, but since my Dock is on the side, all I and use is Grid anyway.

Quick Look

This is going to be really handy, especially now that I’m starting to learn the shortcut (Command-Y). It’s way faster than launching an app just to see what’s in the file. Next, I’m going to have to try a code syntax highlighter.

Random Issues

I have my desktop background set to display photos from a folder. Sometimes, first thing in the morning, my second monitor comes up with the default Leopard background. If I bring up preferences, it does think that it’s displaying the photos. If I wait a half-hour (when the photo is scheduled to change), it gets with the program.

I have noticed that the login dialog doesn’t get focus when I wake from sleep. So I type my password and nothing happens till I click in the login dialog. The worst part is that I sometimes have found bits (or all) of my password sitting in other dialogs or documents - which means those dialogs actually had keyboard focus while they were supposed to be locked out. Scary.

I am disappointed that WiFi with WPA is still not right. I have never been able to hold a connection to our work WAN for more than about 15 minutes (it had steadily grown from 5 to about 15 over several Tiger updates). Now, it seems we’re back to 5 minutes. My home network (Linksys) is not as bad as at work, but it does drop connections occasionally, too. I have no data, but it does seem worse than with the last update of Tiger. I never had these problems with my PowerBook.

Technorati Tags: apple, leopard, mac

However, this is not the first time we’ve been here. But it is hopefully the last.

We are putting in place a test environment to keep these leaks plugged. Actually, to be more precise, we’re adding some instrumentation to our integration test suite that drives tests via the IDE. So this test will behave just like a developer - although it’s a rather anti-social and uninteresting developer :).

Note that this technique is not restricted to WLP or redeployment testing, but hopefully you can find it useful for other situations.

Counting ClassLoader Instances

The basis of this technique is that we know that the leaks we are interested in happen to always hold on to the application or webapp ClassLoader. Further, we also know that these ClassLoaders are instances of GenericClassLoader (for the app) or ChangeAwareClassLoader (for the webapp).

So, our instrumentation relies on being able to count the instances of these classes. Fortunately, JRockit can do this easily, without any extra messing about with the startup settings or whatnot.

JRockit comes with this handy jrcmd command. It lets you interact with a running JRockit JVM, like for example the WLS server. You can use it to do all sorts of interesting things:

• find out the JVM’s command line (how it was started)
• print thread dumps
• change the verbosity of the JVM on the fly (things like garbage collector stats)
• trigger a garbage collection
• print a GC report
• print diagnostics about memory usage

All sorts of cool things - and none of these require you to change anything about your running process. No command line switches or anything messy like that.

So, we are looking for a count of the instances of our ClassLoader objects. We do that by running:

jrcmd <wls server pid> heap_diagnostics

(You get the wls server pid by running just jrcmd - it will print the process ids of running JRockit instances.)

If you run that, you’ll see tons and tons of output - including a listing of every class loaded by the system and how many instances there are. So all we need to do is search for the class we are interested in:

jrcmd <wls server pid> heap_diagnostics | grep '%.*GenericClassLoader

That will grep for the one line we are interested in (turns out it contains a percent sign; why is not important here). We’ll get output something like this:

     0.0% 10k      150    +10k weblogic/utils/classloaders/GenericClassLoader

In the above, we see that there are 150 instances of the GenericClassLoader.

To get a more accurate count, we also want to trigger a full garbage collection and finalization before counting anything. We can do that with jrcmd also:

jrcmd <wls server pid> runfinalization

Using this information, here’s what we are doing for WLP testing:

1. Start the server, with the application undeployed.
2. Run finalization and count the GenericClassLoader and ChangeAwareClassLoader instances. This gives us a baseline for the number of instances used by the server itself.
3. Deploy the application and use it (i.e. wander through the portal in a browser)
4. Count instances again.
5. Redeploy the application, use it, and recount.
6. Repeat the redeploy step several times.
7. Undeploy the application.
8. Count instances.

All these counts (and additionally the [re-]deploy times) are all being recorded and analyzed. The most important metric is that the first and last counts are the same - after the application is undeployed, the ClassLoaders used by the application should be gone. We are collecting the intermediate results because we expect they might be useful in failure analysis.

The one last step is that we are running the JVM instrumented for YourKit, and configured to dump a memory snapshot on exit. So if the ClassLoader counts don’t look right, a developer will have the memory snapshot for analysis, without having to rerun the whole test suite.

Hopefully some of this information can be useful if you have leaks you are hunting.

Technorati Tags: JRockit, leak, memory, portal, YourKit, WebLogic

I generally take the opportunity of an OS upgrade to clean house, and I did it this time with Leopard again. So I made good backups, then did a clean install (not an upgrade). I then spend a few days figuring out exactly what I need from the backups and moving that over. This has several side-effects that I like. It cleans out the junk: old applications, prefs from apps I tried but no longer use, etc. It also gives me a fresh look at the new features, since they are not hidden by my old prefs or hacks.

Backups First

The first step is the backups. I have been using a script (fired from a calendar item) for a while now to backup (via rsync) some critical pieces of my home directory to the Linux box in my office. But that just wasn’t going to cut it for an OS upgrade. I wanted a full, bootable backup of the entire machine.

I got a USB drive and formatted it the same as my main drive (Mac OS Extended Journaled).Then, open Get Info on the drive and uncheck the Ignore Ownership on this drive (at the bottom).Then, use rsync to do a full backup. This will take quite a while, and works best if you close everything (ilke Mail, etc) first.

# turn off spotlight first
sudo mdutil -i off /Volumes/BackupDrive
# backup everything
sudo rsync --one-file-system --recursive --links --perms --times --group --owner --extended-attributes --verbose --progress --delete / /Volumes/BackupDrive
# turn off spotlight again, since we just dropped new index files over there
sudo mdutil -i off /Volumes/BackupDrive

I’m not sure how necessary the spotlight thing is, but I didn’t like all the backup versions of apps and stuff showing up in Spotlight searches.

Next, you need to make the drive bootable:

sudo bless --folder /Volumes/BackupDrive/System/Library/CoreServices

Great - now for the part that most people skip: Test the backup! I shut down the machine, then rebooted holding the Option key. I booted with the USB drive, and checked that everything looked OK. So now I know that if something goes drastically wrong with the upgrade, I can at least get some work done.

Planning

The next step, since I was doing a clean install, was to make a few notes. I scratched out a list of the files I needed, another list of apps and settings that I used every day, and another list of apps, settings, and other things that I wasn’t sure about. Then I spent a few minutes making sure I had links to installers, notes containing license keys, etc.

I also exported backups of things like Address Book, iCal, Safari bookmarks, Mail, and Keychain.

Install

I only encountered one hiccup in my plan. I knew I was going to do a full new install, but I also plan to do an upgrade or archive install at home (to preserve the sanity of my wife and kids). So I was going to try these first on my laptop, then blow everything away and do my clean install.

So I waited forever for the install DVD to verify (don’t need to do that again, thankfully), then proceeded with an upgrade install. But the progress bar was telling me it would take around 4 hours. I didn’t have 4 hours to blow on this experiment, so I bailed out and just went straight to the clean install.

ReCreate

The recreation of my “stuff” went pretty smoothly. I went through my checklist and started copying files, apps, and settings. I did this one at a time, just to keep things clean. For Mail, I did migrate over my old mail forders, accounts, and preferences - but then made sure to go check out the new prefs to see what’s new and what I might want to change.

Right away, I got Mail, Adium, iCal, NetNewsWire and Eclipse set up. Couldn’t wait on those. But other apps are being moved over on a slower pace. I don’t want them just because I used to use them - I want to make sure I really need them.

Impressions

Things seem a bit faster. I’m not sure if it’s just that I have not (yet) bogged the thing down with hacks, or if Leopard is actually a bit zippier. Or maybe it’s the new eye-candy acting as a sort of endorphin or whatever.

I’ll blog other impressions of new feature “likes and gripes” later.

Technorati Tags: apple, leopard, mac

Note that many of these leaks were never released, but were only present in internal revisions. But the ones that did make it out are being back-ported to patches for 10.0 and 9.2.

I’ve also kind-of-categorized them into a few groupings, where similar patterns have emerged. Some did not fit clearly into just one category, but the breakdown should help other leak-hunters in their quests.

Clean Up After Yourself!

This class of problems is caused by creating references to objects without clearing them when you are done with them. The Thread and ThreadLocal issues I discussed in earlier posts generally fall under this category.

Timers not released: If an app created a Timer (via commonj.timers.TimerManager), and the timer did not stop before the application undeployed, it was not cleaned up (and thus leaked). The culprit was a misused WeakHashMap. The Timers were stored like this:

WeakHashMap timers = new WeakHashMap();
...
timers.put(theTimer, theTimer);

The map was just being used as a Set, but the insertion of the timer into the map value ensured that the same value in the keys were never unreferenced, thus loosing the entire benefit of the Weak keys in the first place (see the “Implementation note” in the WeakHashMap javadoc). The fix was simple:

timers.put(theTimer, null);

WSEE Runtime MBeans: WSEE (the web services stack) was creating MBeans to hold the configuration of handlers. They were not cleared when the application was undeployed, which in itself represents a minor leak. Unfortunately, the MBeans also (indirectly) held a reference to the Application’s ClassLoader - turning a small problem into a rather large one.

GroupSpace use of Beehive: GroupSpace was calling Beehive’s beginContext without a matching endContext. Behind the scenes, Beehive created some ThreadLocals, but it does the right thing and removes them when (if) you call endContext. This was fixed by balancing the context usage within a request.

Portal Framework service manager: - Internal class in the System ClassLoader has a Map that was holding references of services to be used. One of the services was loaded by the Application’s ClassLoader. This was fixed by unregistering services when the application was undeployed. Lesson: Keep track of what you are using and let it go when it is no longer needed.

Lesson: The Garbage Collector is not your Mom - you have to clean up after yourself around here.

Hold Me Tight!

Where the above references were necessary but simply not cleaned up, this grouping is more caches that are holding unnecessary or extra information.

Log Buffer: WLS Logging keeps a buffer that holds the last 50 or so log messages for efficient display by console. But the implementation was holding the application-scoped logger objects (rather than simply holding the text it needed to display). The fix was to copy information from the application-scoped objects to generic system-scoped objects for the buffer.

Resource initialization optimization: The ResourceBase class that is the base class for our Entitlement resources, was keeping a collection of previously created objects. It then used data from these objects to initialize fields in new instances referring to the same resource. But our Entitlement resource objects were application-scoped, and thus this collection caused leaks. We fixed this by overriding the base object with our own intermediate implementation. We kept the caches, but wrapped the objects in Weak and Soft references. This is not an ideal solution, so we have planned to revisit the whole design in a future release.

BeanELResolver cache: The javax.el.BeanELResolver (used when EL is used in a JSP) caches bean properties in a static Map. Since these bean objects are generally webapp-scoped, there is a leak. We fixed it the same way Glassfish fixed it by clearing the map on undeploy. Since BeanELResolver does not have a public method to clear the map or uncache entries, this had to be done by reflection of the private map field (ugh). The interesting bit is that BeanELResolver does have methods to provide the close/clear function, but they are private (and never referenced).

Commons Logging in System Classpath: Commons Logging holds a map of loggers, which it may load via the application or webapp ClassLoader. Thus simply having the commons logging jar in your System ClassPath will trigger a leak (if logging is used from an application - as it is with Beehive). For us, the jar was added to the system classloader inadvertently (via jar manifest entries), so the fix was to simply remove it. However, we really can’t do anything about it if a customer decides to stick commons logging in their classpath.

Glassfish JAXB: The Glassfish implementation of JAXB caches some reflection lookups. It has a WeakHashMap with Class objects as keys. Good try, however the values in that map are Constructor objects, which themselves hold an instance to the Class (which therefore means the keys are strongly held, and the Map’s weak keys can never be released). We fixed this in our local version by wrapping the Constructor in a WeakReference. We filed the ssue with Sun, and they adopted our fix in version 2.1.6.

LeaseManager holding a TimerManager: The LeaseManager (a new internal class) was holding a static reference to an app-scoped commonj.timers.TimerManager. This one was fun because it required tripping a race condition in order to trigger the leak. Also, in addition to being a leak, this was an error because the static reference tied the LeaseManager to a single deployed instance of a single application, so it would not work properly with multiple applications. We fixed this by looking up the TimerManager as needed (from JNDI, like you’re supposed to do). Yet another example of premature optimization (i.e. presuming that the JNDI lookup was too slow).

Lesson: You gotta know when to hold ‘em; Know when to fold ‘em.

The Ties that Bind

I’ve covered the Thread and ThreadLocal issues previously, but here are the details to complete the list.

ThreadLocals in Portal Framework: Some of the Framework JDBC code was using ThreadLocal but not cleaning up at end of request. This was fixed by simply removing the ThreadLocal (in a finally block) at the end of the request

new Threads in JSP compiler: See the previous post. It turned out that the thread was unnecessary: a leftover from when the code was ported to the server from a different environment. So it was just removed.

ThreadLocals in Content Management: Content was using ThreadLocal and not removing them after a request. The ThreadLocals were added as a (possibly unnecessary?) performance optimization, and the fix was simply to remove them.

Lesson: Don’t let your code get all tied up in knots - unravel those threads.

Final Thoughts

If you’ve taken the time to look at these issues, you can see that the problems spanned across multiple subsystems: WLP, WLS, WLW, Apache, and Sun. It was quite a project, and quite a fun challenge to track all these down.

Next, I’ll outline how we intend to keep this from happening again, and show you some tricks for your own memory leak testing.

Technorati Tags: leak, memory, portal, WebLogic

First, you can cache some value (presumably expensive to compute) and reuse it later. Since the value is associated with the Thread, it is thread-safe (baring any issues with the attached objects themselves). This makes the ThreadLocal an important tool in performance optimizations.

Another good use of ThreadLocal is to hold some context-related information. For example, maybe you need some data that is on the HTTP Request (like URLs, attributes, query parameters) in order to make an authorization decision for access to certain data. Normally, this would mean that all that data (or the Request itself) would need to appear on the authorizer API - and on any API that called that one - and on any API calling those APIs, etc. Or, you could toss the Request data into a ThreadLocal and retrieve it in the authorizer. It becomes context for the call to the authorizer. This is another really useful technique for loosely tying subsystems together.

But in a server environment (like WebLogic), the Threads are usually pooled and reused for request after request. The ThreadLocal is associated with a Thread, and the Thread is associated with a Request - so that is good. Except that the Thread is only temporarily associated with the Request.

So, if ThreadLocal objects are not managed correctly (such that their life-cycle matches that of the Request), you can get all sorts of evil things happening. First, if you were using ThreadLocal for performance, it just might not work since one user session might be handled by many different Threads. Worse, you might leak data from one user to another (when a user’s Request reuses a Thread containing another user’s data).

And in the cases I saw, if that ThreadLocal holds objects that otherwise should be garbage collected, you will get a leak.

The solution is to ensure that your ThreadLocal usages are properly managed. The ThreadLocal should have the same life cycle as the Request, and you should use ThreadLocal.remove() – preferably in a proper finally block - to clean up at the end of your Request.

Technorati Tags: leak, memory, portal, ThreadLocal, WebLogic