Using requestAnimationFrame to Optimize Dragging Events

requestAnimationFrame is an API that was originally created by Mozilla but has found its way into Chrome and I think it has huge, huge implications for user interface. A lot of the examples I’ve seen have talked about how you can use it to optimize animation, which makes sense. The basic premise of requestAnimationFrame is that instead of trying to move things out of sync with the browser, you get a hook directly into the browser refresh/redraw and on that refresh you can tell the browser to do something specific with requestAnimationFrame. So it’s sort of like a queuing system whereby you tell the browser what you want to happen, and when it goes through its next round of repaints, it executes what’s in that queue. This makes perfect sense for animation since the animation can’t move any faster than the repaint cycles of the browser. But it also turns out it makes great sense to use it for dragging events as well.

The Problem:

For me, the problem manifested when I was working on making the Brackets sidebar resizable. Like with a lot of resize events I was doing the resize on a mousemove event, so I’d track the mousemove and then resize the sidebar accordingly as the mouse moved. In theory it seemed like a decent way to do things until I opened it up in the Chrome Developer tools:

Holy repaint batman!

That screenshot may be a bit confusing, but basically, every mouse move there are a TON of “Recalculate Style/Paint” events happening. That means every single mousemove the browser is trying to repaint and so each mousemove event takes about 50 milliseconds to process. That may not sound like much, but think about how often mousemove fires. What’s worse is that because of how the browser redraws, even though it’s trying to redraw everything on mouse move, it can’t actually do that because the mousemove events are happening faster than the browser can redraw. Here’s the code that’s doing all that repainting:

    /**
     * @private
     * Sets sidebar width and resizes editor. Does not change internal sidebar open/closed state.
     * @param {number} width Optional width in pixels. If null or undefined, the default width is used.
     * @param {!boolean} updateMenu Updates "View" menu label to indicate current sidebar state.
     * @param {!boolean} displayTriangle Display selection marker triangle in the active view.
     */
    function _setWidth(width, updateMenu, displayTriangle) {
        // if we specify a width with the handler call, use that. Otherwise use
        // the greater of the current width or 200 (200 is the minimum width we'd snap back to)
 
        var prefs                   = PreferencesManager.getPreferenceStorage(PREFERENCES_CLIENT_ID, defaultPrefs),
            sidebarWidth            = Math.max(prefs.getValue("sidebarWidth"), 10);
 
        width = width || Math.max($sidebar.width(), sidebarWidth);
 
        if (typeof displayTriangle === "boolean") {
            var display = (displayTriangle) ? "block" : "none";
            $sidebar.find(".triangle-visible").css("display", display);
        }
 
        if (isSidebarClosed) {
            $sidebarResizer.css("left", 0);
        } else {
            $sidebar.width(width);
            $sidebarResizer.css("left", width - 1);
 
            // the following three lines help resize things when the sidebar shows
            $sidebar.find(".sidebar-selection").width(width);
            $projectFilesContainer.triggerHandler("scroll");
            $openFilesContainer.triggerHandler("scroll");
 
            if (width > 10) {
                prefs.setValue("sidebarWidth", width);
            }
        }
 
        if (updateMenu) {
            var text = (isSidebarClosed) ? "Show Sidebar" : "Hide Sidebar";
            $sidebarMenuText.first().text(text);
        }
 
        EditorManager.resizeEditor();
    }
 
    /**
     * @private
     * Install sidebar resize handling.
     */
    function _initSidebarResizer() {
        var $mainView               = $(".main-view"),
            $body                   = $(document.body),
            prefs                   = PreferencesManager.getPreferenceStorage(PREFERENCES_CLIENT_ID, defaultPrefs),
            sidebarWidth            = prefs.getValue("sidebarWidth"),
            startingSidebarPosition = sidebarWidth;
 
        $sidebarResizer.css("left", sidebarWidth - 1);
 
        if (prefs.getValue("sidebarClosed")) {
            toggleSidebar(sidebarWidth);
        } else {
            _setWidth(sidebarWidth, true, true);
        }
 
        $sidebarResizer.on("dblclick", function () {
            if ($sidebar.width() === 1) {
                // mousedown is fired first. Sidebar is already toggeled open to 1px.
                _setWidth(null, true, true);
            } else {
                toggleSidebar();
            }
        });
        $sidebarResizer.on("mousedown.sidebar", function (e) {
            var startX = e.clientX;
            $body.toggleClass("resizing");
            // check to see if we're currently in hidden mode
            if (isSidebarClosed) {
                toggleSidebar(1);
            }
 
            $mainView.on("mousemove.sidebar", function (e) {
                var doResize = true,
                    newWidth = Math.max(e.clientX, 0);
 
                // if we've gone below 10 pixels on a mouse move, and the
                // sidebar is shrinking, hide the sidebar automatically an
                // unbind the mouse event.
                if ((startX > 10) && (newWidth < 10)) {
                    toggleSidebar(startingSidebarPosition);
                    $mainView.off("mousemove.sidebar");
                    $body.toggleClass("resizing");
                    doResize = false;
                } else if (startX < 10) {
                    // reset startX if we're going from a snapped closed position to open
                    startX = startingSidebarPosition;
                }
 
                if (doResize) {
                    // if we've moving past 10 pixels, make the triangle visible again
                    // and register that the sidebar is no longer snapped closed.
                    var forceTriangle = null;
 
                    if (newWidth > 10) {
                        forceTriangle = true;
                    }
 
                    _setWidth(newWidth, false, forceTriangle);
                }
 
                if (newWidth === 0) {
                    $mainView.off("mousemove.sidebar");
                    $("body").toggleClass("resizing");
                }
 
                e.preventDefault();
            });
 
            $mainView.one("mouseup.sidebar", function (e) {
                $mainView.off("mousemove.sidebar");
                $body.toggleClass("resizing");
                startingSidebarPosition = $sidebar.width();
            });
 
            e.preventDefault();
        });
    }

The main issue is that there is a lot of jQuery selecting (and then setting) that’s going on. Look at the mousemove event handler and see how much is going on there. That’s kind of bad. It not only does some math, it also calls _setWidth(), which goes through and modifies significant parts of the DOM. It turns out that it’s pretty expensive to pull from and modify the DOM, and when it’s happening a lot every time the mouse moves, you’re going to get a significant bottleneck. If only there was a way to only do all of that getting and setting when the browser could handle it. That’s the beauty of requestAnimationFrame

Optimizing with requestAnimationFrame

requestAnimationFrame is a pretty straightforward API to use, but it took me a little bit to figure it out. Basically you call window.webkitRequestAnimationFrame() (that’s the webkit-specific prefix) and pass in a function that will be called every time the browser gets to the point where it can redraw the page. What’s a little tricky is that you have to tell the browser to keep listening for it because normally it just gets called once. So what I found easiest was to just call window.webkitRequestAnimationFrame() at the end of the function you’re passing into requestAnimationFrame. Here’s my rewritten example of the _initSidebarResizer():

    function _initSidebarResizer() {
        var $mainView               = $(".main-view"),
            $body                   = $(document.body),
            prefs                   = PreferencesManager.getPreferenceStorage(PREFERENCES_CLIENT_ID, defaultPrefs),
            sidebarWidth            = prefs.getValue("sidebarWidth"),
            startingSidebarPosition = sidebarWidth,
            animationRequest        = null,
            isMouseDown             = false;
 
        $sidebarResizer.css("left", sidebarWidth - 1);
 
        if (prefs.getValue("sidebarClosed")) {
            toggleSidebar(sidebarWidth);
        } else {
            _setWidth(sidebarWidth, true, true);
        }
 
        $sidebarResizer.on("dblclick", function () {
            if ($sidebar.width() === 1) {
                // mousedown is fired first. Sidebar is already toggeled open to 1px.
                _setWidth(null, true, true);
            } else {
                toggleSidebar();
            }
        });
        $sidebarResizer.on("mousedown.sidebar", function (e) {
            var startX = e.clientX,
                newWidth = Math.max(startX, 0),
                doResize = true;
 
            isMouseDown = true;
 
            // take away the shadows (for performance reasons during sidebarmovement)
            $sidebar.find(".scroller-shadow").css("display", "none");
 
            $body.toggleClass("resizing");
            // check to see if we're currently in hidden mode
            if (isSidebarClosed) {
                toggleSidebar(1);
            }
 
            animationRequest = window.webkitRequestAnimationFrame(function doRedraw() {
                // only run this if the mouse is down so we don't constantly loop even
                // after we're done resizing.
                if (isMouseDown) {
                    // if we've gone below 10 pixels on a mouse move, and the
                    // sidebar is shrinking, hide the sidebar automatically an
                    // unbind the mouse event.
                    if ((startX > 10) && (newWidth < 10)) {
                        toggleSidebar(startingSidebarPosition);
                        $mainView.off("mousemove.sidebar");
                        $body.toggleClass("resizing");
                        doResize = false;
                    } else if (startX < 10) {
                        // reset startX if we're going from a snapped closed position to open
                        startX = startingSidebarPosition;
                    }
 
                    if (doResize) {
                        // if we've moving past 10 pixels, make the triangle visible again
                        // and register that the sidebar is no longer snapped closed.
                        var forceTriangle = null;
 
                        if (newWidth > 10) {
                            forceTriangle = true;
                        }
                        // for right now, displayTriangle is always going to be false for _setWidth
                        // because we want to hide it when we move, and _setWidth only gets called
                        // on mousemove now.
                        _setWidth(newWidth, false, false);
                    }
 
                    if (newWidth === 0) {
                        $mainView.off("mousemove.sidebar");
                        $("body").toggleClass("resizing");
                    }
                    animationRequest = window.webkitRequestAnimationFrame(doRedraw);
                }
            });
 
            $mainView.on("mousemove.sidebar", function (e) {
                newWidth = Math.max(e.clientX, 0);
 
                e.preventDefault();
            });
 
            $mainView.one("mouseup.sidebar", function (e) {
                isMouseDown = false;
 
                // replace shadows and triangle
                $sidebar.find(".triangle-visible").css("display", "block");
                $sidebar.find(".scroller-shadow").css("display", "block");
 
                EditorManager.resizeEditor();
                $projectFilesContainer.triggerHandler("scroll");
                $openFilesContainer.triggerHandler("scroll");
                $mainView.off("mousemove.sidebar");
                $body.toggleClass("resizing");
                startingSidebarPosition = $sidebar.width();
            });
 
            e.preventDefault();
        });
    }

There are a few changes there but the biggest is that the mousemove event only sets the newWidth property based on the mouse event. Nothing else. That keeps mousemove, which happens a lot, nice and small. Everything else has been moved up to mousedown but wrapped in a function named doResize(), which is passed into window.webkitRequestAnimationFrame(). So instead of doing everything on mousemove I’m telling the browser that I only want to do the heavy lifting when the browser is going to do a repaint anyway. And because the newWidth is being updated by mousemove I’ve got the correct width updated and the browser will repaint the sidebar in the correct position.

One thing that confused me was how to stop requestAnimationFrame. There is a cancelRequestAnimationFrame but no one seems to use it because requestAnimationFrame() will run only one extra time when it’s called unless you tell it otherwise. Unfortunately, that’s exactly what this code does because we recursively call requestAnimationFrame() as soon as the function finishes so that if the page is still being dragged, we can get new values. But if we don’t find a way to stop it, the browser will constantly try to repaint even if there’s not a drag event going on. That’s why I use the isMouseDown boolean. If the mouse isn’t down, it’s a way to tell requestAnimationFrame() that I don’t want it to fire any more so it’s going to run once more, then stop.

The result is pretty significant:

Ahhh, much better

The mousemove event now takes 0ms, and since it happens often, that’s a fantastic improvement right off the bat. And now all of the layout/paint events are happening based on an Animation Frame Event instead of on mousemove. You can also see spots where mousemove fires a couple of times before the Animation Frame Event fires and in the old code that would have caused an unnecessary layout/repaint series because even though the browser was doing all of the leg-work on it, it wouldn’t even be able to render it until the next animation frame came up. This way we’re only having the browser do the work when it can.

This is more of a real-world scenario of why requestAnimationFrame can be helpful. Now that I’m digging into it more I’m looking at doing a couple of posts on the basics and starting from a smaller example with requestAnimationFrame but this example made me a believer in how cool it is so I wanted to share.

Huge thanks to Paul Irish for info on using requestAnimationFrame and chatting about it. Talking it through helped a ton to wrap my head around it.

A Response to Benjamin Sandofsky’s Shell Apps

Dion Almaer retweeted an article by Benjamin Sandofsky that covers “shell apps”, or HTML/Hybrid apps in contrast to native apps. His thesis seems to be, from his post:

If you plan to write an app for iOS or Android, you will save time and create a better product if you stick to Objective-C and Java, respectively.

Which I agree with in principle, but I also found some of his arguments deeply flawed. He spends a lot of time talking about the cross-platform problem of trying to build a web-experience that looks native on iOS and native on Android. Specifically he calls out differences on iOS and Android:

The harder problems are inconsistencies in platform conventions. The iPhone uses Helvetica as its system font, while Android uses Roboto. iOS transitions between views by sliding new views in from the right, while Android uses a zoom effect.
You could recreate each platform in HTML5, but that requires a lot of work. You could design everything around an iOS look, and settle for things looking weird on Android. You could create a platform agnostic design language, making things equally weird for everyone. In a web view, it’s an uphill battle for consistency, whereas going native provides consistency for free.

Put aside the fact that if you’re doing native development you’re writing two different versions anyway (I’m not sure why that would be a knock against an HTML-centric solution), why do you have to design around an iOS or Android specific look?

The Native Component Question

I know it’s harder, and potentially more expensive, and requires more design sense, but why should developers of HTML-centric apps for mobile devices be locked into the specific UI look and feel of that platform? I’m not talking about UI paradigms (eg, soft back buttons on iOS, menu buttons on Android), but the specific look of apps. Instead of skinning your application to look like an iOS app and then try to make it look like an Android app, do a design that looks unique and polished that would translate well across both platforms. Then as you start to get into the specific differences (he brings up a good point about transitions) you can implement those device/platform-specific patterns. To me one of the beautiful things about the HTML/JS/CSS stack is the freedom with which we can develop. Bringing that to mobile applications, and letting some great CSS/HTML designers loose on that platform is a great thing. And it’s one of the reasons I love PhoneGap.

There are definitely some performance implications by going with a shell app/HTML-centric app approach. And as your application gets more complex, those will pop up more and more. So it’s something to be very aware of. Benjamin quotes some of the reasons he sees for people choosing HTML-centric apps: “we need to release faster”, “write once, run anywhere”, “HTML is easier”. I’m not sure any of those are great reasons to choose HTML-centric apps. Instead I see the primary benefits of the HTML approach as not being locked into a specific platform or deployment model. With HTML you can repurpose code for mobile web and you can target new platforms as they come (no device is not going to support HTML). But maybe most importantly, with HTML you can easily create custom app experiences. There are some great designers out there who can do amazing things with HTML and choosing something like PhoneGap lets you use those skills to create a unique web-like experience with all of the device features users expect from native apps.

Is it a silver bullet? Definitely not. But I think dismissing HTML-centric apps as being done because they’re cheaper/faster/easier doesn’t do them justice. For some kinds of apps, the HTML-centric approach is just better. There’s an amazing community of people that know HTML/JS/CSS, the web is the ultimate platform for reach and flexibility, and shell-apps bring those two elements to the native app world. That’s something to celebrate.

Installing the ChildBrowser Plugin for iOS with PhoneGap/Cordova 1.5

Note: I’ve added a ChildBrowser example to the PhoneGap Starter project. It won’t help with native issues, but if you’re using PhoneGap Build it might give you a working head start.

I’ve seen a bunch of people who are new to PhoneGap/Cordova struggle with getting the ChildBrowser up and running. I fought with it today as well and most of my mistakes were based on old blog posts so I wanted to do a quick post that hopefully shows up in Google for people running into the same problems I had.

I’m starting with a blank project. The ChildBrowser plugin I’m using is here which as of this writing has been updated to support Cordova 1.5 with some potential backwards compatibility with 1.4.1. The simplest thing is to probably just download the source as a zip.

I’m starting with a blank project that has my www folder already referenced by the project. Here are the rest of the steps.

1. Drag all of the .m, .h, and .bundle file(s) from the ChildBrowser folder to the Plugins folder of your project. Make sure the copy option is selected (I think it’s the default).

2. In your project, under the Supporting Files folder, open the Cordova.plist file. You’ll see a bunch of key/value pairs and the one you want is under Plugins. You’ll need to add 2 by clicking the add button that shows when you highlight the Plugins section.

Key: ChildBrowser, Type:String, Value: ChildBrowser.js
Key: ChildBrowserCommand, Type:String, Value:ChildBrowserCommand

3. You also need to make sure that your application can call the external URL you want to load. This is also handled in the Cordova.plist file under ExternalHosts. Add any of the URLs you plan to open to that. You just need the Value, so you can leave the Key as the default. And remember that you can use wildcards, so the entry below lets you go to any subdomain of Google.

Key: Item 0, Type: String, Value: *.google.com

4. Copy over ChildBrowser.js (or the minified version) from the project files to your www folder.

Now you need to go into your HTML and JS files and make some changes. The biggest one is to reference the ChildBrowser.js file.

<script type="text/javascript"charset="utf-8"src="ChildBrowser.js"></script>

And here’s the modified JavaScript I used to get it to work. I created a childbrowser variable, then in onDeviceReady added a call to the install method of the ChildBrowser object. Finally I created a function that handles the URL opening.

    var childbrowser;
 
function onBodyLoad()
{
document.addEventListener("deviceready", onDeviceReady, false);
}
 
function onDeviceReady()
{
// do your thing!
childbrowser = ChildBrowser.install();
navigator.notification.alert("Cordova is working")
}
 
    function onLinkClick() {   
 
    if(childbrowser != null)
    {
        childbrowser.onLocationChange = function(loc){ alert("In index.html new loc = " + loc); };
        childbrowser.onClose = function(){alert("In index.html child browser closed");};
        childbrowser.onOpenExternal = function(){alert("In index.html onOpenExternal");};
 
        window.plugins.childBrowser.showWebPage("http://google.com");
 
    }  
}

This definitely works with Cordova 1.5 and I’m pretty sure it should work with 1.4. Notice that there’s no need to change any of the .m/.h files, so don’t worry about importing anything or changing code that you may have seen in other blog posts.

And I’m pretty sure this is basically how all Cordova/PhoneGap plugins work. Simply copy the OS-specific files into the plugin directory and then copy the JS files and reference them. But ChildBrowser threw me for a loop because of all the blog noise that referenced past versions. So hopefully this helps someone.

Hoping for a Web App Future

If I think back to 2006/2007 I was very happy with how things were shaking out. The web was on the upswing and we were moving away from native applications. All of the great things about the web–its ubiquity, its freedom, its openness–were being harnessed to create native-like experiences that, I hoped at the time, would see us all do away with native apps. At the time, there wasn’t much (I thought) that web apps couldn’t do that we needed native apps for. How wrong I was. With the introduction of the iPhone, and subsequent smartphone releases, we’ve seen a huge shift back to native applications. Part of that is performance, right now native apps just feel better than mobile web apps, but it also came about because of just how many things native mobile apps could do. Geolocation, accelerometers, contact info–the smartphone showed how many things “apps” needed access to and for the most part those features have been exclusive to native applications. So just as the web was starting to really take off, we’ve slid back into native application territory.

It bummed me out, and still does. I thought AIR was an okay solution to the problem, but by the time AIR came around it was pretty clear that “the web” had come to mean HTML/JS, and I’m fine with that. So as PhoneGap started getting traction, and then Adobe took a major interest in the project, I was excited about the prospect of working on it as an Adobe evangelist, and more importantly, working with the teams behind it to see what else they had up their sleeves as the web moves forward.

But another side benefit is that it’s put me on what I’d consider the “right” side of the web argument. Two things got me thinking more about this. One was a very good post by a VC named Mark Suster, who while not telling his companies to focus exclusively on the web, has told them to make it a big part of what they do. So many startups nowadays are thinking completely mobile-first while ignoring the web, I’d say at their peril. When I think of my own usage, I’m still using a lot of native apps (for reasons I can’t quite figure out) but the ones I enjoy most have a web component that is well done and part of the overall experience. Yelp comes to mind. Another is Untappd. I can do almost anything I need to on the website version of Untappd so it’s not as though I’m getting a watered down experience. It more easily lets me move between contexts and devices while still using the service. I contrast that to something like Foursquare or Path. Both are mobile-centric, and with Foursquare I can’t check in because it wants to be sure that I am where I say I am (using GPS) but it still makes the web side of it less useful for me. Path is unusable when you’re not on a mobile device as far as I can tell. When I log in all I get is “download the app”. Which I hate. Mobile is clearly important but the web can’t be ignored.

Path Login

If this is what I see when I log into your site, you're too mobile-first.

The second thing that got me thinking more about this was something Brian Leroux pointed to on the Cordova mailing list, a post by Tim Berners-Lee about Installable Web Apps. This is a model I would love to see take hold. As Tim notes, there are a few things that users need to have when they’re installing web apps, and some trust/permissions issues that need to be figured out. Right now, I think PhoneGap is closest to this model, but a huge, huge, part of me wishes PhoneGap didn’t need to exist. If we could somehow skip the native shim and just take for granted that every platform supported, and at its core used, installable web apps. Maybe something like the WebOS model. But we’re not there yet. So for now, I’m glad I get to work with PhoneGap and build apps with web technologies. Eventually though I think PhoneGap can be used as inspiration for installable web apps. This is kind of how the standards world moves, as more and more people adopt something, people find ways to bring that something back into the standards. I think some of PhoneGap’s APIs and methodologies would make a great start at the idea of installable web apps. And I think the guys behind PhoneGap will be at the forefront of making those things happen, which means Adobe is going to be a really cool place to be over the next few years. It feels like there is a lot of potential to change the world and while I miss spending time with Flash, I feel like the HTML/JS/CSS work I’ve been doing and that Adobe is investing in, will make a similar impact on the web down the road.

Dealing with Binary Data from a Canvas Object using JavaScript TypedArrays

I’m not sure how helpful this will be for anyone, but during the process of doing a binary WebSocket demo I found myself learning a lot about JavaScript typed arrays and how that translates into binary data. The demo I wrote took Canvas image data and sent it over a binary WebSocket connection. The WebSocket server took that data and sent it out to all of the connected clients, who would then render the Canvas data as a PNG. It’s kind of a niche use case but I wanted to specifically create a binary WebSocket demo. It also was a more efficient way to send image data than doing something like base64 encoding it. First off, the Mozilla documentation was REALLY helpful. Major props to them.

Translating Canvas Data Into a Binary Format

Canvas has a getImageData() method that gives you an ImageData object. Within that ImageData object is a data property, which has the actual array data of the image. Normally I would have been able to stop right there because that would (in theory) have the information I needed. But what I had to get at was the ArrayBuffer. The way the spec has been implemented you can’t do anything with the ArrayBuffer. Instead you have to use an ArrayBufferView, which takes the form of TypedArrays in JavaScript. Luckily, to get the actual buffer you can just call the buffer property on any of the typed arrays and do what you want with it. But Canvas (at least in Chrome) is slightly different. The object you get from the ArrayBuffer of ImageData.data is something called CanvasPixelArray. Currently CanvasPixelArray doesn’t behave like a regular typed array, it looks like it will become a Uint8ClampedArray but the way it works in Chrome right now the CanvasPixelArray doesn’t provide access to a buffer property so you can’t send/access the ArrayBuffer data. Luckily getting that data into a Uint8Array, which you can get the buffer data from is pretty easy:

     imagedata = context.getImageData(0, 0, imagewidth,imageheight);
 
     var canvaspixelarray = imagedata.data;
 
 
     var canvaspixellen = canvaspixelarray.length;
     var bytearray = new Uint8Array(canvaspixellen);
 
     for (var i=0;i<canvaspixellen;++i) {
          bytearray[i] = canvaspixelarray[i];
     }

With that new Uint8Array all that we have to do is grab the buffer property and we can send it across the wire (beyond this post, but I’m planning on writing up the binary WebSocket info).

Reassembling the pieces

Now what I wanted to do was take that binary data from my Canvas and render it as a PNG file on the screen. The first step is to use a Canvas object to render out a PNG. But before that we have to get the data into a Canvas. In theory, you should be able to just do what happened above in reverse. But it’s not quite that simple. Once you get back the binary data from somewhere (WebSocket say), you’ve got an ArrayBuffer that you have to deal with. There is a putImageData() that takes an ImageData object, and we can create an ImageData object a few different ways, but you can’t set the data property of it. It’s read only. So we can’t take our data from the ArrayBuffer and just put it into our Canvas. We have to manually loop through the data property and line-by-line change the data.

          var bytearray = new Uint8Array(event.data);
 
 
          var tempcanvas = document.createElement('canvas');
               tempcanvas.height = imageheight;
               tempcanvas.width = imagewidth;
          var tempcontext = tempcanvas.getContext('2d');
 
          var imgdata = tempcontext.getImageData(0,0,imagewidth,imageheight);
 
          var imgdatalen = imgdata.data.length;
 
          for(var i=8;i<imgdatalen;i++)
          {
               imgdata.data[i] = bytearray[i];
          }
 
          tempcontext.putImageData(imgdata,0,0);

If the above isn’t clear, basically I’m just creating a new Uint8Array with the data from the server, then creating the temporary Canvas so I can get image data from it, and when I have that, I’m looping through the data property and inserting my own data from the Uint8Array.

Rendering it as a PNG

So now we have a Canvas (not being displayed) that has all of the data from our server, so it’s an exact graphical copy of the info we received. Turning that into a PNG is actually pretty easy because HTML is awesome. Canvas has a toDataURL() method that will take whatever is in the Canvas and create a string that can be put into the src property of an image. Then putting that image somewhere on the DOM will display the data as an image.

          var img = document.createElement('img');
               img.height = imageheight;
               img.width = imagewidth;
               img.src = tempcanvas.toDataURL();

Fin

Now that I’ve gotten my head around binary data a lot more, I’m kind of excited about JavaScript typed arrays. Looking through the list it looks like the typed arrays will help a lot with byte manipulation because of the different types. It also looks like they’re pretty fast (at least the fastest option at the time of that post).

HTML/JS/CSS and Tooling

I enjoyed this post by Grant Skinner that walks through his view of the evolution of technology and where/how/when tooling starts to come in. Adobe makes tools for web professionals. That’s what we’ve always done and that’s what we’ll do for a long time. You could even paint a broader brush that we make tools for creative people to share their creations. Watching our own evolution over the past year or so with regards to HTML tooling has been very interesting. We got some flack for not moving in earlier, but as Grant rightly points out, tools are a major investment and only once you have stability can you make that investment. It was never a matter of momentum around HTML or a focus on Flash, it was just the fact that things weren’t quite ready for tools.

In fact, I’d argue they still aren’t. But we’ve taken that as something that comes with the web. It’s always evolving, always moving, and while things will start to coalesce more and more, in the end, you have to get in and be ready to move. That’s kind of the approach we’ve taken with Adobe Edge. We just released Preview 4 of Edge which incorporates a lot of features that people have been asking for. Some of it I’m not even sure if it was on the original roadmap. But the Edge team made a conscious decision to be very agile, to build Edge in such a way that it could incorporate customer feedback quickly, and then getting product management on board to do lots of versions very quickly. I think it’s worked out very well and despite being on the earlier side of Grant’s curves, I think Edge will be a very helpful tool for a lot of people because of it.

Developer tools are a bit of a different story in terms of both ecosystem and readiness in my mind. There isn’t one, big, HTML IDE that people seem to like (akin to Flash Builder, Eclipse or Visual Studio). Instead people seem to be using a lot of different things and experimenting. What actually seems to be most popular right now are the basic text editors like TextMate or Sublime (my favorite). These seem to be focused on helping smart people work faster. Lots of shortcuts, lots of snippets, but not a lot in terms of helping along the learning process. And I think that’s just where we are now as far as HTML/JS/CSS tooling. But I’m excited to see that evolution as well and see what happens when frameworks get a bit more standardized and more general web developers start jumping into JS more and more. Will those people need a more robust HTML/CSS/JS editor that’s still developer centric? And I think the answer is yes, but I think it’s also tough to really see what that would look like until the JS/HTML/CSS stack is a bit more solid. But I’m excited to watch it and find out. And from what I’ve seen of Adobe’s HTML tooling side, we’re taking a good approach and I’m excited to see what people think as the PMs share more and more of it.

Slides for my PhoneGap Presentation

I had the chance to present to the Seattle Web App Developers Group (formerly the Flex UG) and here are the slides from that presentation. Thanks a ton to everyone who came out. It was a lot of fun and it was exciting to see so many people interested in PhoneGap.

Building a Mobile-Themed Slider without jQuery Mobile

I’m generally a big fan of jQuery mobile. I think it’s an ambitious project and it’s got some areas that could be improved, but I’m glad we’re contributing and think it’s going to be a huge boon for mobile web applications as they get bigger and bigger. But building PhoneGap applications with jQuery Mobile is a bit problematic partly because the goal for the project is “Delivering top-of-the-line JavaScript and a unified User Interface across the most-used smartphone web browsers…” and partly because jQuery Mobile can be tough to really customize if you want a specific look and feel. One of the things I wanted for something I’m working on is a slider that is a bit more boxy than the traditional slider used with jQuery Mobile. Some of this might have been possible with customizing the jQuery Mobile theme but I wanted to see what it would take to build my own from scratch.

This example is part of an app that’s using Backbone.js so I won’t post a working example because the Backbone.js infrastructure makes it kind of tough to strip out just the toggle button part. I’ll post the whole thing later, but wanted to show the general process.

The Look and Feel

To set it up I wrapped a div tag around a select tag with a couple of options. The div tag provided the border so it ended up being the track that the sider went along. This makes it pretty easy to customize the look and feel while still being very nicely degraded into a regular HTML select box.

                         <div id="toggle" class="toggleoff">
                         <select name="activetoggle" id="activetoggle" class="selectoff">
                              <option value="all">All</option>
                              <option value="active">Active</option>
                         </select>
                         </div>

Initially both the div and the select start with “off” classes. At this point the CSS starts to take over and create a more slider-looking button. There is a cool -webkit-appearance property that can be used on the slider to give it some instantaneous look/feel. But for the most part I rely on -webkit-border-image to customize the look and feel of the slider button. The CSS for the toggle button itself, the div tag, and the off states give us a basic looking toggle with no real toggle functionality because it behaves just like a regular HTML select box.

Toggle button off state

select {
     -webkit-appearance: push-button;
     margin-top: 0px;
     margin-bottom: 0px;
     height: 50px;
     width: 155px;
     border-style: solid;
     border-color: #cdcdcd;
     border-width: 1px;
     color: #555;
       font-size: 25px;
       overflow: hidden;
     padding-top: 2px;
       padding-bottom: 1px;
     white-space: nowrap;
     text-align: center;
 
}
 
.selectoff {
     -webkit-border-image: url( ../assets/button.up.png ) 4 4 4 4 stretch stretch;
     padding-left: 60px;
 
}
 
#toggle {
     margin-left: 5px;
     margin-right: 5px;
     margin-top: 5px;
     height: 50px;
     width: 312px;
     -webkit-border-radius: 2px;
     -webkit-box-shadow: 0px 1px 3px rgba(0, 0, 0, 0.1);    
     border: 2px solid #cdcdcd;    
}
 
.toggleoff {
     padding-left: 158px;
     background-image: -webkit-linear-gradient(#ffffff, #e7e7e7);
}

I have the benefit of using absolute positioning because I’m building this for an iPhone, but the CSS is pretty straightforward. I’ve got some padding-left properties so I can control how the text looks and where the actual toggle button appears within the div tag. This is why this particular strategy works. When I move between states I’m just going to change the position of the toggle button, I’m not actually doing anything with the select object itself other than changing how it looks and where it is. Here are the “on” classes.

 
.selecton {
     -webkit-border-image: url(../assets/button.down.png) 4 4 4 4 stretch stretch;
     padding-left: 30px;
}
 
.toggleon {
     padding-left: 0px;
     background-image:-webkit-linear-gradient(#bccead, #cbdfba);
}

Just a couple of small changes to gradients and backgrounds so that when our button is on or active, it looks like this:

Toggle button on state

Functionality

Now that the CSS classes are set up I used jQuery to first add/remove the classes I wanted and finally set the value of the select box programatically. As I mentioned, this is using Backbone.js so the code may look a bit odd.

          onMouseDown:function(event){
               event.preventDefault();
               if(this.displayingAll) {
                    $("#activetoggle").removeClass("selectoff");
                    $("#activetoggle").addClass("selecton");
                    $("#toggle").removeClass("toggleoff");
                    $("#toggle").addClass("toggleon");
                    $("#activetoggle").val("active");         
               } else {
                    $("#activetoggle").removeClass("selecton");
                    $("#activetoggle").addClass("selectoff");
                    $("#toggle").removeClass("toggleon");
                    $("#toggle").addClass("toggleoff");    
                    $("#activetoggle").val("all");                             
               }

My Backbone view has a variable, displayingAll, which I set to either true or false depending on what is currently visible. It helps track which classes need to be added/removed. But the first, and probably most important thing, is that I’m capturing the event and keeping it from doing anything. I don’t want the default select behavior at all so I can trap it and make sure it doesn’t keep doing the rest of its work. That means no pop up where the user can select values. Instead I go through and swap out the on/off classes to change the look of my button. Finally I use the val() method to set the value of the select so that what the screen is showing is the actual value of the select input.

So now the slider is working but we haven’t done any animation. So right now the slider just jumps back and forth. It works but it isn’t all that pretty. Luckily CSS3 is pretty awesome and we can leverage the very powerful -webkit-transition to help us out.

Animation

Animation for this example is insanely simple. Using the -webkit-transition CSS property we can listen for changes in the DOM on a specific property and whenever that property changes we can play a specific transition. The way I have set this up is that I’m changing the padding-left property between on/off classes so that’s the one we have to listen for.

Quick sidenote: I struggled with this for a bit. At first I was using the align property of the div to move the toggle button back and forth. That had the benefit of being able to scale better for multiple pixel-widths. The button would move back and forth just fine but there wasn’t a way to animate the align property that I found, which makes sense. So to do animation I had to use a value that could be animated and padding-left ended up being a decent option.

In order to get the animation to trigger we have to add a single CSS property to the #toggle div tag:

     -webkit-transition: padding-left 250ms ease-out;

So whenever the padding-left property changes (on a Webkit browser) a quarter of a second ease-out transition will play and move the toggle button back and forth. Under the hood we’re changing the value of the select so that if we have to rely on that value it’s all in sync while still creating a very specific look-and-feel for our toggle button.

I realize this isn’t entirely helpful without a working demo and I’ll work on getting a jsFiddle or something up to show it but I’m working on some other bits of the project and want to be able to share the whole thing when it’s done. One of the main things I learned: CSS3 transitions aren’t entirely intuitive when coming from the Flash world, but they’re very powerful.

Must Read: JavaScript: The Good Parts

JavaScript: The Good Parts

JavaScript: The Good Parts

I just finished JavaScript: The Good Parts by Douglas Crockford and I highly, highly recommend it for anyone who is coming from a programming world that isn’t JavaScript. It’s an intelligent, and well-written book, but it also does the best job of anything I’ve seen yet at explaining some of the nuances of JavaScript and why they can be beneficial.

I was absolutely one of those people who looked at JavaScript as a runty kid brother when I compared it to ActionScript 3. I started off wanting a lot of what ActionScript 3 had in JavaScript. This book completely changed my opinion because it did a fantastic job of laying out how some of the seemingly less-well designed parts of JavaScript end up being pretty powerful. It also highlights some of the areas (like Scope) where JavaScript uses C-like syntax but doesn’t implement it in the same way and may trip up developers. Finally there are a couple of chapters on the bad and awful parts of JavaScript to be aware of.

In a lot of ways it’s the perfect book for someone coming from AS3 to JavaScript. I think it will give you some new found respect for JavaScript and help with some of the parts of JS that seem a little contrary.

Recreating the Path Menu with Adobe Edge

I saw this blog post by Victor Coulon about redoing the Path menu in CSS3 and thought it was really slick. I’m not as up to speed on CSS3 as I need to be yet, but after taking a look at the post, I thought this was a perfect use case for animation in an application. It’s unintrusive, adds some overall polish, and generally improves the user interface. I thought Victor’s demo was awesome and since this use case is a good one for animation, I thought it would be an ideal experiment for Adobe Edge. (Final result is here)

The Animation

Building the animation itself was quite simple. I took Victor’s graphics, made them transparent PNGs, and imported them into Edge. At that point it was just a matter of lining everything up and using the timeline to tweak the animation. I didn’t sit down and do the math on the circle, so the elements aren’t aligned perfectly (apologies) but it’s basically just six elements that all pop in from behind the plus button. With Edge it was pretty easy to add the bounce effect. I just set the easing property to easeOutBack and that gave me the bounce.

The project loaded in Adobe Edge

Interaction

While the animation was easy, it took me a bit to get the interaction down correctly. Basically whenever the red button was clicked I wanted to play the timeline animation. When it was clicked again I wanted to reverse it. Luckily Edge has a decent API that makes working with the code it generates pretty straightforward. The biggest issue I ran into was that the playReverse() API can only be run on a symbol and I was having trouble figuring out how to get a symbol instance from within my click event. Running play() on the Composition (kind of the main Edge element, and the one Edge uses by default) runs off an entire set of other operations that made it tougher to do what I wanted.

So instead I used comp.getStage() to get an instance of the “stage” symbol where I could play and reverse the timeline at will. The result is pretty close to what Victor created and what the Path app uses. To do it all, I replaced the Adobe Edge DOM Ready Event Handler code in my project_name_edge.js file with the following:

/**
 * Adobe Edge DOM Ready Event Handler
 */
$(window).ready(function() {
     comp = new Edge.Composition(compId, {stage: "." + compId}, {});
        /**
 * Adobe Edge Timeline Launch
 */
     comp.ready(function() {
         var symb = comp.getStage();
 
         $('#stage_add_button').click(function(e){
               if(played)
               {
                    symb.playReverse();
                    played = false;
               } else {
                    symb.play();
                    played = true;
               }
 
         });
     });
});

Downsides

One downside is that once I make a change in Edge it overwrites all of the changes I’ve made. I’m not sure if there’s a safe place to make JavaScript edits like the one above but I’m going to check with the team. The other downside is that it’s all JavaScript, which makes it kind of heavy and you won’t get the hardware transforms that some browsers use for CSS3. I’m not much of a designer but after this little proof of concept I’m excited to see how Edge could potentially be used to add some fine-tuned interactions like the Path menu.

You can see a working demo here and the source for the project is up on GitHub.