Bookboard – A Model for the Future of the Web

One of the missions of PhoneGap has always been to bring about its own demise. The PhoneGap team wants the web to have all the functionality that we currently enjoy on devices so that there isn’t a need for PhoneGap to exist. It’s one of the reasons I love the project; I think that’s the perfect goal. I hope we get away from app stores and back to a world where the web rules all. There are obviously many things that have to happen before that becomes a reality but every once in a while I see a glimpse of what the web could be if that vision comes to pass. The latest iteration of that is Bookboard. (Sign up here)

Book Board Selection Screen

Bookboard is a web app that was written for the iPad. It makes heavy use of iOS specific features that allow websites to add themselves to the home screen for a full-screen experience and provide specific icons so that to the end user, it kind of feels like an app that just didn’t come from the app store. Even though it’s iOS-specific right now, the UI design is such that it would work across platforms. So many apps, even PhoneGap apps, follow the specific list->detail form that brings a lot of baggage with it. You have to work at making sure your list/detail view works the way it’s supposed to on iOS or Android (or other platforms). But Bookboard has a very design-heavy UI that lends itself to any platform. The books intuitively ask to be swiped from side to side as you are looking through them (with a nice parallax effect in the background) and when you go into the book itself you use the same swipe motions you would expect. It’s list-detail but brought to life in an interactive way.

Reading a Bookboard Book

Bookboard also makes use of hardware acceleration of CSS for many of its transitions. Everything on my iPad 2 is incredibly fluid and it’s difficult to tell that it’s *not* native. One of the things many app developers have to deal with is making their content feel native without it actually being native. The Bookboard approach seems ideal to me. They’re using a UI that hasn’t been replicated by every other app so they get a bit of leeway in terms of performance, but they don’t need it because they’re offloading to the device hardware via CSS. It works out very well.

Bookboard Menu Screen

Beyond the tech side Bookboard is simply a beautiful, well-designed app. It’s meant for children to read (and they can unlock achievements for reading more books) so it has to be intuitive, but it also looks great. Attention to detail like the parallax scrolling add a friendly touch and my daughter loved the app. She knows how to use an iPad and scrollable books, friendly gestures, and words that are magnified on touch all came naturally to her. If you’re a parent with a kid who enjoys reading (or you want them to enjoy it more) Bookboard is well worth a try.

This is how I want the mobile web to look and behave. Bookboard leverages the mobile-app centric parts of iOS to create an app-like experience while still retaining the unique, design-centric approach that has made the web so great. It’s this kind of custom UI and design influence that starts to make developers stop and think about whether they go native or web. A basic list-centric app isn’t that tough to do in native so more often than not, it probably makes sense just to go native. But the kind of custom UI and design that Bookboard uses lets you involve the designers much more deeply in the process. Since most designers (should) feel comfortable with CSS, they can jump in and contribute directly to the end result. If you can get performance like this using web technologies, I think it becomes a tougher sell to try and do native when web gets you more platforms, more reach, more designer input, and more deployment flexibility.

Hopefully in the next couple of years apps will have been replaced by experiences like Bookboard. It would make my home screen a much more interesting and beautiful place while giving developers ultimate flexibility. That’s a big win for the web. I encourage you to go sign up and see what I mean.

Using Ratchet with PhoneGap

When I saw the Ratchet project I was stoked. While the initial site talked about being able to prototype iPhone applications with HTML/JS/CSS it was pretty apparent that this could be something even bigger than that. And sure enough, on their README page it talks about long term goals:

We eventually want to extend Ratchet beyond the prototying for iPhone and create HTML/CSS/JS components for iPad and Android devices. Our dream is that Ratchet becomes the best way to create mobile experiences with web standard languages.

To me, that means this could be a great project for PhoneGap applications. So I threw it into a project and looked at getting it running. Unfortunately nothing happened. I could get some components to work but once I tried to move between pages it wouldn’t respond. I tracked it down to the push.js-esque functionality that they’re using. Specifically, it’s a single line of code that causes issues.

if (xhr.readyState == 4) xhr.status == 200 ? success(xhr, options) : failure(options.url);

The issue is that PhoneGap only seems to return a status of 0 when using XMLHttpRequest. I’m fairly sure this has something to do with the fact that it’s loading from the file system and not a server, but I’m not 100% sure. By making sure that Ratchet accounts for status of 0, page transitions work perfectly.

if (xhr.readyState == 4) xhr.status == 200 || (xhr.status == 0 && options.url.indexOf('file:///') != -1) ? success(xhr, options) : failure(options.url);

I’ve got the full project up on GitHub and you can bring it into PhoneGap Build and test it yourself. I’ve only tested it on iOS but it works pretty damn well. I’m definitely intrigued. I’ve also posted this to the Google Group to see if they can support this out of the box.

Update: Simon mentioned in the comments below that by tweaking the code to check for status 0 AND file system, it would be safer. So I modified the ratchet.js file in my project to take that into account. Thanks Simon.

The Mobile Web is Always the Right Answer

It’s been interesting to continue to watch the fallout of Facebook’s native announcement because so many smart people are discussing the web versus native dynamic. What I find both fascinating and odd (yet unsurprising) is that “HTML5″ continues to have vague and very subjective connotations. In the context of this debate, HTML5 seems to really mean hybrid applications that are built with technologies like Cordova/PhoneGap. And for proponents of native that’s a great way to frame the debate because when compared on equal ground, native will win most of the time. That’s because native apps were meant to be apps whereas PhoneGap/Cordova apps are really web content that’s built to behave in an application world. The suit will never fit quite as well as it does on Mr. Native.

But in some ways, that’s not the point. The real debate, and the most important one is the role of the web in the future of mobile devices. Currently there are number of techniques to bring the web to mobile devices. There’s responsive design, there’s the m-dot, there’s PhoneGap/Cordova, etc. They all have specific plusses and minuses depending on what you’re trying to accomplish. But they’re also all very rooted in the ethos of the web.

When people talk about HTML5 versus native it’s often going to be in the app context. And then the narrative can become “HTML5 isn’t ready”. That’s bullshit. HTML as a technology is more than ready to help you create apps and content for mobile devices. You may not get access to every device API and it may not fit into an app store, but the corollary is that you don’t need an app store. You can create whatever content you want and give it to the world. You have to make sure the experience is good on devices, but you don’t have to pass anyone’s arbitrary test to play the game, you don’t have to lock yourself into a specific platform. You just create your experience and let the world decide if it’s worth visiting. And users still do a ton of interacting with the web on their phone through a browser.

Facebook Mobile Usage Numbers

Facebook’s mobile usage numbers by platform

The image above, from PhoneGap Day EU during a talk by Facebook’s Simon Cross, perfectly illustrates that. There’s definitely a legitimate argument to be made that part of the reason the web usage is so high is because the hybrid Facebook apps were horrible to use. But look at the graph again. Mobile web usage is levels of magnitude above the native apps. That’s not just from a bad experience. That’s because people spend a lot of time in their browser on smartphones.

We’re in a fantastic time for the mobile web because there are so many options for creating mobile web content. But no matter how you want to show your content, it’s never a good idea to ignore the mobile web. Every great app I can think of has some tie to the web. Foursquare, Yelp, Untappd, and Twitter. The native app experience is a critical part- and it may even provide the best experience currently- but the web is even more vital to the long term survival of those companies. Never, ever discount HTML and the web. You’ll regret it. You can ask Path in a year how it feels.

Don’t Settle for a Mediocre HTML5 App Experience

Since Facebook announced their decision to ditch their HTML-centric app experience for a native one, I’ve been doing a lot of thinking about what that means for the HTML5 versus native debate. Unfortunately we’re at a point in the circle of tech where it’s easier to build a great native experience than it is to build a great HTML experience. I’d argue it’s not impossible to build a great HTML experience on mobile devices, just more difficult (I think Untappd is a great example, photo bugs not withstanding). The problem is that runs directly counter to why most people choose to go the HTML route and that causes a host of issues.

Today, when someone decides to build a mobile app from HTML, it’s primarily a developer-focused decision and not a user-focused decision. What I mean by that is that often times HTML makes it easier/faster/cheaper for someone to deploy a mobile app. Maybe they have a bunch of web developers and want to quickly crank out an app. Maybe they want to target multiple devices and they see HTML as a way to reuse code. Those aren’t bad reasons to chose HTML5 necessarily, but they do drastically change the goals of a project from “make this as awesome as possible” to “make this as good as we can with our constraints”.

So it’s no wonder that a lot of the HTML mobile app experiences are subpar. They started with the very premise of wanting to do it cheaper/faster/easier. That’s not a path to software greatness. Facebook again provides an example here. They had a ton of HTML content they wanted to reuse and an HTML-centric app was a fast, easy way to repurpose that content for mobile apps. Plus they could deploy it to lots of different kinds of devices with minimal code changes. They didn’t start from a place of wanting to make a great mobile experience, and the results spoke for themselves. When pressure finally got to a breaking point and they had to create a great mobile experience, they went with the clearest path to that route – native. This tweet by Josh Clark implies as much. It wasn’t that the HTML app was fundamentally unfixable, it was that (I assume) they looked at what it would take to make it as good as the native app and decided it wasn’t worth the effort.

I don’t blame people for making that judgement call. The PR boost that people used to get from throwing HTML5 into their marketing has started to subside and at the end of the day your users aren’t going to forgive a crappy mobile experience because you got 5 in a row on buzzword bingo.

HTML developers are in a rough spot right now because once you take out the faster/easier/cheaper argument, it’s tough to provide reasons why HTML is better than native apps. I see three main reasons. One, the web is a fantastic medium for designers and allows very unique experiences. Two, the ecosystem around web technologies has never been stronger and the number of cool frameworks, tools, and libraries that are available to web developers could have a significant impact on the kind of app you can build. Three, it never pays to bet against the web.

The first two I’ll cover in later posts and the third is the most subjective anyway so I want to talk briefly about that. For most apps, the future isn’t going to matter, so this isn’t a “future proof your code” argument. It speaks deeper than that. Do you want to be the kind of company who makes the future, or do you want to be the kind of company who follows along? HTML is the future. Just like native fell out of favor on the desktop, native is going to fall out of favor on mobile devices. The APIs are getting better for device access. Core graphical capabilities are being added that will make for nicer components, better experiences, and faster UIs. And the web is bigger than one company. The web is the great equalizer in tech. That leads to its own sets of drama about how we add features, but at the end of the day, no one will own or control the web. I see the web as a glacier. It tends to move very slowly, which can be a drawback. But when it moves, it brings everything with it and it carves a deep, permanent path that completely changes the landscape. That’s a tough thing to try and stop or stand in front of.

When you see an HTML mobile app that’s not as fast or snappy as it should be, think about what problem they were trying to solve. If the answer isn’t “build the best app possible” then don’t take that as a good example of what HTML is capable of. And if you’re building an HTML app put the same amount of effort, thought, and design into it as you would a native app. With the breadth of design possibilities, the HTML/JS/CSS ecosystem, and some forward-looking vision, you can build something that helps turn the conversation back to why the web is such an awesome medium for applications.

Join Us on the Create the Web Tour

Today we announced the Create the Web tour, a series of events that we will be doing around the world to talk about all the things Adobe has been doing with the web including new tools, new features in CSS, and services for web developers.

The tour is going to be in two parts, the first part is a full day event in four cities (San Francisco, London, Tokyo, and Sydney) that covers the full gamut of Adobe web tools and technologies. The second part is a much wider tour where the Adobe evangelist team will be spreading out across the globe to talk in more detail about specific tools and technologies at user groups, hack-a-thons, and other events.

We’ll have more information about that bigger tour soon, but in the meantime, if you’re in one of the 4 cities for the full day event, we’d love to see you.

Using requestAnimationFrame to Optimize Dragging Events

requestAnimationFrame is an API that was originally created by Mozilla but has found its way into Chrome and I think it has huge, huge implications for user interface. A lot of the examples I’ve seen have talked about how you can use it to optimize animation, which makes sense. The basic premise of requestAnimationFrame is that instead of trying to move things out of sync with the browser, you get a hook directly into the browser refresh/redraw and on that refresh you can tell the browser to do something specific with requestAnimationFrame. So it’s sort of like a queuing system whereby you tell the browser what you want to happen, and when it goes through its next round of repaints, it executes what’s in that queue. This makes perfect sense for animation since the animation can’t move any faster than the repaint cycles of the browser. But it also turns out it makes great sense to use it for dragging events as well.

The Problem:

For me, the problem manifested when I was working on making the Brackets sidebar resizable. Like with a lot of resize events I was doing the resize on a mousemove event, so I’d track the mousemove and then resize the sidebar accordingly as the mouse moved. In theory it seemed like a decent way to do things until I opened it up in the Chrome Developer tools:

Holy repaint batman!

That screenshot may be a bit confusing, but basically, every mouse move there are a TON of “Recalculate Style/Paint” events happening. That means every single mousemove the browser is trying to repaint and so each mousemove event takes about 50 milliseconds to process. That may not sound like much, but think about how often mousemove fires. What’s worse is that because of how the browser redraws, even though it’s trying to redraw everything on mouse move, it can’t actually do that because the mousemove events are happening faster than the browser can redraw. Here’s the code that’s doing all that repainting:

    /**
     * @private
     * Sets sidebar width and resizes editor. Does not change internal sidebar open/closed state.
     * @param {number} width Optional width in pixels. If null or undefined, the default width is used.
     * @param {!boolean} updateMenu Updates "View" menu label to indicate current sidebar state.
     * @param {!boolean} displayTriangle Display selection marker triangle in the active view.
     */
    function _setWidth(width, updateMenu, displayTriangle) {
        // if we specify a width with the handler call, use that. Otherwise use
        // the greater of the current width or 200 (200 is the minimum width we'd snap back to)
 
        var prefs                   = PreferencesManager.getPreferenceStorage(PREFERENCES_CLIENT_ID, defaultPrefs),
            sidebarWidth            = Math.max(prefs.getValue("sidebarWidth"), 10);
 
        width = width || Math.max($sidebar.width(), sidebarWidth);
 
        if (typeof displayTriangle === "boolean") {
            var display = (displayTriangle) ? "block" : "none";
            $sidebar.find(".triangle-visible").css("display", display);
        }
 
        if (isSidebarClosed) {
            $sidebarResizer.css("left", 0);
        } else {
            $sidebar.width(width);
            $sidebarResizer.css("left", width - 1);
 
            // the following three lines help resize things when the sidebar shows
            $sidebar.find(".sidebar-selection").width(width);
            $projectFilesContainer.triggerHandler("scroll");
            $openFilesContainer.triggerHandler("scroll");
 
            if (width > 10) {
                prefs.setValue("sidebarWidth", width);
            }
        }
 
        if (updateMenu) {
            var text = (isSidebarClosed) ? "Show Sidebar" : "Hide Sidebar";
            $sidebarMenuText.first().text(text);
        }
 
        EditorManager.resizeEditor();
    }
 
    /**
     * @private
     * Install sidebar resize handling.
     */
    function _initSidebarResizer() {
        var $mainView               = $(".main-view"),
            $body                   = $(document.body),
            prefs                   = PreferencesManager.getPreferenceStorage(PREFERENCES_CLIENT_ID, defaultPrefs),
            sidebarWidth            = prefs.getValue("sidebarWidth"),
            startingSidebarPosition = sidebarWidth;
 
        $sidebarResizer.css("left", sidebarWidth - 1);
 
        if (prefs.getValue("sidebarClosed")) {
            toggleSidebar(sidebarWidth);
        } else {
            _setWidth(sidebarWidth, true, true);
        }
 
        $sidebarResizer.on("dblclick", function () {
            if ($sidebar.width() === 1) {
                // mousedown is fired first. Sidebar is already toggeled open to 1px.
                _setWidth(null, true, true);
            } else {
                toggleSidebar();
            }
        });
        $sidebarResizer.on("mousedown.sidebar", function (e) {
            var startX = e.clientX;
            $body.toggleClass("resizing");
            // check to see if we're currently in hidden mode
            if (isSidebarClosed) {
                toggleSidebar(1);
            }
 
            $mainView.on("mousemove.sidebar", function (e) {
                var doResize = true,
                    newWidth = Math.max(e.clientX, 0);
 
                // if we've gone below 10 pixels on a mouse move, and the
                // sidebar is shrinking, hide the sidebar automatically an
                // unbind the mouse event.
                if ((startX > 10) && (newWidth < 10)) {
                    toggleSidebar(startingSidebarPosition);
                    $mainView.off("mousemove.sidebar");
                    $body.toggleClass("resizing");
                    doResize = false;
                } else if (startX < 10) {
                    // reset startX if we're going from a snapped closed position to open
                    startX = startingSidebarPosition;
                }
 
                if (doResize) {
                    // if we've moving past 10 pixels, make the triangle visible again
                    // and register that the sidebar is no longer snapped closed.
                    var forceTriangle = null;
 
                    if (newWidth > 10) {
                        forceTriangle = true;
                    }
 
                    _setWidth(newWidth, false, forceTriangle);
                }
 
                if (newWidth === 0) {
                    $mainView.off("mousemove.sidebar");
                    $("body").toggleClass("resizing");
                }
 
                e.preventDefault();
            });
 
            $mainView.one("mouseup.sidebar", function (e) {
                $mainView.off("mousemove.sidebar");
                $body.toggleClass("resizing");
                startingSidebarPosition = $sidebar.width();
            });
 
            e.preventDefault();
        });
    }

The main issue is that there is a lot of jQuery selecting (and then setting) that’s going on. Look at the mousemove event handler and see how much is going on there. That’s kind of bad. It not only does some math, it also calls _setWidth(), which goes through and modifies significant parts of the DOM. It turns out that it’s pretty expensive to pull from and modify the DOM, and when it’s happening a lot every time the mouse moves, you’re going to get a significant bottleneck. If only there was a way to only do all of that getting and setting when the browser could handle it. That’s the beauty of requestAnimationFrame

Optimizing with requestAnimationFrame

requestAnimationFrame is a pretty straightforward API to use, but it took me a little bit to figure it out. Basically you call window.webkitRequestAnimationFrame() (that’s the webkit-specific prefix) and pass in a function that will be called every time the browser gets to the point where it can redraw the page. What’s a little tricky is that you have to tell the browser to keep listening for it because normally it just gets called once. So what I found easiest was to just call window.webkitRequestAnimationFrame() at the end of the function you’re passing into requestAnimationFrame. Here’s my rewritten example of the _initSidebarResizer():

    function _initSidebarResizer() {
        var $mainView               = $(".main-view"),
            $body                   = $(document.body),
            prefs                   = PreferencesManager.getPreferenceStorage(PREFERENCES_CLIENT_ID, defaultPrefs),
            sidebarWidth            = prefs.getValue("sidebarWidth"),
            startingSidebarPosition = sidebarWidth,
            animationRequest        = null,
            isMouseDown             = false;
 
        $sidebarResizer.css("left", sidebarWidth - 1);
 
        if (prefs.getValue("sidebarClosed")) {
            toggleSidebar(sidebarWidth);
        } else {
            _setWidth(sidebarWidth, true, true);
        }
 
        $sidebarResizer.on("dblclick", function () {
            if ($sidebar.width() === 1) {
                // mousedown is fired first. Sidebar is already toggeled open to 1px.
                _setWidth(null, true, true);
            } else {
                toggleSidebar();
            }
        });
        $sidebarResizer.on("mousedown.sidebar", function (e) {
            var startX = e.clientX,
                newWidth = Math.max(startX, 0),
                doResize = true;
 
            isMouseDown = true;
 
            // take away the shadows (for performance reasons during sidebarmovement)
            $sidebar.find(".scroller-shadow").css("display", "none");
 
            $body.toggleClass("resizing");
            // check to see if we're currently in hidden mode
            if (isSidebarClosed) {
                toggleSidebar(1);
            }
 
            animationRequest = window.webkitRequestAnimationFrame(function doRedraw() {
                // only run this if the mouse is down so we don't constantly loop even
                // after we're done resizing.
                if (isMouseDown) {
                    // if we've gone below 10 pixels on a mouse move, and the
                    // sidebar is shrinking, hide the sidebar automatically an
                    // unbind the mouse event.
                    if ((startX > 10) && (newWidth < 10)) {
                        toggleSidebar(startingSidebarPosition);
                        $mainView.off("mousemove.sidebar");
                        $body.toggleClass("resizing");
                        doResize = false;
                    } else if (startX < 10) {
                        // reset startX if we're going from a snapped closed position to open
                        startX = startingSidebarPosition;
                    }
 
                    if (doResize) {
                        // if we've moving past 10 pixels, make the triangle visible again
                        // and register that the sidebar is no longer snapped closed.
                        var forceTriangle = null;
 
                        if (newWidth > 10) {
                            forceTriangle = true;
                        }
                        // for right now, displayTriangle is always going to be false for _setWidth
                        // because we want to hide it when we move, and _setWidth only gets called
                        // on mousemove now.
                        _setWidth(newWidth, false, false);
                    }
 
                    if (newWidth === 0) {
                        $mainView.off("mousemove.sidebar");
                        $("body").toggleClass("resizing");
                    }
                    animationRequest = window.webkitRequestAnimationFrame(doRedraw);
                }
            });
 
            $mainView.on("mousemove.sidebar", function (e) {
                newWidth = Math.max(e.clientX, 0);
 
                e.preventDefault();
            });
 
            $mainView.one("mouseup.sidebar", function (e) {
                isMouseDown = false;
 
                // replace shadows and triangle
                $sidebar.find(".triangle-visible").css("display", "block");
                $sidebar.find(".scroller-shadow").css("display", "block");
 
                EditorManager.resizeEditor();
                $projectFilesContainer.triggerHandler("scroll");
                $openFilesContainer.triggerHandler("scroll");
                $mainView.off("mousemove.sidebar");
                $body.toggleClass("resizing");
                startingSidebarPosition = $sidebar.width();
            });
 
            e.preventDefault();
        });
    }

There are a few changes there but the biggest is that the mousemove event only sets the newWidth property based on the mouse event. Nothing else. That keeps mousemove, which happens a lot, nice and small. Everything else has been moved up to mousedown but wrapped in a function named doResize(), which is passed into window.webkitRequestAnimationFrame(). So instead of doing everything on mousemove I’m telling the browser that I only want to do the heavy lifting when the browser is going to do a repaint anyway. And because the newWidth is being updated by mousemove I’ve got the correct width updated and the browser will repaint the sidebar in the correct position.

One thing that confused me was how to stop requestAnimationFrame. There is a cancelRequestAnimationFrame but no one seems to use it because requestAnimationFrame() will run only one extra time when it’s called unless you tell it otherwise. Unfortunately, that’s exactly what this code does because we recursively call requestAnimationFrame() as soon as the function finishes so that if the page is still being dragged, we can get new values. But if we don’t find a way to stop it, the browser will constantly try to repaint even if there’s not a drag event going on. That’s why I use the isMouseDown boolean. If the mouse isn’t down, it’s a way to tell requestAnimationFrame() that I don’t want it to fire any more so it’s going to run once more, then stop.

The result is pretty significant:

Ahhh, much better

The mousemove event now takes 0ms, and since it happens often, that’s a fantastic improvement right off the bat. And now all of the layout/paint events are happening based on an Animation Frame Event instead of on mousemove. You can also see spots where mousemove fires a couple of times before the Animation Frame Event fires and in the old code that would have caused an unnecessary layout/repaint series because even though the browser was doing all of the leg-work on it, it wouldn’t even be able to render it until the next animation frame came up. This way we’re only having the browser do the work when it can.

This is more of a real-world scenario of why requestAnimationFrame can be helpful. Now that I’m digging into it more I’m looking at doing a couple of posts on the basics and starting from a smaller example with requestAnimationFrame but this example made me a believer in how cool it is so I wanted to share.

Huge thanks to Paul Irish for info on using requestAnimationFrame and chatting about it. Talking it through helped a ton to wrap my head around it.

A Response to Benjamin Sandofsky’s Shell Apps

Dion Almaer retweeted an article by Benjamin Sandofsky that covers “shell apps”, or HTML/Hybrid apps in contrast to native apps. His thesis seems to be, from his post:

If you plan to write an app for iOS or Android, you will save time and create a better product if you stick to Objective-C and Java, respectively.

Which I agree with in principle, but I also found some of his arguments deeply flawed. He spends a lot of time talking about the cross-platform problem of trying to build a web-experience that looks native on iOS and native on Android. Specifically he calls out differences on iOS and Android:

The harder problems are inconsistencies in platform conventions. The iPhone uses Helvetica as its system font, while Android uses Roboto. iOS transitions between views by sliding new views in from the right, while Android uses a zoom effect.
You could recreate each platform in HTML5, but that requires a lot of work. You could design everything around an iOS look, and settle for things looking weird on Android. You could create a platform agnostic design language, making things equally weird for everyone. In a web view, it’s an uphill battle for consistency, whereas going native provides consistency for free.

Put aside the fact that if you’re doing native development you’re writing two different versions anyway (I’m not sure why that would be a knock against an HTML-centric solution), why do you have to design around an iOS or Android specific look?

The Native Component Question

I know it’s harder, and potentially more expensive, and requires more design sense, but why should developers of HTML-centric apps for mobile devices be locked into the specific UI look and feel of that platform? I’m not talking about UI paradigms (eg, soft back buttons on iOS, menu buttons on Android), but the specific look of apps. Instead of skinning your application to look like an iOS app and then try to make it look like an Android app, do a design that looks unique and polished that would translate well across both platforms. Then as you start to get into the specific differences (he brings up a good point about transitions) you can implement those device/platform-specific patterns. To me one of the beautiful things about the HTML/JS/CSS stack is the freedom with which we can develop. Bringing that to mobile applications, and letting some great CSS/HTML designers loose on that platform is a great thing. And it’s one of the reasons I love PhoneGap.

There are definitely some performance implications by going with a shell app/HTML-centric app approach. And as your application gets more complex, those will pop up more and more. So it’s something to be very aware of. Benjamin quotes some of the reasons he sees for people choosing HTML-centric apps: “we need to release faster”, “write once, run anywhere”, “HTML is easier”. I’m not sure any of those are great reasons to choose HTML-centric apps. Instead I see the primary benefits of the HTML approach as not being locked into a specific platform or deployment model. With HTML you can repurpose code for mobile web and you can target new platforms as they come (no device is not going to support HTML). But maybe most importantly, with HTML you can easily create custom app experiences. There are some great designers out there who can do amazing things with HTML and choosing something like PhoneGap lets you use those skills to create a unique web-like experience with all of the device features users expect from native apps.

Is it a silver bullet? Definitely not. But I think dismissing HTML-centric apps as being done because they’re cheaper/faster/easier doesn’t do them justice. For some kinds of apps, the HTML-centric approach is just better. There’s an amazing community of people that know HTML/JS/CSS, the web is the ultimate platform for reach and flexibility, and shell-apps bring those two elements to the native app world. That’s something to celebrate.

Web Versus Native – Asking the Wrong Questions

There’s been a lot of talk lately about web apps versus native. It’s not that the question has ever gone away, but just over the past couple of weeks it seems like it’s been called out a bit more in blog posts. There was a Pew Internet report that covered some of the pros/cons of going web or native. Darryl Taft, one of my favorite press guys, did a writeup earlier this month on the landscape of web versus native and some of the companies involved in blending those two worlds. And then today I saw news about Mojito, a hybrid client-server solution from Yahoo, which is part of a wider initiative called Cocktails with the aim of “making web applications feel native“. And that got me thinking about the problems that people are trying to solve when it comes to web apps on mobile devices.

Web apps are in a tough spot when it comes to native, but they’re getting better. As I see it, one of the main problems with web apps is that they aren’t currently better. They’re cheaper, they’re faster (to write), and they allow you to target more platforms, but they’re not better. What’s even worse is that you could argue there are only 2 platforms for mobile that matter right now, Android and iOS. But it was the exact same situation back in 2003/2004 when the web really started coming into its own as an app platform and taking away from traditional native apps on the desktop. The one problem is that back then the web apps were better. They didn’t look as fancy and they didn’t run quite as fast, but there were few things that people needed to do that the web couldn’t do. I remember talking about AIR and mentioning things like drag and drop, file system access, and file type associations. Those were nice, but people didn’t really need them. What made web apps better for end users was the fact that they ran everywhere, were lightweight, always up-to-date and moved with the user so when they switched between the library, school, work, and home, their stuff went with them.

But fast forward to today. We never leave our mobile devices. They’re always with us. Sometimes we change contexts from our mobile device to a desktop/laptop/tablet (think email) but ultimately one of the things that made web apps so great, that they moved with the user, has been completely negated by smartphones. And increasingly, what web apps can’t do is much more important than what web apps couldn’t do 5 years ago. All of the sensors; GPS, Accelerometer, Camera, not to mention the data on the phone, things like contacts, that can be used to make things on the device so much more personal and completely change the experience. Some of the basic ones are available to web apps, and the specs are in place to get more, but there’s still a significant disconnect between what can be done with native apps on the device and what can be done on the mobile web.

To me that’s the biggest worry for the web on mobile devices right now. Sure, performance isn’t ideal, but as the browsers get better and devices get more powerful, that one will fix itself. The technology is there in the form of Canvas, SVG, great JS frameworks, faster JS performance. So the core interactive features work on these devices (with some performance caveats). But the benefits of the web are in many ways nullified by smartphones today. The web needs to come up with something that makes it better once again. And the deck is really, really stacked against it. Think of the distribution model of apps and in-app purchases. I think you could argue that those aren’t great for customers as much as they are great for content creators. Mobile apps are easier to monetize, easier to track, and easier to manage than the web right now. And on one hand, that’s fantastic. Finally designers and developers can directly make money off of their creations. But it’s coming at the expense of the web.

In order for web apps to thrive again the web community needs to start thinking about how mobile web apps can be better and more useful to customers than native apps. I have no doubt that it’s going to happen. How many times have you said to yourself “I don’t want your app, I just want to see the content” while browsing around the web on a mobile browser? So the sentiment from users is there, but for now the web just doesn’t provide enough value for users to get beyond the inflection point. So I’m hoping the conversation starts to move away from web apps not being fast enough, or “feeling native enough” to one of how web apps can be better than native from a user perspective. It’s going to be a fun conversation to watch unfold.

Hoping for a Web App Future

If I think back to 2006/2007 I was very happy with how things were shaking out. The web was on the upswing and we were moving away from native applications. All of the great things about the web–its ubiquity, its freedom, its openness–were being harnessed to create native-like experiences that, I hoped at the time, would see us all do away with native apps. At the time, there wasn’t much (I thought) that web apps couldn’t do that we needed native apps for. How wrong I was. With the introduction of the iPhone, and subsequent smartphone releases, we’ve seen a huge shift back to native applications. Part of that is performance, right now native apps just feel better than mobile web apps, but it also came about because of just how many things native mobile apps could do. Geolocation, accelerometers, contact info–the smartphone showed how many things “apps” needed access to and for the most part those features have been exclusive to native applications. So just as the web was starting to really take off, we’ve slid back into native application territory.

It bummed me out, and still does. I thought AIR was an okay solution to the problem, but by the time AIR came around it was pretty clear that “the web” had come to mean HTML/JS, and I’m fine with that. So as PhoneGap started getting traction, and then Adobe took a major interest in the project, I was excited about the prospect of working on it as an Adobe evangelist, and more importantly, working with the teams behind it to see what else they had up their sleeves as the web moves forward.

But another side benefit is that it’s put me on what I’d consider the “right” side of the web argument. Two things got me thinking more about this. One was a very good post by a VC named Mark Suster, who while not telling his companies to focus exclusively on the web, has told them to make it a big part of what they do. So many startups nowadays are thinking completely mobile-first while ignoring the web, I’d say at their peril. When I think of my own usage, I’m still using a lot of native apps (for reasons I can’t quite figure out) but the ones I enjoy most have a web component that is well done and part of the overall experience. Yelp comes to mind. Another is Untappd. I can do almost anything I need to on the website version of Untappd so it’s not as though I’m getting a watered down experience. It more easily lets me move between contexts and devices while still using the service. I contrast that to something like Foursquare or Path. Both are mobile-centric, and with Foursquare I can’t check in because it wants to be sure that I am where I say I am (using GPS) but it still makes the web side of it less useful for me. Path is unusable when you’re not on a mobile device as far as I can tell. When I log in all I get is “download the app”. Which I hate. Mobile is clearly important but the web can’t be ignored.

Path Login

If this is what I see when I log into your site, you're too mobile-first.

The second thing that got me thinking more about this was something Brian Leroux pointed to on the Cordova mailing list, a post by Tim Berners-Lee about Installable Web Apps. This is a model I would love to see take hold. As Tim notes, there are a few things that users need to have when they’re installing web apps, and some trust/permissions issues that need to be figured out. Right now, I think PhoneGap is closest to this model, but a huge, huge, part of me wishes PhoneGap didn’t need to exist. If we could somehow skip the native shim and just take for granted that every platform supported, and at its core used, installable web apps. Maybe something like the WebOS model. But we’re not there yet. So for now, I’m glad I get to work with PhoneGap and build apps with web technologies. Eventually though I think PhoneGap can be used as inspiration for installable web apps. This is kind of how the standards world moves, as more and more people adopt something, people find ways to bring that something back into the standards. I think some of PhoneGap’s APIs and methodologies would make a great start at the idea of installable web apps. And I think the guys behind PhoneGap will be at the forefront of making those things happen, which means Adobe is going to be a really cool place to be over the next few years. It feels like there is a lot of potential to change the world and while I miss spending time with Flash, I feel like the HTML/JS/CSS work I’ve been doing and that Adobe is investing in, will make a similar impact on the web down the road.

Dealing with Binary Data from a Canvas Object using JavaScript TypedArrays

I’m not sure how helpful this will be for anyone, but during the process of doing a binary WebSocket demo I found myself learning a lot about JavaScript typed arrays and how that translates into binary data. The demo I wrote took Canvas image data and sent it over a binary WebSocket connection. The WebSocket server took that data and sent it out to all of the connected clients, who would then render the Canvas data as a PNG. It’s kind of a niche use case but I wanted to specifically create a binary WebSocket demo. It also was a more efficient way to send image data than doing something like base64 encoding it. First off, the Mozilla documentation was REALLY helpful. Major props to them.

Translating Canvas Data Into a Binary Format

Canvas has a getImageData() method that gives you an ImageData object. Within that ImageData object is a data property, which has the actual array data of the image. Normally I would have been able to stop right there because that would (in theory) have the information I needed. But what I had to get at was the ArrayBuffer. The way the spec has been implemented you can’t do anything with the ArrayBuffer. Instead you have to use an ArrayBufferView, which takes the form of TypedArrays in JavaScript. Luckily, to get the actual buffer you can just call the buffer property on any of the typed arrays and do what you want with it. But Canvas (at least in Chrome) is slightly different. The object you get from the ArrayBuffer of ImageData.data is something called CanvasPixelArray. Currently CanvasPixelArray doesn’t behave like a regular typed array, it looks like it will become a Uint8ClampedArray but the way it works in Chrome right now the CanvasPixelArray doesn’t provide access to a buffer property so you can’t send/access the ArrayBuffer data. Luckily getting that data into a Uint8Array, which you can get the buffer data from is pretty easy:

     imagedata = context.getImageData(0, 0, imagewidth,imageheight);
 
     var canvaspixelarray = imagedata.data;
 
 
     var canvaspixellen = canvaspixelarray.length;
     var bytearray = new Uint8Array(canvaspixellen);
 
     for (var i=0;i<canvaspixellen;++i) {
          bytearray[i] = canvaspixelarray[i];
     }

With that new Uint8Array all that we have to do is grab the buffer property and we can send it across the wire (beyond this post, but I’m planning on writing up the binary WebSocket info).

Reassembling the pieces

Now what I wanted to do was take that binary data from my Canvas and render it as a PNG file on the screen. The first step is to use a Canvas object to render out a PNG. But before that we have to get the data into a Canvas. In theory, you should be able to just do what happened above in reverse. But it’s not quite that simple. Once you get back the binary data from somewhere (WebSocket say), you’ve got an ArrayBuffer that you have to deal with. There is a putImageData() that takes an ImageData object, and we can create an ImageData object a few different ways, but you can’t set the data property of it. It’s read only. So we can’t take our data from the ArrayBuffer and just put it into our Canvas. We have to manually loop through the data property and line-by-line change the data.

          var bytearray = new Uint8Array(event.data);
 
 
          var tempcanvas = document.createElement('canvas');
               tempcanvas.height = imageheight;
               tempcanvas.width = imagewidth;
          var tempcontext = tempcanvas.getContext('2d');
 
          var imgdata = tempcontext.getImageData(0,0,imagewidth,imageheight);
 
          var imgdatalen = imgdata.data.length;
 
          for(var i=8;i<imgdatalen;i++)
          {
               imgdata.data[i] = bytearray[i];
          }
 
          tempcontext.putImageData(imgdata,0,0);

If the above isn’t clear, basically I’m just creating a new Uint8Array with the data from the server, then creating the temporary Canvas so I can get image data from it, and when I have that, I’m looping through the data property and inserting my own data from the Uint8Array.

Rendering it as a PNG

So now we have a Canvas (not being displayed) that has all of the data from our server, so it’s an exact graphical copy of the info we received. Turning that into a PNG is actually pretty easy because HTML is awesome. Canvas has a toDataURL() method that will take whatever is in the Canvas and create a string that can be put into the src property of an image. Then putting that image somewhere on the DOM will display the data as an image.

          var img = document.createElement('img');
               img.height = imageheight;
               img.width = imagewidth;
               img.src = tempcanvas.toDataURL();

Fin

Now that I’ve gotten my head around binary data a lot more, I’m kind of excited about JavaScript typed arrays. Looking through the list it looks like the typed arrays will help a lot with byte manipulation because of the different types. It also looks like they’re pretty fast (at least the fastest option at the time of that post).