About ryanstewart

Ryan Stewart is a Developer Evangelist for Adobe focusing on web technologies. He lives in Seattle where he likes beer, anything to do with geolocation, and spending as much time in the outdoors with his family as possible.

A Response to Benjamin Sandofsky’s Shell Apps

Dion Almaer retweeted an article by Benjamin Sandofsky that covers “shell apps”, or HTML/Hybrid apps in contrast to native apps. His thesis seems to be, from his post:

If you plan to write an app for iOS or Android, you will save time and create a better product if you stick to Objective-C and Java, respectively.

Which I agree with in principle, but I also found some of his arguments deeply flawed. He spends a lot of time talking about the cross-platform problem of trying to build a web-experience that looks native on iOS and native on Android. Specifically he calls out differences on iOS and Android:

The harder problems are inconsistencies in platform conventions. The iPhone uses Helvetica as its system font, while Android uses Roboto. iOS transitions between views by sliding new views in from the right, while Android uses a zoom effect.
You could recreate each platform in HTML5, but that requires a lot of work. You could design everything around an iOS look, and settle for things looking weird on Android. You could create a platform agnostic design language, making things equally weird for everyone. In a web view, it’s an uphill battle for consistency, whereas going native provides consistency for free.

Put aside the fact that if you’re doing native development you’re writing two different versions anyway (I’m not sure why that would be a knock against an HTML-centric solution), why do you have to design around an iOS or Android specific look?

The Native Component Question

I know it’s harder, and potentially more expensive, and requires more design sense, but why should developers of HTML-centric apps for mobile devices be locked into the specific UI look and feel of that platform? I’m not talking about UI paradigms (eg, soft back buttons on iOS, menu buttons on Android), but the specific look of apps. Instead of skinning your application to look like an iOS app and then try to make it look like an Android app, do a design that looks unique and polished that would translate well across both platforms. Then as you start to get into the specific differences (he brings up a good point about transitions) you can implement those device/platform-specific patterns. To me one of the beautiful things about the HTML/JS/CSS stack is the freedom with which we can develop. Bringing that to mobile applications, and letting some great CSS/HTML designers loose on that platform is a great thing. And it’s one of the reasons I love PhoneGap.

There are definitely some performance implications by going with a shell app/HTML-centric app approach. And as your application gets more complex, those will pop up more and more. So it’s something to be very aware of. Benjamin quotes some of the reasons he sees for people choosing HTML-centric apps: “we need to release faster”, “write once, run anywhere”, “HTML is easier”. I’m not sure any of those are great reasons to choose HTML-centric apps. Instead I see the primary benefits of the HTML approach as not being locked into a specific platform or deployment model. With HTML you can repurpose code for mobile web and you can target new platforms as they come (no device is not going to support HTML). But maybe most importantly, with HTML you can easily create custom app experiences. There are some great designers out there who can do amazing things with HTML and choosing something like PhoneGap lets you use those skills to create a unique web-like experience with all of the device features users expect from native apps.

Is it a silver bullet? Definitely not. But I think dismissing HTML-centric apps as being done because they’re cheaper/faster/easier doesn’t do them justice. For some kinds of apps, the HTML-centric approach is just better. There’s an amazing community of people that know HTML/JS/CSS, the web is the ultimate platform for reach and flexibility, and shell-apps bring those two elements to the native app world. That’s something to celebrate.

Presenting PhoneGap at the AT&T Hackathon – Seattle, April 14th

I’m going to be presenting/talking about PhoneGap at the AT&T Hackathon here in Seattle on Saturday, April 14th. The hackathon actually starts on Friday night, and the last one I was at was a really fun event. If you’re in the area I encourage you to come check it out, build an app, and win some prizes. I’ll be there in and out (not totally sure when I’m speaking) but I’m hoping to see some cool PhoneGap projects and chat with developers. Here are the details:

F5 Networks HQ
401 Elliott Avenue West
Seattle, WA 98119

Friday, April 13, 2012 at 6:00 PM – Saturday, April 14, 2012 at 9:00 PM (PT)

The PhoneGap Starter Project

As I’ve been digging more and more into PhoneGap Build I’ve also been discovering that there are some gaps in the workflow and that it’s not always easy to go from a feature I want to use to a cross-platform implementation for it. Most of the time I’ll use the PhoneGap documentation as a starter point and then try to wrangle it into a local project and then make that local project work with PhoneGap Build. One of my favorite features of PhoneGap Build is that it can pull directly from a git/svn repository and build binaries from that repo. So last night I did a little bit of work to pull a couple of features/code from the documentation into GitHub projects that can be quickly loaded into PhoneGap Build and turned into binaries. Fueled by some NW Peaks beer last night I built some initial ideas for the PhoneGap Starter Project. As you can see it’s pretty sparse but there are a few goals I have in mind.

  • Showing how to structure projects for use in PhoneGap build. I struggled with this a bit–with things like making the config.xml file work, setting the files up correctly (in a www folder, and referencing cordova.js in the script tag but not including it), etc. Each one of these projects works with PhoneGap Build out of the box.
  • Linking documentation to a real project. The documentation for PhoneGap is pretty good and I wanted an easy, out-of-the-box way for people to see the features in action. So I took the code from the documentation and turned them into projects. Load them into PhoneGap Build and you can see them in action on your device in a matter of seconds.
  • Helping fix documentation! I’m looking for ways to contribute to the Cordova project and some of the documentation code is a bit out of date. What better way than to go through that code, make sure it works, fix bugs, and then submit them back?
  • Showing how kick ass PhoneGap Build is. I really, really like PhoneGap Build and once you get a good workflow down I think it’s a great help in creating binaries quickly. This shows it in action.

To use the project in PhoneGap Build, just create a new app from SVN and put in the URL of the repo.

Adding an App to PhoneGap Build

Creating a new app based on Git/SVN project

I’m going to be filling out some of other documented features over the week as well as doing an example for the ChildBrowser plugin. I’m curious to hear if this seems useful for people or solves any kind of pain point. Since it helps me find the gaps/issues in the documentation, it serves a higher purpose for me, but if it seems useful then I’ll work on keeping it up to date.

Web Versus Native – Asking the Wrong Questions

There’s been a lot of talk lately about web apps versus native. It’s not that the question has ever gone away, but just over the past couple of weeks it seems like it’s been called out a bit more in blog posts. There was a Pew Internet report that covered some of the pros/cons of going web or native. Darryl Taft, one of my favorite press guys, did a writeup earlier this month on the landscape of web versus native and some of the companies involved in blending those two worlds. And then today I saw news about Mojito, a hybrid client-server solution from Yahoo, which is part of a wider initiative called Cocktails with the aim of “making web applications feel native“. And that got me thinking about the problems that people are trying to solve when it comes to web apps on mobile devices.

Web apps are in a tough spot when it comes to native, but they’re getting better. As I see it, one of the main problems with web apps is that they aren’t currently better. They’re cheaper, they’re faster (to write), and they allow you to target more platforms, but they’re not better. What’s even worse is that you could argue there are only 2 platforms for mobile that matter right now, Android and iOS. But it was the exact same situation back in 2003/2004 when the web really started coming into its own as an app platform and taking away from traditional native apps on the desktop. The one problem is that back then the web apps were better. They didn’t look as fancy and they didn’t run quite as fast, but there were few things that people needed to do that the web couldn’t do. I remember talking about AIR and mentioning things like drag and drop, file system access, and file type associations. Those were nice, but people didn’t really need them. What made web apps better for end users was the fact that they ran everywhere, were lightweight, always up-to-date and moved with the user so when they switched between the library, school, work, and home, their stuff went with them.

But fast forward to today. We never leave our mobile devices. They’re always with us. Sometimes we change contexts from our mobile device to a desktop/laptop/tablet (think email) but ultimately one of the things that made web apps so great, that they moved with the user, has been completely negated by smartphones. And increasingly, what web apps can’t do is much more important than what web apps couldn’t do 5 years ago. All of the sensors; GPS, Accelerometer, Camera, not to mention the data on the phone, things like contacts, that can be used to make things on the device so much more personal and completely change the experience. Some of the basic ones are available to web apps, and the specs are in place to get more, but there’s still a significant disconnect between what can be done with native apps on the device and what can be done on the mobile web.

To me that’s the biggest worry for the web on mobile devices right now. Sure, performance isn’t ideal, but as the browsers get better and devices get more powerful, that one will fix itself. The technology is there in the form of Canvas, SVG, great JS frameworks, faster JS performance. So the core interactive features work on these devices (with some performance caveats). But the benefits of the web are in many ways nullified by smartphones today. The web needs to come up with something that makes it better once again. And the deck is really, really stacked against it. Think of the distribution model of apps and in-app purchases. I think you could argue that those aren’t great for customers as much as they are great for content creators. Mobile apps are easier to monetize, easier to track, and easier to manage than the web right now. And on one hand, that’s fantastic. Finally designers and developers can directly make money off of their creations. But it’s coming at the expense of the web.

In order for web apps to thrive again the web community needs to start thinking about how mobile web apps can be better and more useful to customers than native apps. I have no doubt that it’s going to happen. How many times have you said to yourself “I don’t want your app, I just want to see the content” while browsing around the web on a mobile browser? So the sentiment from users is there, but for now the web just doesn’t provide enough value for users to get beyond the inflection point. So I’m hoping the conversation starts to move away from web apps not being fast enough, or “feeling native enough” to one of how web apps can be better than native from a user perspective. It’s going to be a fun conversation to watch unfold.

Experimenting with CSS Exclusions

Okay, this is going to be scattered and on the cutting edge side, so buckle in. Adobe has been hard at work making the web a little bit richer when it comes to how type is displayed. A great example of that is CSS Regions, which let you create linked islands that text can flow between so you have a lot of control over how text gets laid out on the screen. That’s currently in the “Canary” builds of Chrome and if you go to about://flags and turn on “CSS Regions” you’ll be able to view the demos. The other thing we’ve been working on is a bit further out, but it’s in the same vein. It’s called CSS exclusions. It essentially lets you create more interesting islands of text in the form of shapes that text will flow around. At this stage it’s still very, very early. In fact, the only way to test CSS exclusions content is with a special build of WebKit that’s available on Adobe Labs.

The Backstory

A lot of the demos associated with these features are associated with magazines, and especially with the iPad 3 coming out, it’s going to be critical to be able to build expressive content that scales. Since magazines are mostly text, expressive text will be a big deal. I’ve been mulling over doing a beer-related magazine and figured it would be cool to try out some of the CSS demos to create something that might look unique and fit in a beer magazine. So I thought “hey, I could create shapes of different beer glasses and run text through those!” That was the basic premise. As I dug in, I found a couple of caveats. Most I’ll get to throughout the post, but the biggest one is that there currently isn’t a version of WebKit that has both CSS Exclusions and support for flowing across CSS Regions. So my original idea ran into a small roadblock but I’ll be able to fix that down the road.

Creating regions

The CSS Exclusions spec supports SVG’s basic shapes. That includes circles, rectangles, and polygons. At this point it can’t do arbitrary path data but you will be able to use SVG data according to the spec. So I started with creating the various types of beer glasses in Illustrator with the polygon tool. Then I had to set up the glasses on the page and make text flow correctly. The standard method seems to be to create two CSS classes, one for the properties that every region or shape will have, and another class for the specific region, in this case, glass type. I called my main region container and it looks like this:

     .container{
          position:absolute;
          height:500px;
          width:300px;
          text-align:justify;
          margin-top:0px;
          font-size:10px;
 
          -webkit-wrap-shape-mode: content;
          -webkit-render-wrap-shape: none;
          -webkit-hyphens:auto;
     }

The main properties are those three on the bottom. The first one tells the browser what I’m doing, and that’s wrapping the content within a shape. I think the other option here is around, which would make the shape an exclusion but I don’t think that’s working in my build of WebKit (or I’m doing it wrong). The render-wrap-shape is good for debugging as it shows you an outline of the shape so you can see how it’s appearing. I found this incredibly helpful as I laid objects out on the page. The last one, hyphens, lets you tell the browser how you want it to break up text. If you go for none there, it will only flow full words within the content. Otherwise it fills the content as much as it can before breaking words with a hyphen. For fuller-looking shapes, go with hyphens set to auto.

Next I had to set up the specific glass shapes. The hardest part for me was lining things up correctly. The way I created the polygons I think made it tougher to lay out. It also doesn’t change as you resize the page because I’m not using percentages (if that’s even possible with polygon objects). It looks kind of ugly but to define a shape we just have to use webkit-wrap-shape and give it the values. Here are a couple I did.

     #irishpint1 {
          padding-left:0px;
     }
 
    .irishpint {
          -webkit-wrap-shape: polygon(157px,473px 106px,472px 93px,453px 91px,432px 87px,401px 84px,368px 78px,332px 70px,294px 61px,255px 57px,225px 55px,191px 55px,165px 57px,128px 63px,118px 165px,118px 274px,120px 279px,128px 282px,164px 282px,190px 282px,226px 274px,263px 267px,296px 257px,337px 253px,368px 250px,397px 246px,427px 244px,453px 233px,473px 173px,473px);
     }
 
 
     #weissbierflute1 {
          padding-left:250px;    
     }
 
     .weissbierflute {
          -webkit-wrap-shape:  polygon(104px,470px 82px,459px 81px,439px 89px,400px 105px,344px 102px,294px 86px,238px 72px,183px 66px,128px 62px,77px 59px,39px 58px,17px 57px,0 280px,0 279px,17px 278px,41px 276px,74px 271px,128px 265px,182px 251px,238px 234px,299px 232px,344px 248px,400px 255px,434px 252px,463px 215px,473px);
     }

So they’re some semi-complex polygons. I also used the div tags to set up padding and lay things out. One of the things I learned is that CSS regions/shapes can only be applied through classes, you can’t define the properties based on id’s. But that may change.

So with all of those glass polygons, it looks like this:

What the page looks like in a browser that supports CSS Exclusions

What the page looks like in a browser that supports CSS Exclusions

That page is up on GitHub but you won’t be able to view it in your browser so you’ll have to be looking at it with the special browser from Adobe labs.

Making it More Interesting

After this I thought it would be cool to do a bit more with the shapes. Beer has a rating scale for color called the SRM scale. Types of beer have a general range of SRM scales and since some glasses roughly correspond to specific types of beers I wanted to see about creating a background for each shape that would correspond to the SRM scale. But even more than that I wanted it to change with the glass so as the glass got narrower, the background would become more transparent just like a real beer glass. Unfortunately it didn’t work quite the way I wanted, but I learned a bit about using CSS shapes along the way. Originally I just started by giving the div a background:

     #irishpint1 {
          padding-left:0px;
          background: -webkit-gradient(linear, left top, left bottom,
                                             color-stop(.8, rgba(152,83,54,1)),
                                             color-stop(.2, rgba(152,83,54,.7)));
     }

But the CSS shapes/exclusions don’t actually change the shape of the div, they’re only worried about wrapping the text within the div. So my page just looked like this:

What the div tag looked like when trying to set the background

What the div tag looked like when trying to set the background

Luckily webkit-mask-image came to the rescue. With that, I could create a mask based on a file, in my case an SVG file, and use that to mask a block element like a div tag. So I went through and created an SVG file for every glass shape and then used those as the mask and set the background accordingly with rbga() properties in a gradient. The final version looks like this:

And here’s a clean(ish) version without the backgrounds. I kind of like this one better.

Update: If you have a browser that supports exclusions you can check out the demo here.

Final Thoughts

It’s really, really early for CSS Exclusions but there is a LOT of potential there and even the stuff that works now is a lot of fun to use. This was just an initial dive into the feature to kick the tires, and because it’s so new, there are a lot of gaps both in my knowledge and what I was able to do. One of the things I tried was using CSS3 transitions to animate the -webkit-wrap-shape property. That didn’t seem to work, it could be on the roadmap.

It’s a fun feature and it’s really fun to be able to give it a shot while it’s still in the early stages. For magazines or any kind of rich text content, exclusions are going to be a fantastic way to make stuff pop. If you want, you can check out the project on Github. I’ll probably be modifying it and improving it as the spec evolves.

Installing the ChildBrowser Plugin for iOS with PhoneGap/Cordova 1.5

Note: I’ve added a ChildBrowser example to the PhoneGap Starter project. It won’t help with native issues, but if you’re using PhoneGap Build it might give you a working head start.

I’ve seen a bunch of people who are new to PhoneGap/Cordova struggle with getting the ChildBrowser up and running. I fought with it today as well and most of my mistakes were based on old blog posts so I wanted to do a quick post that hopefully shows up in Google for people running into the same problems I had.

I’m starting with a blank project. The ChildBrowser plugin I’m using is here which as of this writing has been updated to support Cordova 1.5 with some potential backwards compatibility with 1.4.1. The simplest thing is to probably just download the source as a zip.

I’m starting with a blank project that has my www folder already referenced by the project. Here are the rest of the steps.

1. Drag all of the .m, .h, and .bundle file(s) from the ChildBrowser folder to the Plugins folder of your project. Make sure the copy option is selected (I think it’s the default).

2. In your project, under the Supporting Files folder, open the Cordova.plist file. You’ll see a bunch of key/value pairs and the one you want is under Plugins. You’ll need to add 2 by clicking the add button that shows when you highlight the Plugins section.

Key: ChildBrowser, Type:String, Value: ChildBrowser.js
Key: ChildBrowserCommand, Type:String, Value:ChildBrowserCommand

3. You also need to make sure that your application can call the external URL you want to load. This is also handled in the Cordova.plist file under ExternalHosts. Add any of the URLs you plan to open to that. You just need the Value, so you can leave the Key as the default. And remember that you can use wildcards, so the entry below lets you go to any subdomain of Google.

Key: Item 0, Type: String, Value: *.google.com

4. Copy over ChildBrowser.js (or the minified version) from the project files to your www folder.

Now you need to go into your HTML and JS files and make some changes. The biggest one is to reference the ChildBrowser.js file.

<script type="text/javascript"charset="utf-8"src="ChildBrowser.js"></script>

And here’s the modified JavaScript I used to get it to work. I created a childbrowser variable, then in onDeviceReady added a call to the install method of the ChildBrowser object. Finally I created a function that handles the URL opening.

    var childbrowser;
 
function onBodyLoad()
{
document.addEventListener("deviceready", onDeviceReady, false);
}
 
function onDeviceReady()
{
// do your thing!
childbrowser = ChildBrowser.install();
navigator.notification.alert("Cordova is working")
}
 
    function onLinkClick() {   
 
    if(childbrowser != null)
    {
        childbrowser.onLocationChange = function(loc){ alert("In index.html new loc = " + loc); };
        childbrowser.onClose = function(){alert("In index.html child browser closed");};
        childbrowser.onOpenExternal = function(){alert("In index.html onOpenExternal");};
 
        window.plugins.childBrowser.showWebPage("http://google.com");
 
    }  
}

This definitely works with Cordova 1.5 and I’m pretty sure it should work with 1.4. Notice that there’s no need to change any of the .m/.h files, so don’t worry about importing anything or changing code that you may have seen in other blog posts.

And I’m pretty sure this is basically how all Cordova/PhoneGap plugins work. Simply copy the OS-specific files into the plugin directory and then copy the JS files and reference them. But ChildBrowser threw me for a loop because of all the blog noise that referenced past versions. So hopefully this helps someone.

Speaking at the LA PhoneGap Meetup on Wednesday, February 15th

I’m doing a presentation in conjunction with HTC at the LA PhoneGap meetup on Wednesday, February 15th, so if you’re in the area, come by. I’m going to be covering Debugging and Deploying with PhoneGap, which includes talking about using the debug tools like weinre and using PhoneGap Build to deploy it across a bunch of platforms. From the site:

AGENDA:

1. Android Tablet Development (Lance Nanek & Michael Ludden, HTC)

2. Debugging and Deploying with PhoneGap (Ryan Stewart, Adobe)

===============================

Android Tablet Development. HTCdev has created a PhoneGap plugin for their Scribe Pen enabled tablets, one of which they will be giving away to a lucky developer. Check out www.htcdev.com for details.

Two of their lead developer evangelists will be demonstrating how HTCdev can help apps gain visibility, and how to easily program apps that let you create using Pen-based input.

Presenters:

Michael Ludden (twitter.com/Michael_Ludden) is a Developer Evangelist at HTC. Tasked with developer enablement, Michael is focusing on how best to help developers differentiate without fragmenting on HTC devices – creating great, unique user experiences that widen appeal of apps and catch that all important user’s attention. He will be showing how developers can take advantages of HTC’s app promotional programs, support, co marketing and robust toolsets to maximize an app’s influence. He has a bachelor’s degree from UCLA and has been working in development and with developers for years.

Lance Nanek (http://nanek.name) is a Developer Evangelist at HTC. He has ten years of experience in information technology: implementing web metrics for ibm.com at IBM, writing Java web applications at State University of New York System Administration, and working on Android apps for many companies, including CardioTrainer by WorkSmart Labs with over three million downloads. He has a master’s degree in Computer Science from State University of New York University at Albany.

Debugging and Deploying with PhoneGap. In this session, Ryan Stewart, a web developer evangelist from Adobe, will cover how to debug and deploy PhoneGap applications. He will talk about how to use Winere to get some insight into how your PhoneGap application is behaving on the device and how to set up the application so that it can easily be debugged. He will also cover some of the deployment options, including a deep dive on the ways to use PhoneGap Build to deploy quickly across multiple devices and operating systems. You’ll learn about things to be aware of, and things you can do to streamline the process.

Hoping for a Web App Future

If I think back to 2006/2007 I was very happy with how things were shaking out. The web was on the upswing and we were moving away from native applications. All of the great things about the web–its ubiquity, its freedom, its openness–were being harnessed to create native-like experiences that, I hoped at the time, would see us all do away with native apps. At the time, there wasn’t much (I thought) that web apps couldn’t do that we needed native apps for. How wrong I was. With the introduction of the iPhone, and subsequent smartphone releases, we’ve seen a huge shift back to native applications. Part of that is performance, right now native apps just feel better than mobile web apps, but it also came about because of just how many things native mobile apps could do. Geolocation, accelerometers, contact info–the smartphone showed how many things “apps” needed access to and for the most part those features have been exclusive to native applications. So just as the web was starting to really take off, we’ve slid back into native application territory.

It bummed me out, and still does. I thought AIR was an okay solution to the problem, but by the time AIR came around it was pretty clear that “the web” had come to mean HTML/JS, and I’m fine with that. So as PhoneGap started getting traction, and then Adobe took a major interest in the project, I was excited about the prospect of working on it as an Adobe evangelist, and more importantly, working with the teams behind it to see what else they had up their sleeves as the web moves forward.

But another side benefit is that it’s put me on what I’d consider the “right” side of the web argument. Two things got me thinking more about this. One was a very good post by a VC named Mark Suster, who while not telling his companies to focus exclusively on the web, has told them to make it a big part of what they do. So many startups nowadays are thinking completely mobile-first while ignoring the web, I’d say at their peril. When I think of my own usage, I’m still using a lot of native apps (for reasons I can’t quite figure out) but the ones I enjoy most have a web component that is well done and part of the overall experience. Yelp comes to mind. Another is Untappd. I can do almost anything I need to on the website version of Untappd so it’s not as though I’m getting a watered down experience. It more easily lets me move between contexts and devices while still using the service. I contrast that to something like Foursquare or Path. Both are mobile-centric, and with Foursquare I can’t check in because it wants to be sure that I am where I say I am (using GPS) but it still makes the web side of it less useful for me. Path is unusable when you’re not on a mobile device as far as I can tell. When I log in all I get is “download the app”. Which I hate. Mobile is clearly important but the web can’t be ignored.

Path Login

If this is what I see when I log into your site, you're too mobile-first.

The second thing that got me thinking more about this was something Brian Leroux pointed to on the Cordova mailing list, a post by Tim Berners-Lee about Installable Web Apps. This is a model I would love to see take hold. As Tim notes, there are a few things that users need to have when they’re installing web apps, and some trust/permissions issues that need to be figured out. Right now, I think PhoneGap is closest to this model, but a huge, huge, part of me wishes PhoneGap didn’t need to exist. If we could somehow skip the native shim and just take for granted that every platform supported, and at its core used, installable web apps. Maybe something like the WebOS model. But we’re not there yet. So for now, I’m glad I get to work with PhoneGap and build apps with web technologies. Eventually though I think PhoneGap can be used as inspiration for installable web apps. This is kind of how the standards world moves, as more and more people adopt something, people find ways to bring that something back into the standards. I think some of PhoneGap’s APIs and methodologies would make a great start at the idea of installable web apps. And I think the guys behind PhoneGap will be at the forefront of making those things happen, which means Adobe is going to be a really cool place to be over the next few years. It feels like there is a lot of potential to change the world and while I miss spending time with Flash, I feel like the HTML/JS/CSS work I’ve been doing and that Adobe is investing in, will make a similar impact on the web down the road.

Dealing with Binary Data from a Canvas Object using JavaScript TypedArrays

I’m not sure how helpful this will be for anyone, but during the process of doing a binary WebSocket demo I found myself learning a lot about JavaScript typed arrays and how that translates into binary data. The demo I wrote took Canvas image data and sent it over a binary WebSocket connection. The WebSocket server took that data and sent it out to all of the connected clients, who would then render the Canvas data as a PNG. It’s kind of a niche use case but I wanted to specifically create a binary WebSocket demo. It also was a more efficient way to send image data than doing something like base64 encoding it. First off, the Mozilla documentation was REALLY helpful. Major props to them.

Translating Canvas Data Into a Binary Format

Canvas has a getImageData() method that gives you an ImageData object. Within that ImageData object is a data property, which has the actual array data of the image. Normally I would have been able to stop right there because that would (in theory) have the information I needed. But what I had to get at was the ArrayBuffer. The way the spec has been implemented you can’t do anything with the ArrayBuffer. Instead you have to use an ArrayBufferView, which takes the form of TypedArrays in JavaScript. Luckily, to get the actual buffer you can just call the buffer property on any of the typed arrays and do what you want with it. But Canvas (at least in Chrome) is slightly different. The object you get from the ArrayBuffer of ImageData.data is something called CanvasPixelArray. Currently CanvasPixelArray doesn’t behave like a regular typed array, it looks like it will become a Uint8ClampedArray but the way it works in Chrome right now the CanvasPixelArray doesn’t provide access to a buffer property so you can’t send/access the ArrayBuffer data. Luckily getting that data into a Uint8Array, which you can get the buffer data from is pretty easy:

     imagedata = context.getImageData(0, 0, imagewidth,imageheight);
 
     var canvaspixelarray = imagedata.data;
 
 
     var canvaspixellen = canvaspixelarray.length;
     var bytearray = new Uint8Array(canvaspixellen);
 
     for (var i=0;i<canvaspixellen;++i) {
          bytearray[i] = canvaspixelarray[i];
     }

With that new Uint8Array all that we have to do is grab the buffer property and we can send it across the wire (beyond this post, but I’m planning on writing up the binary WebSocket info).

Reassembling the pieces

Now what I wanted to do was take that binary data from my Canvas and render it as a PNG file on the screen. The first step is to use a Canvas object to render out a PNG. But before that we have to get the data into a Canvas. In theory, you should be able to just do what happened above in reverse. But it’s not quite that simple. Once you get back the binary data from somewhere (WebSocket say), you’ve got an ArrayBuffer that you have to deal with. There is a putImageData() that takes an ImageData object, and we can create an ImageData object a few different ways, but you can’t set the data property of it. It’s read only. So we can’t take our data from the ArrayBuffer and just put it into our Canvas. We have to manually loop through the data property and line-by-line change the data.

          var bytearray = new Uint8Array(event.data);
 
 
          var tempcanvas = document.createElement('canvas');
               tempcanvas.height = imageheight;
               tempcanvas.width = imagewidth;
          var tempcontext = tempcanvas.getContext('2d');
 
          var imgdata = tempcontext.getImageData(0,0,imagewidth,imageheight);
 
          var imgdatalen = imgdata.data.length;
 
          for(var i=8;i<imgdatalen;i++)
          {
               imgdata.data[i] = bytearray[i];
          }
 
          tempcontext.putImageData(imgdata,0,0);

If the above isn’t clear, basically I’m just creating a new Uint8Array with the data from the server, then creating the temporary Canvas so I can get image data from it, and when I have that, I’m looping through the data property and inserting my own data from the Uint8Array.

Rendering it as a PNG

So now we have a Canvas (not being displayed) that has all of the data from our server, so it’s an exact graphical copy of the info we received. Turning that into a PNG is actually pretty easy because HTML is awesome. Canvas has a toDataURL() method that will take whatever is in the Canvas and create a string that can be put into the src property of an image. Then putting that image somewhere on the DOM will display the data as an image.

          var img = document.createElement('img');
               img.height = imageheight;
               img.width = imagewidth;
               img.src = tempcanvas.toDataURL();

Fin

Now that I’ve gotten my head around binary data a lot more, I’m kind of excited about JavaScript typed arrays. Looking through the list it looks like the typed arrays will help a lot with byte manipulation because of the different types. It also looks like they’re pretty fast (at least the fastest option at the time of that post).

HTML/JS/CSS and Tooling

I enjoyed this post by Grant Skinner that walks through his view of the evolution of technology and where/how/when tooling starts to come in. Adobe makes tools for web professionals. That’s what we’ve always done and that’s what we’ll do for a long time. You could even paint a broader brush that we make tools for creative people to share their creations. Watching our own evolution over the past year or so with regards to HTML tooling has been very interesting. We got some flack for not moving in earlier, but as Grant rightly points out, tools are a major investment and only once you have stability can you make that investment. It was never a matter of momentum around HTML or a focus on Flash, it was just the fact that things weren’t quite ready for tools.

In fact, I’d argue they still aren’t. But we’ve taken that as something that comes with the web. It’s always evolving, always moving, and while things will start to coalesce more and more, in the end, you have to get in and be ready to move. That’s kind of the approach we’ve taken with Adobe Edge. We just released Preview 4 of Edge which incorporates a lot of features that people have been asking for. Some of it I’m not even sure if it was on the original roadmap. But the Edge team made a conscious decision to be very agile, to build Edge in such a way that it could incorporate customer feedback quickly, and then getting product management on board to do lots of versions very quickly. I think it’s worked out very well and despite being on the earlier side of Grant’s curves, I think Edge will be a very helpful tool for a lot of people because of it.

Developer tools are a bit of a different story in terms of both ecosystem and readiness in my mind. There isn’t one, big, HTML IDE that people seem to like (akin to Flash Builder, Eclipse or Visual Studio). Instead people seem to be using a lot of different things and experimenting. What actually seems to be most popular right now are the basic text editors like TextMate or Sublime (my favorite). These seem to be focused on helping smart people work faster. Lots of shortcuts, lots of snippets, but not a lot in terms of helping along the learning process. And I think that’s just where we are now as far as HTML/JS/CSS tooling. But I’m excited to see that evolution as well and see what happens when frameworks get a bit more standardized and more general web developers start jumping into JS more and more. Will those people need a more robust HTML/CSS/JS editor that’s still developer centric? And I think the answer is yes, but I think it’s also tough to really see what that would look like until the JS/HTML/CSS stack is a bit more solid. But I’m excited to watch it and find out. And from what I’ve seen of Adobe’s HTML tooling side, we’re taking a good approach and I’m excited to see what people think as the PMs share more and more of it.