A Template for Lunches

work safe

If you want to show anything on your page, you’re going to have to write a template. A template is the html to display on each page view. And if you use resources, they nest! Ember, by default, uses the Handlebars template engine.

Lunchtime!

It’s a Friday and I want ribs for lunch. But I don’t want to go alone. I’ll just write an intranet app that lets me invite a few coworkers to lunch. Because email is hard.

Begin at the beginning.

The default template is the application template. you can tell it’s the application because it doesn’t have a name. The application template is used to lay out the main page structure and all the site branding. all other templates are nested inside and show up where the {{outlet}} tag is used.

<br/>
<script type="text/x-handlebars">  
<h1>Lunch-o-Matic!</h1>  
<div class="subtext">Meet your coworkers for lunch</div>  
{{outlet}}  
</script><br/>

But, that alone isn’t very interesting, is it?

Okay, lets start off by managing some expectations. I’ll cover Controllers next week, but I’ll be mentioning them often. About 1/2 of the cool things about templates have some counterpart in the controller supporting them. That said, I’m getting hungry. Let’s gather up the team and head out for Barbecue!

I Decide who Leaves and who Dines!

I need to display a list of my coworkers so I can pick a couple of them.

<br/>
<script type="text/x-handlebars" data-template-name="lunchPicker">  
<h2>Who's in?</h2>  
{{input value=lunchType placeholder="What's for Lunch"}}  
<table><tr><th>invite?</th><th>name</th><th>email</th></tr>  
{{#each employee in employees}}  
  {{#if employee.isAvailable}}  
    <tr><td>{{input type="checkbox" checked=employee.isSelected}}</td>  
        <td>{{employee.fullName}} ({{employee.initials}})</td>  
        <td>{{employee.email}}</td>  
    </tr>  
  {{/if}}  
{{/each}}  
</script><br/>

A few things to note in the above example. I’m using an {{input}} block to make a text field that binds to the controller property lunchType. That will be used to let everyone know what kind of food we’re going to eat! Next, I’m looping over the a list of employees using an {{#each}} block. The collection comes from another controller property. Each element is then scoped as ‘employee’ inside the each block. I can conditionally skip displaying an employee if they are not available. more correctly, I can use an {{#if}} block to show only the available employees. I display a checkbox using a built in helper, {{input}}. It binds the checked state of the input control to the isSelected property of the employee. The last three are simple data bindings to text. They just forward properties from the model to html.

The Web Without Links isn’t the True Web.

Let’s tweak the employee listing. I want to make the name link to their info page. I can use the {{link-to}} block.

<br/>
...
<br/>
 <tr><td>{{input type='checkbox' checked=employee.isSelected}}</td><br/>
 <td>{{link-to 'employee' employee.employee_id}}<br/>
 {{employee.fullName}}<br/>
 {{/link-to}}<br/>
 ({{employee.initials}})</td><br/>
 <td>{{employee.email}}</td><br/>
...<br/>

But we need to DO something!

Wonderful! Things are moving right along. Now, how do I submit my selections? With an action! An action is a function in the controller. I promise, we will get to that later. Just trust me when I say our bindings put all the information in the model, so we don’t need to pass anything to the function. We just need to bind the action to a button, right after the end of our {{#each}} block.

<br/>
...
<br/>
<button {{action 'pickThem'}}>Choose these Coworkers</button><br/>

We can pass parameters in to the function, but we don’t need to at this time. In addition, we don’t have to worry about controlling event propagation or target a different controller. All of these are possible, just not necessary for us.

Your Bill Comes with Mints

There are a few things we can do to help us clean up our code from here.

Helper isn’t just for Hamburgers

A helper is a function that gets called like a block. It returns text to be displayed in the Html. If you need to format dates, for example, or want to display an employee’s spark-line for their time on the tread mill in the past two weeks.

Partials

Partials are templates that can be reused. They bind to what ever model or controller is in the context, just incase your data doesn’t look like you expected because you named something slightly wrong.

Views

A view can pick it’s own template. It has a bit more logic, but I’ wouldn’t go crazy building complex controls with them. In a couple weeks, we’ll cover Components and then we’ll go nuts.

1 Comment

A Picture is Worth 1000 Lies

work safe

Last week, I wrote about Routing in EmberJS. And I kinda lied a bit. I laid out the Pokédex route like this …

<br/>
App.router.map(function(){<br/>
 this.route("pokedex", {path: '/pokedex/:poke_id'});<br/>
});<br/>

That’s probably the wrong choice, but I made it for Pedagogical reasons. I regret nothing, but we are going to play with the routes of an example application to think through some design decisions and pick up a small detail of the router I overlooked. Resources.

A Picture is worth 1000 words

I’m a Father, Husband, etc. That means we are generating Gigabytes of pictures. Some are even in focus. She’s two, and fast. So, we are going to build a photo sharing site! Let’s give it a classy name. I’m thinking Phlickr. Um, legal informs me that’s a bit too similar some other upstart.

Introducing PhotoAlbum

PhotoAlbum is the latest in picture sharing technology. Some of our neat features are:

  • Friends
  • Albums
  • Tags
  • Favorites
  • Popular Pictures
  • Searching
  • Dashboard

Laying the Groundwork

I’m going to start by giving some examples of pages and their Urls.

  • Index – ‘/’
  • Login – ‘/login’
  • Dashboard – ‘/dashboard’
  • Popular Pictures – ‘/popular’
  • My Friends – ‘/friends’
  • Search – ‘/search/’
  • Tags – ‘/tags/’
  • Tag – ‘/tag/Sunny%20Days’
  • My Page – ‘/U/Brian’
  • Viewing My Album – ‘/U/Brian/album/sketchnotes’
  • My Albums – ‘/U/Brian/albums’
  • Creating a new album – ‘/U/Brian/albums/new’
  • My Friends Albums – ‘/U/Matt/albums’
  • A Photo – ‘/U/Brian/photo/24153′
  • My Photos – ‘/U/Brian/photos’
  • Uploading a photo – ‘/U/Brian/photos/new’

Cartography in Code (Making a Map)

Router.map(function() {
 this.route('login');
 this.route('dashboard');
 this.route('popular');
 this.route('friends');
 this.route('search');
 this.route('tags');
 this.route('tag', {path: '/tag/:tagname'});
   this.resource('posters', {path: '/U'}, function(){
     this.resource('user', {path: '/:username'}, function(){
       this.route('album', {path: '/album/:albumname'});
       this.resource('albums', function(){
         this.route('new');
       });
       this.route('photo', {path: '/photo/:photo_id'});
       this.resource('photos', function(){
         this.route('new');
       });
     });
   });
});

What’s the Difference?

Routes are for individual pages. Routes cannot be nested. If we want nested pages, we use resources. And resources give us a few new defaults.

Defaults

By default, the router gives us the route named ‘index’. And that in turn builds the controller, route, and loads the template named ‘index’. A resource is like a mini router in that way. If we name a resource ‘posters’, then all the child routes are prefixed with the name, including a new default ‘index’. The tetrad is listed out below. When deeply nested, only the resource and route matter for naming as opposed to the parent resources. Example, the template for the new albums is ‘albumsNew’ despite the albums resource being a child of the user resource.

  • RouteName – postersIndex
  • Controller – PostersIndexController
  • Route – PostersIndexRoute
  • template – postersIndex

Chains

The defaults are not the only things. They nest, just like applications. The posters resource looks for a template named ‘posters’ and then the child template. While I won’t dwell too much templates now, remember when the application template had an {{outlet}}? Well, the ‘posters’ template can have an {{outlet}} as well. this allow the templates to nest.

But it’s not just the templates, the controllers nest as well. So the model and properties of a parent controller can be accessed by a child. And the routes nest. Each parent finishing before the child runs.

A Time to Choose

Of course we can use routes to describe everything here. I’m fairly sure. But we will have all kinds of issues with reuse. the nesting of controllers and routes gives us a sensible place to hang some code reuse. The really interesting thing is I chose my routes and resources strictly based on what Urls made sense, what parts belong to other pieces of information. For example, there is only one dashboard, yours. The login is also independent of other data. However, each photo album belongs to a user. Matt’s album Dinners I Made is totally different from my album Dinners I Made. And while a picture has a tag, the collection of tags is only useful when it belongs to the system as a whole, allowing me to find all the pictures tagged sunrise regardless of who posted it. Once again, it all comes down to following the Urls and letting them tell you all you need to know.

No Comments

Getting from Here to There

work safe

Where do we go from here? We begin with the Url. The history of the web is based around the URL and our ability to transition from one url to another. Ember is built around the Url.

You are a Web Developer if you build apps with Urls

- Tom Dale

Every view we want to represent in our application will be represented as a different Url. When we want to show a specific piece of data, we use a different Url. Want to modify how we are showing the screen, then we add data to the Ulr. This will allow us to Deep Link and share specific views and app state with colleagues and coworkers. This is how the Web has worked since the web began. And Below, we see how the Url communicates with Ember to make choices on what to show the browser.

Ember Url Structure

Ember’s Url Structure taken from the presentation mentioned above.

The Router

The router dissects the Url and chooses which Route, Controller, Model, and Template to compose, execute, and use the results for displaying to the screen. By default, you get an ‘index’ route for free. As if I went to http://example.com/, that would be the default index.

Baby-steps to Customization

I can specify just a route named ‘about’ and that will show an ‘about’ template that will use the path http://example.com/about without specifying that path.

App.Router.map(function(){
    this.route("about");
});

A Route With a Different Name, Would It Navigate as Sweet?

Lets say I have a dumb legacy Url (http://example.com/meet-the-team) I need to keep working. Because Urls are important. But I want my internal code to use clear names (about). I can specify the path used in the route mapping.

App.Router.map(function(){
    this.route("about", {path: '/meet-the-team' });
});

My Pokémans, let me show you them.

One more mapping to cover for now, before we start talking about the magic a route actually does. What if we want to show the same page, but vary the data based on part of the path? I have a super awesome collection of Pokémans, let me show you my favorite Squirtle. I keep him at http://www.example.com/pokedex/poke_1. I can specify part of the path to be a dynamic segment. The name of the dynamic segment will show up as a param in the Route code.

App.router.map(function(){
    this.route("pokedex", {path: "/pokedex/:poke_id"});
});

Spent All That Time Defining

It’s neat and all that I’ve just created a route with a dynamic segment. But I need to use that to actually do work. I can create a PokedexRoute and override it’s model creation!

App.PokedexRoute = Ember.Route.extend({
    model: function(params){
        return Ember.$.getJson('api/pokemon/' + params.poke_id);
    }
});

What’s a lifecycle?

There are so many hooks in the route that I can override. I can override the controller creation. I can specify a non-standard template. Oh, that might be fun! This is the BEST pokemans, so he needs a custom template.

App.PokedexRoute = Ember.Route.extend({
    renderTemplate: function(controller, model){
      if(model.name === "Squirtiepie") {
         this.render("pokedex_squirtle");
      } else {
        this.render('pokedex');
      }
    }
});

Odds and Ends

If your code is slow, because for example not tied to anything real and not to name names or anything you query exchange, you can create a template called ‘loading’. Or you an overload the loading behavior.

The same thing with handling errors.

Customizing the Base

You may notice that the Url ember gives you has a hash (#). You can pick the location strategy as one of the following three. If you don’t like the hash, you can use the ‘location’. If you’re running an app out of a page, you can use ‘none’. The ‘hash’ strategy is the default.

App.Router.reopen({
    location: 'history'
});

Transition Overrides

When you edit the Pokedex, you don’t want to update the data until the Pokemaster is finished with her entry and presses the submit button. But what if she clicks on a link to another page? We can intercept the transition and quit!

App.PokedexEditRoute = Ember.Route.extend({
    action: {
	    willTransition: function(transition){
			if(this.controller.get('formHasEdits') &&  confirm("Are you sure you want to throw away your changes?")){
				transition.abort();
			} else {
				return true;
			}
		}
	}
}	
});

We could also validate the input for correctness and prevent a transition if we found it necessary.

Redirecting

Let’s say you load a model and need to decide to redirect because any Pikachu needs to redirected to a list of rats. That jerk gets all the attention anyway.

App.PokedexRoute = Ember.Route.extend({
   afterModel: function(model, transition){
      if(model.get('type') ==='Pikachu'){
        this.transitionTo('rats');
      }
   }
});

You Are Here

The router is all about defining the transitions between your views. In all honesty, the next time I write a desktop application, I may have to write something like this. It’s super easy to manage and override just about any transition. It’s useful, flexible, and the heart and soul of your application.

2 Comments

Rise of the Tetrad

work safe

Ember has about six concepts that once you understand a bit will help you unlock the documentation and figure out how to develop your site. I have written before about not needing to fully understand ember before working on a project, but once you start getting your feet under you and need to branch out. My hope is that by the time you are done reading, you will have a roadmap to the Ember Guides and the Ember APIs.

Rise of the Tetrad

Ember uses naming conventions to tie together the four moving parts of an ember application; the Route Name, the Route, the Controller, and the Template.

The Route Name

When the route is defined like I did below, it is given a name. By default, the path mapped to the route is the same as the route. For example, the route below defines a route named “level1” and maps the url “/level1” to the route of the same name.

Route.map(function(){
 this.route("level1");
 ...
});

The Route

By default we are generating a route named App.Level1Route. If we want to change the behaviors of the route, which we will cover in more depth in a different post, we need to define our Route and provide it with some overrides. A common override is to provide some model data, like below

App.Level1Route = Ember.Route.extend({
 model: function(){
 return Ember.$.getJSON("http://mygame.com/api/level1.json");
 }
});

The Controller

Again by default, we have a controller called App.Level1Controller created for us. But if we want to override its behavior, we just make a new one with the correct name! For example, we can add an action to save the current state back to the server.

App.Level1Controller = Ember.Controller.extend({
 actions: {
 save: function(){
 Ember.$.postJSON("http://mygame.com/api/pause/", model);
 }
 }
});

The Template

We need to show screens for our level. This is done by showing a template. The default template is, of course, named ‘level1’. The template is written in handlebars.

<script type="text/x-handlebars" data-template-name="level1">
  <button {{action 'save'}}>save</button>
  <canvas name="game-level" width="500" height="400">
  </canvas>
</script>

Numbers are tricky. You said there were six, told us about four, but it looks more like only three.

Right. The Route, Controller, and Template are all based off of the Route’s Name. So, four for the price of three. The Other Fourth is the model. I’m just going to skip it for now. The three Ember apps I have written have not yet needed the complexity of an Ember Data model.

The other pieces you will come across are the Components, the views, and Router.

The Router

The router is the part that wires all the components together. We used it to set up our route mapping.

Views

Views let you add behavior to an html tag. You can add behaviors and event handlers to the tag.

Components

Components are an implementation of the W3C Custom Element Spec. They are far more functional than overriding an html tag, they are custom tags backed by a template and code that allow you to build custom controls. If you need a calendar control, start here. Need a tabbed page control? Check out Ember-Components.com for some examples.

Next Steps

Now that we know our way around the major concepts of the Ember framework, we can start to understand them one at a time. Maybe next time we will start with the routes.

No Comments

A quick note on EmberJs and ReactiveJS

work safe

I tweeted yesterday that I was having trouble with EmberJS and ReactiveJS integration.

Well, I figured it out! With some help from the fine ReactiveX folks.

The long story short is RxJs is smart about Ember, and there are consequences.

Yeah, it boiled down the the difference between mousedown and mouseDown. Did you catch the capitalization of the D? I never even thought to look…

So, what’s it look like in the end?

App.CanvasView = Ember.View.extend({
    tagName: 'canvas',
    didInsertElement: function(){
        this.subscription =
          Rx.Observable.fromEvent(this, 'mouseDown')
                       .subscribe(function(e){
                           console.log("x="+e.offsetX + " y="+e.offsetY);
                        });
    },
    willDestroyElement: function(){
        this.subscription.dispose();
    }
});

And now to start the magic of using RxJs.

COMPOSE ALL THE EVENTS

update

I still think there is one thing left. I’ll try and test it out and add a Pr in a bit, but it seems to be important for my situation.

If I look at line 124 and 125 of fromEvent.js, I see an unused argument. I passed that through to the ember listener calls.

    if (ember) {
      return fromEventPattern(
        function (h) { Ember.addListener(element, eventName, h); },
        function (h) { Ember.removeListener(element, eventName, h); },
        selector);
    }  

update 2

I submitted a pull request. I hope I’m right about this, or I’ll be beet-red.

1 Comment

The Ember Path

work safe

There was a recent blog post by Rob Conery about learning EmberJs by just flinging yourself at it. That was something that kind of resonated with me. I’ve been trying to understand EmberJS for a while, but I wasn’t getting it. I wasn’t feeling confident enough to actually start a project and get it right. You can never learn without actually doing and Rob reminded me of this.

So I tried it. I flung myself in to the void of two projects.

Define the Problems

I chose two projects.

Project one is a rewrite of a comic strip site I wrote using JavaScript and HTML in about 2002. Considering I’ve not written anything for the comic itself in almost 10 years, It’s not important, but it is a healthy amount of data to display.

Project two is much smaller and uses an ASP.NET web service to wrap the Microsoft Exchange APIs. It simply asks what is currently happening with the different conference rooms at SEP.

Layout the path

I wrote both of the sites using mostly the same path. I wrote them at about the same time, so this is only logical. I accepted as fact that I would be rewriting functionality and moving code between responsibilities as I got more familiar with Ember’s various pieces.

Wonder Twins Routes and Templates (via URLs)

With routes and templates, a large amount of the work can be done. I can lay out a number of screens and how they transition. I can specify the Urls for each page.

According to Tom and Yehuda, the Ember core team, The heart of a web application is the URL and the Router is how Ember interfaces with them.

Some Urls map to a route literally to a template. The Comic’s about page has no logic or data. It’s just text. Navigating to the /about Url displays the “about” template.

App.Routes.map(function(){
    this.route("about", {path: "/about"});
    ...
});

The template uses Handlebars wrapped in a script tag. The element ID is used to map the name to the route.

<script type="text/x-handlebars" id="about">
  Some About Text
</script>

Wait, there’s data?

If we want to add a touch of data to a page, we can change the route. Let’s look at the cast page. We don’t alter how we define the route mapping.

    this.route("cast", {path: "/cast"});

Then we can override the default router by adding a touch of code. Here, we have a static data structure defining our cast members. I then override the default router for the Cast route and define a model attribute.

var cast = [ {name: "Jay Barnes",
              occupation: "Coffee Shop Manager",
              head: "http://space-for-rent.net/cast-jay-head.png"},
              ... ];
 App.CastRoute = Ember.Route.extend({
   model: function () { return cast; }
});

And then we can consume it in the template.

<script type="text/x-handlebars" id="cast">
    {{#each item in model}}
        <div><div class="head"><img {{bind-attr src=item.head}}></div>
             <div class="cast_name">{{item.name}}</div>
        </div>
    {{/each}}
</script>

That’s Fine for Merlin, but I Have Dynamic Data

I had this same problem with my Exchange service. I needed to hit my web service for some data.

App.IndexRoute = Ember.Route.extend({
   model: function () {
        return Ember.$.getJSON("https://myendpoint/api/rooms");
    };
});

But in all honestly, that takes forEVER because exchange is SLOW. So I just define a ‘loading’ template and ember makes it all better because of it’s internal use of promises to handle the async call to my server. Ember automatically shows this while waiting for my data, then shows the page with actual data.

<script type="text/x-handlebars" id="loading">
<h2>Please wait, Exchange is <em>SLOW</em></h2>
<center><img src="/Content/spinner.gif" alt="please wait"/></center>
</script>

Let’s Be Specific

If I have a url that needs a data portion, like asking for details of a specific room, I can define that in the route map and access the parameters in my model function.

App.router.map(function(){
    this.route("room", {path: "room/:room_name"});
    ...
});
App.RoomRoute = Ember.Router.extend({
    model : function(params){
        return Ember.$.getJSON("https://myendpoint/api/room/" + params.room_name);
    }
});

Is That It?

Nope. That’s not it. Granted, I’ve gotten a bunch of functionality out of those two parts of Ember; Routes and Templates. I’m now picked up on my bootstraps and I can pick my next battles based on my needs. I’ll sum up the learning path I have left for me using these two apps, and more importantly why I need to learn these things.

  • Components – I’ve used these to reduce the duplication in my templates. They can be passed data so I use them to define how I render an Exchange meeting or for the navigation bar for my comics site.
  • Controllers – I’m currently using them to handle some button triggered actions on the Exchange site. they can also be used to compute properties based on the model values or add two way binging to things like text input fields.
  • Models – I don’t know if my current projects have a need for Ember.Data, but I’m going to use it to represent the comic site’s data using fixtures. This will, hopefully, give me the experience to start with a larger data driven application.
  • Testing – When learning a framework, NEVER worry about automated programatic testing until you have a decent understanding of how things are working. THEN add tests. THEN start considering adding new features test-first. Ember has a fairly decent testing story that I want to start exploring. The more work I do in JavaScript, the more I’m going to need to test. While not needed for sites I have here, thin wrappers around data or APIs, I will be writing some sites with far more complex logic.

There is more EmberJs to learn, but I don’t need it yet. But when I do, I’m confident the documentation will be clear. And so far, the code has bee easy to use. This is the first time I’ve enjoyed web development in several years. I think I’m ready to take back the web.

3 Comments

Manipulating Data With F#

work safe

I’ve been working with our corporate website for the past few days. I’m trying to get analytics for some A/B split testing. As a part of that, I got to play with some really neat FSharp features. Type providers have allowed me to use a snippet of json to build a type hierarchy without doing the heavy lifting myself. And recursive sequences allow me to consume a paginated webservice as though it was a single data set.

Bringing My Customers Together

The data I’m currently interested in is, as far as I can discover, only available by asking each Hubspot contact for their list of form submissions so I can count the number of submissions to each form. But I can only ask for 100 customers at a time. The response will tell me if I can fetch more and the vid-offset I can use to get to the right spot in the pagination.

let rec Contacts offset =
   seq { let data = HubspotContacts.Parse(Http.RequestString("http://api.hubapi.com/contacts/v1/lists/all/contacts/all",
                                                             // apikey not included ;) 
                                                             query=["hapikey", apikey; "count", "100"; "vidOffset", offset],
                                                             headers=[Accept HttpContentTypes.Json]))
         yield! data.Contacts
         if data.HasMore then
            yield! Contacts (data.VidOffset.ToString())}

What’s happening? The seq structure creates a seq computation (think IEnumerable) and allows us to write code that generates a seq. The yield! keywork takes a collection and doles each element out in turn. If we find the collection has more, we yield the results of a call with the offset. This means our first call will use the offset “”.

Types are your friends

FSharp type providers are wonderful. They can, at compile time, parse out some code and generate a type tree. In particular, using the FSharp.Data library I can give it a snippet of JSON and get back a strongly typed parser.

// I'm leaving a lot out of the json snippet.
// You don't need to see the details of my customer data
let [] ExampleResponse = """ { "contacts" : [ {"form-submissions": [{ ... }]}
                                                       { ...} ],
                                        "has-more" : true,
                                        "vid-offset": 1234 } """
type HubspotContacts = JsonProvider

Reductio ad absurdum

The only thing left for me to do is take this and map / reduce my way to a collection of form-ids and counts.

Contacts ""
// Transform from a seq of contacts to a seq of form submissions
|> Seq.map (fun c -> c.FormSubmissions)
|> Seq.concat
// Extract the form id
|> Seq.map (fun fs -> fs.FormId)
// Collect the counts of unique formIds
|> Seq.fold (fun s v -> Map.tryFind v s with
                        | Some(count) -> Map.add v (count+1) s
                        | None -> Map.add v 1 s) Map.empty
// Report the FormId and Count
|> Map.iter (fun formId count -> printfn "%A %i" formId count)
No Comments

Are You Doing Anything Real With It?

work safe

I’ve been playing with Clojure recently and loving it. However, most of the stuff I’ve been doing with it recently has been Exercism.io, and not actual application code. To rectify that, I pounded out (slammed my head against a few problems) a simple application (sure, now that it’s written) to get my secret feed from NSScreencast and download the videos.

Two Great Houses

Reading a feed and downloading movie files is very simple on the surface. There are two things complicating the issue, but they are two very important parts. First, feeds are stored as RSS, and XML is best treated as a tree data structure, not as text to Regex over. Second, NSScreencast uses a CDN, but that means the video URLs return a redirect.

The Forest of Xml

The Clojure community has taken a shining to using Zippers for traversing large trees of data. Xml is just a tree of data. Once you parse the damn thing. Fortunately, Clojure has a few libraries for handling this; Clojure.xml, Clojure.zip, and Clojure.data.zip.xml.

Our Imports

            [clj-http.xml :as xml]
            
            

Building a zipper

Eventually, I need to have an xml zipper from the Clojure.zip library. For that to happen, I need to give Clojure.xml’s parse an input-stream based on the feed’s url.

(defn make-feed [uri]
  (-> uri
      input-stream
      xml/parse
      zip/xml-zip))

The feed I process is build the following:

(def feed (make-feed "https://nsscreencast.com/private_feed/«secret_key»?video_format=mp4"))

Yeah, I’m not giving you my secret key. Go subscribe! They are good!

Traversing a zipper

Here’s an example of the xml we are navigating. This is taken from NSScreencast’s free video feed and only has the first two entities.

<?xml version="1.0" encoding="UTF-8"?>
<feed xml:lang="en-US" xmlns="http://www.w3.org/2005/Atom">
  <id>tag:nsscreencast.com,2005:/feed</id>
  <link rel="alternate" type="text/html" href="http://nsscreencast.com"/>
  <link rel="self" type="application/atom+xml" href="http://nsscreencast.com/feed"/>
  <title>NSScreencast (free videos)</title>
  <updated>2014-01-29T00:00:13Z</updated>
  <entry>
    <id>tag:nsscreencast.com,2005:Episode/93</id>
    <published>2013-09-26T10:02:32Z</published>
    <updated>2013-09-26T10:02:32Z</updated>
    <link rel="alternate" type="text/html" href="http://nsscreencast.com/episodes/87-xcode-5-autolayout-improvements"/>
    <title>#87 - Xcode 5 Autolayout Improvements</title>
    <content type="text">This week we have another free bonus video on the improvements that Xcode 5 brings to Autolayout.  As something that has been quite obnoxious to work with in the past, many people dismissed auto layout when it was introduced to iOS 6.  With these improvements it is much more friendly and dare I say... usable?</content>
    <link rel="enclosure" xml:lang="en-US" title="Xcode 5 Autolayout Improvements" type="video/mp4" hreflang="en-US" href="http://nsscreencast.com/episodes/87-xcode-5-autolayout-improvements.m4v"/>
    <author>
      <name>Ben Scheirman</name>
    </author>
  </entry>
  <entry>
    <id>tag:nsscreencast.com,2005:Episode/91</id>
    <published>2013-09-19T10:01:59Z</published>
    <updated>2013-09-19T10:01:59Z</updated>
    <link rel="alternate" type="text/html" href="http://nsscreencast.com/episodes/85-hello-ios-7"/>
    <title>#85 - Hello, iOS 7</title>
    <content type="text">To celebrate the launch of iOS 7, here is a bonus free screencast covering a few of the concepts in iOS 7 such as the status bar behavior, tint color, and navigation bar transitions.  We'll also take a look at Xcode 5 with a couple of the new features, including the integrated test runner.</content>
    <link rel="enclosure" xml:lang="en-US" title="Hello, iOS 7" type="video/mp4" hreflang="en-US" href="http://nsscreencast.com/episodes/85-hello-ios-7.m4v"/>
    <author>
      <name>Ben Scheirman</name>
    </author>
  </entry>
</feed>

Once I have the zipper for the feed, I need to traverse the feed’s tree. What I want eventually is a list of file names and urls for that file. the file name will come from the feed, but I need to find the url from traversing all entry tags, their child link tags, but only the link tags with a rel="enclosure". But honestly, I only care about the href attribute.

(zip-xml/xml-> feed
               :entry
               :link
               [(zip-xml/attr= :rel "enclosure")]
               (zip-xml/attr :href))

Once we have this, we need to find the file name. Anything between the last / and the only ? is the file name. The way clojure returns Regex matches, however, is a match followed by the group. So, if we spread out the regex to match the whole Uri, we have the two peices we need for further processing. Well, we need to map over first before doing anything because of the way re-seq returns data.

(defn link-file-pairs []
  (->> «feedTraversal»
       (map #(re-seq #".*/(.*)\?.*" %))
       (map first)))

Looking in the wrong place

While Clojure’s input-stream is capable of handling a uri, it has some trickyness with SSL urls and more importantly, the redirect to the CDN is handed back as a document I have to parse by hand. Fortunately, clj-http.client handles all this for us. We’ll require the lib as client. To copy a file from a uri to the disk, we use our copy-file routine.

(defn copy-file [[uri file]]
  (with-open [w (output-stream (fullname file))]
             (.write w (:body (client/get uri {:as :byte-array})))))

the function client/get returns a map of data, and we only care about the :body. We ask for the body to be returned as a byte-array with {:as :byte-array}. with-open is used to close the output-stream once we’re done. And since we are using this to map over a vector of uri and file, we will destructure the vector in the argument list.

Fleshing out the rest

The main algorithm of the downloader is to get a list of link-file-pairs, filter the videos I’ve already downloaded, and copy all the files. I tag a doall to ensure any lazy seqs actually execute.

(defn -main []
   (->> (link-file-pairs)
        (filter not-downloaded)
        (map copy-file)
        (doall)))

Determining if a file is downloaded is fairly simple. Figure out the full path of the expected file, make it a file object, and ask if it exists. But I want the inverse of exists, so we’ll tag a not on the end.

(defn not-downloaded [[uri file]]
  (-> file
      fullname
      as-file
      .exists
      not))

What’s in a name?

I’ve already downloaded some files, so I wanted to make sure they had the same naming scheme. Also, the fullname function knows where on my machine the files are supposed to go. The details arent interesting as the prefix makes life more difficult than it needs to be.

(defn zero-pad [n number-string]
  (->> (concat (reverse number-string) (repeat \0))
       (take n)
       reverse
       (apply str)))

(defn prefix [file-name]
  (clojure.string/replace file-name #"^\d+" #(zero-pad 3 %)))

(defn fullname [file]
  (str "C:\\Users\\bjball\\Videos\\NSScreencasts\\ns" (prefix file)))

The Details

The code is available on Github. As a part of a literate programming exercise, the structure is all laid out below.

(ns nsscreencast-fetcher.core
  (:use ])
        (:require [clj-http.client :as client]
                  «xmlDecls»))
«makeFeed»
«feed»
«linkFilePairs»
«fullName»
«copyFile»
«notDownloaded»
«main»
No Comments

Principles In Practice – It Will Go Wrong

work safe

It’s Murphy’s Law. If it can go wrong, it will. What’s your application’s opinion on errors? If we take a stance on errors, these principles will tell us exactly how the application will behave. Shared understanding means less miscommunication. Here are some of the things I’m talking about.

Operators have the accurate information – your data is already wrong

If you’ve ever heard the following conversation, you know data entry will always be out of sync with the real world. “I know the database is telling me that bus 7 has transmission 8 installed, but I’m looking at transmission 3. Don’t tell me I can’t uninstall transmission 3, because I just did.”

What does that mean for implementation? Favor “soft validations,” warning that errors are present, but allow overrides. If I have a misallocated part in the database, I need to uninstall it from where the database “thinks” it is and mark that uninstall as something needing follow up by people in the field.

It’s all fun and games until someone dies. – The data must have integrity

Particularly in the medical field, the configuration of a medical device should be exactly what you intend with no mistakes or corruptions. Corrupt data can kill. Also, data must meet certain validations to be used, but can fail during editing.

Never write over the data file. Always write to a shadow file, test the integrity and in one atomic action, move the new data file over the old one. If there is ANY failure of integrity, the old data file is a safe fallback position.

In addition, the internal data structures might be “persistent” allowing undoing to a known good state if any operation fails an integrity check. Some transient operations might lead to bad values, like changing a medical dosing schedule might temporarily exceed safe dosages within a time frame, but you should not exit the schedule edit until the schedule validates.

If it crashes, I can use the browser, right? – The application must stay running

This is useful for kiosk or embedded applications. However some long running applications also need to handle failure.

Use multiple executables. The less an application does, the less likely it is to crash. In addition, a supervisor application can restart parts of the application that fail. Operations with risky components like using third party drivers or that can enter bad states because of bad data can run in their own sandbox and die if they ever enter a corrupt state.

There is real money involved

Real money requires an audit log. In addition, duplicate submissions are a killer and should be identified ASAP to avoid duplicate charges.

Operations, like purchases and fulfillments, are seen and recorded as first class data. The engineering team may lean on CQRS for these operations. Operations may be given an identifier early in the process. For example your cart my always have a “next order Id” so commands may be made idempotent preventing duplicate submissions.

Take a bit and think about how your application can fail and what can go wrong. Pick your battles and identify the ones that are the most likely and the ones that cause the most havoc. How are you going to handle them?

No Comments

Principles of Architecture

work safe

There is no correct software architecture. But there are an unlimited number of ways to incorrectly architect your software. The best we can do is state some principles and use those to help narrow the solution space of all possible architectures. This will help us identify a more correct architecture.

Architecture – a quick definition

Building Architecture is a combination of two things, making the building suitable for purpose and making the overall structure pleasing to the eye.

Making a building suitable for purpose is mostly dividing the space in to boundaries. Like rooms are often grouped near one another. Some structural considerations may help this grouping, such as bathrooms being above the kitchen to limit the number of wet walls. Some rooms may have internal subdivisions like nooks and archways that don’t limit the flow or utility of a space but visually define these regions. This defines the utility of a building.

Aesthetic considerations help make the building feel as it belongs. This means it fits in the surrounding landscape as well as feels balanced and consistent with itself. The balance and feel contribute to the habitability of a space.

Utility and Habitability in Software

Software architecture also controls the utility and habitability of a program.

Software Architectural utility is not the UX, but the software’s ability to perform its tasks quickly enough, reliably enough, and a host of other so-called -ilities. This is done by determining if you need extra caching for speed, separation of the active data set from the reporting data set, splitting the application in to multiple programs to better handle failure, and interfacing with other systems. And that’s just a start!

Software Architectural habitability is about the ease of development. Simple changes should be simple. Changes that depend on multiple business concepts interacting should only be as complex as those interactions. This is done by being deliberate about what code interacts with other code, by drawing boundaries between software concepts that reinforce boundaries between business concepts.

These forces can be in tension. Adding robustness to an application can make changes harder. Making it more flexible can degrade performance. Focusing solely on making easily changeable software if fine for small scale software. Wringing every last ounce of throughput is perfect for less complex software. Finding the balance is the art of Software Architecture.

Principles for Improving Communication

If you need someone called “Architect” on your project, the odds are you have a team larger than the two-pizza rule. That means that several decisions a day are going to need the architect to be there, and some times she can’t be present in person. Be stating not only the design goals but what purpose they serve, the architect empowers the team to make correct choices when they find themselves on their own.

It isn’t sufficient to say the software must be robust. But if we say minimize the impact of failure on non-related operations and enable a manual retry of failed transactions we are clearly stating what we mean by robust. When dealing with a design decision I can now ask myself a few questions to guide my implementation.

  1. Does failure impact other users?
  2. Do I have enough information to recreate this request if it fails to complete?
  3. When I fail, can I undo the partial operation?
  4. Are my operations Idempotent?

By stating the outcomes desired instead of the means we use to get there, the architect prevents the same problem being addressed by nothing more than a bunch of null checks and try/catch blocks.

It’s Only Business

Theses principles should come from the business case for the software. How many users are you supporting at once? Are they _occasionally connected_ or always connected? What is the impact of failure on a sale? The impact of failure on trust in the system? How about the impact of failure in the lives of thousands of children? Will someone die if your software falls down or corrupts data?

What are the reasons for your architecture choices? Have your design choices helped or hindered those goals? Is there tension between the design impacts of your business goals? Do you have some design changes to go deal with?

No Comments
« Older Posts