Planet Enterprise 2.0

Last week I started looking into ways to Internet feeds into TV feeds. Although I did come up with a way to turn a data feed into a video file, that wound up being overkill. It turns out that the local station is willing to broadcast the signal from a computer display. To create that signal, several folks suggested using PowerPoint, but I found that its scrolling credits feature doesn’t accommodate really long lists of credits. So I decided to try XAML, the application markup language that works with Silverlight and the Windows Presentation Foundation (WPF), in concert with IronPython.

The plan was as follows. A long-running IronPython script periodically fetches the feed from a web service, interpolates the text into a XAML template that animates the crawl, and displays the XAML in a fullscreen white-on-black WPF window.

Here’s the XAML template:

<Grid xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" Width="%s" Height="%s"> <TextBlock xml:space="preserve" FontSize="%s" Margin="%s,%s,0,0" FontFamily="Arial"> <TextBlock.RenderTransform> <TranslateTransform x:Name="translate" /> </TextBlock.RenderTransform> <TextBlock.Triggers> <EventTrigger RoutedEvent="FrameworkElement.Loaded"> <BeginStoryboard> <Storyboard RepeatBehavior="Forever"> <DoubleAnimation From="%s" To="-%s" Storyboard.TargetName="translate" Storyboard.TargetProperty="Y" Duration="00:%s:%s" /> </Storyboard> </BeginStoryboard> </EventTrigger> </TextBlock.Triggers> <Run> <![CDATA[ %s ]]> </Run> </TextBlock> </Grid>

Some of the values depend on the number of items in the feed, so the script interpolates those values into the template. Then it formats the feed and plugs the formatted text into the template’s CDATA section. The formatted text looks like this:

EVENTS FOR MON JUN 30 2008 FROM THE ELMCITY.INFO CALENDAR 06:00 AM: lap swim (ymca) 07:00 AM: AA: On Awakening Group (eventful: Keene Unitarian Universalist Church)

After generating the XAML, the IronPython script fires up an Application object, creates a window, loads in the XAML to start the crawl, and sets a timer to refresh the XAML.

I ran into a snag when I tried to set that timer, though. There are a few different timers you might imagine using in this context, including Python’s own timer object and various timers available in the .NET Framework. All but one of these, however, will complain about invalid cross-thread access when you try to update the application’s user interface from a timer event handler.

The right timer to use, it turns out, is .NET’s System.Windows.Threading.DispatcherTimer. But when I tried it, I ran into another snag. In C#, you create a WPF-friendly timer like so:

DispatcherTimer timer = new DispatcherTimer(); timer.Tick += new EventHandler(event_handler);

event_handler is a method, but EventHandler returns a delegate that encapsulates that method. I couldn’t find a straightforward way to create a delegate, and do that encapsulation, in IronPython.

If you know how, I’d love to hear about it. Then again, it really doesn’t matter. Logically this program has two loosely-coupled parts. The engine part reads the feed from a web service and formats it as XAML. It can be a Python script that runs on a scheduled basis to fetch and format the feed.

The user interface part loads, displays, and then periodically refreshes the XAML. It can be a little C# program that runs forever, displays the animation, and refreshes the data, like so:

using System; using System.Windows; using System.Windows.Markup; using System.Windows.Input; using System.Windows.Media; using System.IO; using System.Windows.Threading; namespace CalendarCrawl { public class CalendarCrawler { static private Application app = new Application(); static private StreamReader getXaml() { StreamReader sr = new StreamReader("WPF.xaml"); return sr; } [STAThread] static public void Main(string[] args) { Window win = new Window(); win.WindowStyle = WindowStyle.None; // go fullscreen win.WindowState = WindowState.Maximized; // go fullscreen win.Topmost = true; // go fullscreen win.Cursor = Cursors.None; // go fullscreen win.Content = XamlReader.Load(getXaml().BaseStream); win.Background = Brushes.Black; win.Foreground = Brushes.White; DispatcherTimer timer = new DispatcherTimer(); timer.Interval = new TimeSpan(0, 1, 0); // every hour timer.Tick += new EventHandler(eventHandler); // wire up handler timer.Start(); app.Run(win); } static private void eventHandler(Object sender, EventArgs args) { app.Windows[0].Content = XamlReader.Load((getXaml().BaseStream)); } } }

It was odd how reluctantly I came to this division of labor. Evidently I still need to remind myself that in a world of loosely-coupled applications and services, when you need to get something done, There Is More Than One Way To Do It.

Here’s another way. If the engine doesn’t have to talk to the .NET Framework’s WPF machinery, there’s no need to use IronPython. Any flavor of Python makes a handy tool for talking to RESTful web services, wrangling text, and interacting with the file system.

Here’s yet another way: A Silverlight version of the user interface. It’s nice to know that option is available. However, I’m leaning toward the C# version. The target machine is Vista, it already has .NET and WPF, why use a long-running browser instance just to host this tiny little thing?

One final point is worth mentioning. XAML is really just another source language for the .NET runtime and framework, like C# and IronPython and others. You can, for example, create an application window by writing a Window tag in XAML markup, and specifying parameters as attributes. Or you can do it by invoking System.Windows.Window from IronPython or C# or another .NET language, and specifying parameters in code. The boundary between markup and code is very fluid, and you can draw the line for reasons of convenience, maintainability, and taste. It’s a very flexible system, and it becomes even more flexible when you can use a dynamic language like Python to generate the XAML, the code, or both.

For many of us, the podcasting revolution has opened up the audio channel as a new option for receiving information that we might otherwise read. But for the sight-impaired, like Susan Gerhart, who joins me for this week’s ITConversations show, the audio channel isn’t optional. Her myopic retinal degeneration has forced her to shift almost entirely into audio mode in order to read, and to work on the computer.

As a lifelong technologist, Susan is a capable user and evaluator of software and computational devices. When she entered the world of assistive technologies — including the NVDA screen reader, the LevelStar Icon, the Kurzweil NFB Reader — she decided to share her experiences on a blog. In our interview she summarizes what she’s learned so far about using these technologies to adapt to her changing vision.

As per the comments on yesterday’s item about creating a video crawl for local TV, it turns out there’s no need to produce a video file. Instead it’ll be OK to use a computer display directly. The computer could be running, for example, a PowerPoint slideshow in a loop.

Here’s the apparently standard recommendation for making scrolling credits in PowerPoint. It was written for earlier versions, but seems applicable also to the current 2007 version:

Create movie-style crawling credits in PowerPoint presentations

1. In a PowerPoint presentation, create a new slide for credits or any other list that you want to scroll from bottom to top.

2. Type your credits or other text. Don’t worry about text running off the bottom of the slide. In fact, it should run off the bottom if you are going to have enough text to make a crawl effect work well.

3. Right-click the text, and on the shortcut menu, click Custom Animation.

4. Select the text that you want to scroll. In the Custom Animation task pane, click Add Effect. Point to Entrance, and click Credits.

5. Click Play to see how the effect will look on-screen.

6. Move the text block completely off the top of the slide. When you play your presentation, the text will crawl or scroll from the bottom of the screen and disappear off the top.

But it doesn’t look like you can get more than three screenfuls of data into the crawl. For example, I made a textbox with 200 lines of text numbered accordingly. Then I animated it using several varations on this technique.

First I put the top of the textbox at the top of the slide, like so:

The effect: Line 0 crawls into view from the bottom of an empty slide, and the crawl ends with line 25 at the top and line 50 at the bottom.

Next I put the top of the textbox at line 25, like so:

The effect: Line 0 appears at the top of the slide, the crawl ends with line 50 at the top and line 75 and the bottom.

Is there a way to include more than three screenfuls of data in the crawl? If not, it looks like it’d be necessary to create a series of slides, each with two screenfuls of data. The first slide would need to have its first line of data at its top. But the second and following slides would need to have their middle lines of data at their tops. Gnarly.

I’m sure that could be done, but why bother? Absent a requirement to produce a video file, there a zillion ways to make text crawl up a computer screen. This might be a good opportunity to explore the combination of IronPython and XAML.

I’ve cobbled together a way to turn an Internet data feed into a video crawl that can run on my local public access cable TV channel. Before explaining how, I need to explain why. Here’s the short answer: As much as I want everyone to use the Internet for all it’s worth, most people don’t yet.

A couple of years ago, I was campaigning in my community to open up the parent portal into PowerSchool, a student information system that was being used internally by teachers and administrators but wasn’t available to parents via the Internet. At one point I made a screencast that addressed the perceived risks, and showed the compelling benefits, of opening up the portal. The screencast was published on the Internet, available to the whole world, and the whole world includes Keene NH, so that ought to be a good way to bring my message to the community, right?

Wrong. Nobody watched it.

A while later, it hit me. There still aren’t many folks here who are inclined to receive a message like that from InfoWorld.com, or from YouTube, or from any other Internet destination I might use. But there are significant numbers who tune into the local public access station. Why not show the screencast there?

So I dubbed it onto a MiniDV tape, took it down to the station, and gave it to the executive producer.

Him: What’s this?

Me: A demo and discussion of the PowerSchool software. Will you run it?

Him: Sure.

And lo, a couple of weeks later, I heard from the assistant superintendant of schools. He thanked me for applying the external pressure that they’d been needing in order to break through an internal logjam, and he invited me into the beta program. Now, two years later, it’s fully deployed and making a big difference.

Meanwhile, I’ve been working on a community information project that’s all about feeds and syndication. But slow learner that I am, I continue to invite people to use Internet feeds and Internet syndication. And people continue to mostly decline the invitation.

For example, I’ve been working on calendar syndication. The syndication flows two ways. First, inward. The service pulls events from various local websites, and I’m working with the proprietors of those sites to clarify why and and to publish true calendar feeds.

Second, it syndicates outward. With a JavaScript call, you can include the day’s events in another website, like CitizenKeene or Cheshire TV.

But this is all still just Internet stuff. And as we’ve seen, the community doesn’t (yet) tune into the Internet for local information. It does tune into public access cable TV.

So why can’t Internet data feeds show up there?

Well, of course, they can. Here’s a prototype video crawl (the link goes to an animated gif, just for convenience) made from yesterday’s combined calendar. We’ll need to work out the details of format and workflow, but I think it’ll work. And it seems like a great way to connect two worlds.

Calendars are just part of the story. Consider, for example, the public library’s RSS feeds announcing new books and DVDs. I’m one of probably a handful of subscribers to those feeds. Now imagine that the feeds showed up as a video crawl on TV. I bet a lot more folks would find out about new books and DVDs. And maybe, just maybe, the reception of that feed via TV would lead to discovery and use of the more convenient and powerful Internet feed.

We’ll see. Meanwhile, below the fold, I describe the method I’ve come up with to do this. The paint isn’t dry, and I’ll be very interested in comments and suggestions.

… the fold …

Our public access TV station, as may be typical (though I dunno), is a mostly Windows-based operation. As is surely typical, there’s little money to spend, either on people to produce these feeds interactively or on software to produce them automatically. So the requirements seem to be:

1. Windows-based
2. Cheap or free
3. Fully automatable

My first idea was to leverage SMIL. I knew it would be easy and free to transform a feed into markup that can be played by Real, or QuickTime, or Windows Media. And I hoped it would also be easy and free to render that markup into a video format. But there I ran aground. If there’s a free, or at least cheap, SMIL renderer that can be scheduled to run automatically, I’d like to know about it, because that’d probably be the ideal solution. But I haven’t found one.

The next idea was to produce the animation frame by frame. And that’s what I’m actually doing for now. It sounded a lot harder than it turned out to be. After installing the Python Imaging Library, it was possible to write this very concise frame generator:

import Image, ImageFont, ImageDraw s = """ EVENTS FOR WEDS JUNE 30 FROM ELMCITY.INFO (HTTP://ELMCITY.INFO/EVENTS) 06:00 AM lap swim (ymca) 07:00 AM Cheshire Walkers: Indoor Walking Program (eventful: Keene Recreation Center) ... Trainers Academy - Level II (eventful: Monadnock Humane Society) TOR 7pm (swamp bats) """ lines = s.split('\n') def frame(index,top): image = Image.new('RGB',(720,480),(0,0,0)) draw = ImageDraw.Draw(image) font = ImageFont.truetype("arial.ttf", 18) for line in lines: draw.text((10, top), line, (255,255,255), font=font) top += 25 image.save('cal%0.3d.gif' % index) top = 450 for index in range(len(lines)*8): print index,top frame(index,top) top -= 4

This yields a sequence like cal000.gif…calnnn.gif.

I wasn’t sure how to make a video directly from that sequence, but I knew that ImageMagick could turn it into an animated GIF, like so:

convert -adjoin cal???.gif animation.gif

So I did that, and went looking for ways to convert that into a video format. ffmpeg will do it, but the results weren’t pretty, and ffmpeg can be a dicey thing to ask people to install. QuickTime, I found, did a better job. You’d need QuickTime Pro for Windows, which isn’t free, but $30 won’t break the bank.

Now the question became: How to automate the QuickTime conversion? I installed the QuickTime SDK, went looking for examples, and found just what the doctor ordered. Thanks, Luc-Eric!

Luc-Eric’s JavaScript solution, which runs on the Windows command line courtesy of the Windows Script Host, turned out to provide a double benefit. In addition to showing how to automate the conversion of a batch of GIFs to an AVI, it showed me that there was, in fact, no need to produce an intermediate animated GIF. You can just point QuickTime at the sequence in the same way that you can point ImageMagick at the sequence. I hadn’t known that! So ImageMagick dropped out of the toolchain, and there was one less component to require the station to install.

So that’s where things stand. I’m pretty sure there’s a better way to meet the requirements, and I’ll be delighted to discover it. But maybe there isn’t, in which case it looks like this will work.

Either way, it’s the end result that will — or maybe won’t — matter. We’ll do the experiment, and we’ll find out.

On this week’s ITConversations show I finally got to meet Jean-Claude Bradley, the Drexel chemistry professor who coined the phrase open notebook science and who champions the principles behind it.

There were a couple of surprises for me. First, I was intrigued to learn about Jean-Claude’s vision for mechanized research. I’ve always thought of open notebook science as a way to speed up the iterative cycle of research and publication, and to engage more human minds in collaboration. Of course Jean-Claude thinks so too. But he also thinks that when data are published in accessible formats, and exposed to computational processes running in the cloud, we’ll be able to automate certain aspects of research.

It reminds me of George Hripcsak’s effort to mechanize the interpretation of electronic health records. In general, we’re collecting way more data than the collectors can analyze. Crowdsourcing is one solution to this problem. Mechanization is another. We’ll need both.

The other surprise was hearing about Drexel’s fairly aggressive use of Second Life. I’ve been an amused skeptic on that front, but Jean-Claude’s passionate advocacy requires me to rethink that stance.

What didn’t surprise me, but might well surprise tuition-paying parents of Drexel students, was Jean-Claude’s attitude toward the classroom. He mostly doesn’t see a need for it. The content delivery aspect of education, he feels, is best handled in other ways, including screencasts and podcasts as well as traditional texts. There can, and should, be a range of sources, to accommodate the differing inclinations of learners. And teachers need to be competent producers and orchestrators of those sources. But for Jean-Claude, the best way to engage directly with students is to meet with individuals, not with whole classes.

Now admittedly, a chemistry class doesn’t invite and thrive on group discussion in the same way that, for example, a literature class does. And yet Jean-Claude says that a literature class was one of the models for his use of Second Life. When group interaction is central to the educational experience, he thinks that virtual environments — though he doesn’t require their use — may outperform real ones.

I remain skeptical on that point, but I’m always open-minded, so I hope Jean-Claude will take me up on my offer to visit one of his virtual environments and document the interactions that happen there.

My guests for this week’s Perspectives are Microsoft researcher Curtis Wong and Harvard-Smithsonian science educator Roy Gould. At Ted 2008 they jointly delivered the first preview of the WorldWide Telescope, an elegant and powerful application for exploring the sky and weaving narratives about it. In this extended interview, you can hear (or read) the whole story behind the WWT.

I’d known that the WWT was based on Jim Gray’s work, and also that it was dedicated to him. I’d also heard several of the talks he’d given about SkyServer, SkyQuery, and the SQL and XML web services technologies powering those projects.

What I hadn’t fully grasped, until I began preparing for the interview with Curtis and Roy, was Jim Gray’s larger vision for that work. In 2002, with Alex Szalay of Johns Hopkins, he published a paper entitled The World-Wide Telescope: An Archetype for Online Science. Here’s the abstract:

Most scientific data will never be directly examined by scientists; rather it will be put into online databases where it will be analyzed and summarized by computer programs. Scientists increasingly see their instruments through online scientific archives and analysis tools, rather than examining the raw data. Today this analysis is primarily driven by scientists asking queries, but scientific archives are becoming active databases that self-organize and recognize interesting and anomalous facts as data arrives.

Although the WWT isn’t an instrument for professional scientists, Roy Gould thinks it will be used by citizen scientists to collaboratively search the fast-growing corpus of sky imagery. That is, of course, a poignant echo of the collaborative search for Jim Gray when his sailboat went missing.

But for Curtis Wong and Roy Gould, who grew up in Los Angeles and New York, respectively, where neither had access to the dark night sky, the WWT is first and foremost a way to reacquaint our society with the night sky, and to teach us about the universe.

Roy Gould says that when his team surveyed high school students around the country, they found that a majority believe that stars reside within the orbit of Pluto. They also believe that galaxies are closer than stars, because “stars are just point sources, no matter what the magnification, so they must be very far away, whereas galaxies, whatever they are, look big, so they must be closer.”

To fulfill its educational mission the WWT delivers seamless navigation of the sky, contextualized in a variety of ways. Objects are described onscreen, and linked to sources on the web. When you find your way to a stellar neighborhood, thumbnails of the objects in that neighborhood invite you to explore images from a variety of catalogs: the Sloan Digital Sky Survey, Hubble, Chandra.

What’s more, the imagery is correlated so you can see the same object in any of the wavelengths of light used to observe it. If you look at the Milky Way in the standard view, and then switch to infrared, a band of incandescent whiteness emerges from the cloud of stars.

You can use the WWT to explore the sky randomly, but most people will enjoy taking one of the guided tours. Curtis Wong’s lifetime of experience as a creator of interactive multimedia is distilled into this feature of the WWT. Tours are slideshows that move from one object in the sky to the next, and may be annotated with text, spoken-word audio, and music. But at any point you can pause the tour — or hop off the bus, as Curtis says — and explore the neighborhood on your own.

The WWT isn’t just a player of tours, it’s also an authoring tool for creating them. You create slides, navigate to objects in the sky, annotate them, and save the results in an XML format that you can reuse and share.

Like images from catalogs, tours are contextually available. So if you happen upon the Ring Nebula while exploring randomly, and if there’s a tour that mentions the Ring Nebula, then that tour will surface.

Curtis envisions a hypermedia web of sky narratives. For him, this storytelling aspect really is the heart of the project. In the interview he reveals for the first time that an early prototype for the WWT, shelved years ago, was to have been called John Dobson’s Universe.

Dobson, a leading amateur astronomer and innovative telescope builder, founded San Francisco Sidewalk Astronomers, a group that encourages telescope owners to take their telescopes out in public and share their knowledge of the sky. The WorldWide Telescope is poised to carry on that great tradition, and take it in some amazing new directions.

George Hripcsak, professor of biomedical informatics, is one of the recipients of a Microsoft Research grant to support work on the computational challenges of genome-wide association studies. These studies involve scanning complete human genomes, and looking for correlations between certain markers of genetic variation and certain diseases.

Doing that correlation is a computational challenge, but as I learned in my interview with George Hripcsak for Perspectives, that isn’t the challenge his research addresses. Instead he’s tackling a different challenge: mining electronic health records to figure out what they say about the diseases patients may have.

Why? Suppose you’ve sequenced the DNA of thousands of people for a study. If you’re trying to correlate genetic markers with disease, you need to know what diseases those people have. George calls this “collecting the phenotype” — that is, the expression of the genes responsible for diabetes, or a tendency to complications in labor, or whatever.

Traditionally that’s done by interviewing patients, a painstaking process that doesn’t scale. Given electronic health records, how much of this phenotype collection can be done automatically, and to what level of accuracy? That’s a different kind of computational challenge.

There are basically two ways to go. You can try to templatize the process of clinical data collection, so that health records can be harvested more effectively by researchers. Or you can try to understand the language that clinicians actually use when they describe patients.

For a decade now, George Hripcsak and his colleagues have been pursuing the latter approach, using a system for understanding natural language called MedLEE, which was developed at Columbia.

Ultimately I believe, as George Hripcsak does, that we’ll need a hybrid system that makes use of both structured templates and natural language understanding. But given that health records must primarily serve patient care, and can only secondarily serve research, I like how he harmonizes those objectives:

To the degree we make documentation efficient in serving health care, I think it’ll also be more accurate for the sake of research. If you’re filling out a record for the sake of billing, you’ll have an incentive to use diagnosis codes that optimize billing. Does that then reflect clinical accuracy? And would that then be useful for research?

The important thing is to be grounded in the clinical truth. Put health care first, and then use new computational methods to extract accurate information.

Amen.

As recently announced by Doug Kaye, the Conversations Network is embarking on a new phase. The existing channels, including ITConversations and Social Innovation Conversations, will continue. But rather than creating more such channels, the Conversations Network wants to help individuals and organizations capture and publish their own spoken-word audio, mainly in the form of events that are experienced only by attendees but that could be experienced by anyone, anywhere.

This new mission dovetails with PodCorps, a matchmaking service that connects event producers with volunteer stringers who can record those events. When it launched I wrote:

There’s a huge opportunity here to transform communication patterns in a fundamental way. Checking my local events calendar, for example, I see that the following event is scheduled for tonight at the local college:

Mon., Apr. 16
7 to 8:30pm
Pond Side 2 located on Bruder St - Keene State College

Building Smart - Highlighting Local Best Practices

Come and join us in discussing the challenges and successes of implementing innovative building materials, technologies, and design solutions into the built environment.

The information exchanged at that meeting, and at countless meetings like it, has historically been available only to those who attend. There are a million reasons why local folks who might want to attend nevertheless cannot: no babysitter, schedule conflict, etc. And of course remote folks have no opportunity to attend, even though the information exchanged might be highly relevant to them.

Assuming that more of these kinds of events become available, how will we find them? Doug writes:

We will do this using a social-networking model, which allows anyone to post links to recordings he or she finds, to build collections or playlists of their favorite recordings, to share those playlists with others, and to rate and comment on playlists or individual recordings posted by others.

In other words, Webjay for spoken-word audio. It’ll be interesting to see how this unfolds.

In my interview with Webjay’s creator, Lucas Gonze, we talked about some of the reasons why the curatorial model that Webjay promoted hasn’t yet succeeded. One of them, amplified in this comment by Greg Borenstein, is the fear, uncertainty, and doubt that pervades any distribution of — or even just linking to — MP3 files.

That kind of FUD shouldn’t be an issue for spoken-word audio that is explicitly free and legal. So I hope that we can evolve a culture of uninhibited collaborative curation. We’ll see.

I’ll also be curious to see what kinds of new channels and shows may arise from this effort. That isn’t the primary focus. Rather, the idea is to capture, share, and find recordings of events that have already been planned, organized, and held. The Conversations Network mainly seeks to enable the curation of those events. So someone might, for example, assemble the best recorded material in the alternative energy genre, from a variety of sources. I’d like to subscribe to that curator.

But there’s another kind of curation. It’s what I do when I select, from among the many people and ideas that I encounter, those I’ll feature on my two series of interviews: Interviews with Innovators and Perspectives. The world is full of interesting people and ideas, and we may also see the emergence of curators who select and highlight them in original ways. I’d like to subscribe to those curators too.

The Exchange-to-iCalendar script that I mentioned here is now published to CodePlex. It’s intended for organizations that run Exchange and would like to publish selected calendars in iCalendar (aka iCal, or .ICS) format without having to rely on a client machine running Outlook 2007.

I’ve never run a real Exchange server, so I’m wide open to suggestions as to how to actually publish the ICS file created by this little IronPython script. Right now, it just emits that file. For user Jon on Exchange host Zanzibar, you would do something like this:

ipy Zanzibar Jon > jon.ics

There are lots of ways jon.ics could get pushed to a web-accessible location, but I’m not sure what the default should be, or whether to do a filesystem operation, or an FTP transfer, or something else.

My idea is that you’d schedule this command to run on a regular basis, and that it would run under an account that has the necessary privileges to access the specified user’s calendar. But again, I’m not an Exchange admin, so if that sounds like the wrong thing, let me know what the right thing would be.

As for the iCalendar output, this script currently does the Simplest Thing That Could Possibly Work. It doesn’t, for example, try to “fold” long lines in the output (e.g., event summaries and unique IDs), which I gather the spec recommends but does not require.

There’s only been minimal testing. I’ve run it against a couple of different Exchange servers (2003 and 2007), validated the ICS output using this handy validator, and verified that the resulting files — containing both individual and recurring events — can be successfully imported, or subscribed to, in Outlook 2007, Google Calendar, and Apple iCal.

If you have a need for such a thing, try it and let me know how it goes.

Nick Carr’s essay in the current Atlantic Monthly crystallizes a lot of what I’ve been feeling for a couple of years about how our use of the Net is changing us. Not co-incidentally I read the essay in the printed magazine whose non-hypertextuality I experienced as a feature, not a bug. (Nick writes: “Unlike footnotes, to which they’re sometimes likened, hyperlinks don’t merely point to related works; they propel you toward them.”)

Although Nick doesn’t invoke Linda Stone’s touchstone phrase “continuous partial attention,” that’s the effect he’s noticing and reacting to. Whatever you think about the consequences, it’s clear that the Net doesn’t invite sustained attention, deep reading, and quiet contemplation.

Of course neither did cable television which, in the pre-Internet era, taught us to click-slice our attention much more radically than was formerly possible. But TV was just entertainment. You didn’t have to turn it on to do your job. Most of us do have to turn on the computer. As Paul Graham points out in Disconnecting Distraction, that’s a problem. He solves it by disconnecting his main work computer from the Net. But as work increasingly entails the use of software, and as software migrates into the cloud, I wonder how feasible that will be for most so-called knowledge workers.

Although Nick acknowledges that he’ll be branded “a Luddite and a nostalgist” I think his essay is spot on. The Net surely is rewiring our brains. What should we think about that, and what should we do about it? These are important questions.

Since we can’t turn back the clock, and we wouldn’t want to even if we could, I guess we’ll have to learn to live happily and productively in the noosphere we are creating.

Part of the answer is to develop — and teach — strategies that enable us to graze on the information commons in the most effective ways. I work hard at that.

But we also need strategies that enable us to retreat from that commons, and to experience sustained attention, deep reading, and quiet contemplation.

On that front, technology sometimes gives back with one hand what it takes away with the other. I’ll admit that it’s sometimes been a struggle for me, in recent years, to find that retreat in books, particularly fiction, and particularly in printed form. But portable long-form audio has been transformative. This weekend, on several multi-hour bike rides, I listened to the LibriVox version of Jules Verne’s Around the World in Eighty Days, beautifully read by Mark Smith (thanks Mark!).

It’s a great tale! I’d never have found the hours of couch time to re-read it, but I loved listening to it on my rides. (It was also nice to recall that I had written the software that simplifies downloading the book to a podcatcher.)

As I’ve said before, the quality of sustained attention that I find I can bring to long-form audio has been, for me, an unexpected benefit of great value. I’m sure our technologies are rewiring are brains in all sorts of ways, for better and worse. Often, those changes propel us into new and uncharted territory. But they can also help us reactivate ancient traditions, like oral storytelling, and rediscover their powerful neural effects.

On this week’s Interviews with Innovators show I spoke with Harry Lewis and Ken Ledeen, two of the three authors of the forthcoming book Blown To Bits: Your Life, Liberty, and Happiness After the Digital Explosion. The book explores why information technologies continue to produce surprising outcomes, and how society responds to them.

One of the threads running through the conversation is a perennial topic among thoughtful technologists: How much can most people understand about the underlying principles of information technology, and how much should they need to understand?

The authors believe that those principles are more explainable than we think, and as evidence they cite the remarkable tale of Hedy Lamarr, the famous Hollywood actress who — in collaboration with the musician George Antheil — invented and patented the idea of frequency hopping. Their goal was to help American torpedo guidance systems resist jamming by the Nazis. Now, of course, it has become one of the foundations of wireless communication.

Ken Ledeen tells the story at length, and with great animation, in our interview. Coincidentally, today’s NYTimes has a review of “an imaginative, two-character multimedia 80-minute play, ‘Frequency Hopping’”, that brings that same story to life.

We need to be careful about the lessons we draw from this tale. Although Lamarr the actress and Antheil the musician are usually referred to as the “improbable inventors” of frequency hopping, it didn’t happen by accident. They may have been scientific and engineering amateurs, but they were inspired and talented amateurs. And as Ken Ledeen explains, Hedy Lamarr’s social circle included a number of leading professionals from whom she absorbed a great deal of knowledge and understanding.

Still, if a pair of inspired and talented amateurs could invent such a thing, maybe we should have more faith in the ability of most people to understand its basis, and to reason about its implications.

I’m loving YouTube’s new video annotation feature, which Phil Shapiro alerted me to. Lots of people are going to have lots of fun with that. If you remember when MTV first started doing popup video, you’ll have some idea how much fun.

But from Phil’s perspective and mine, this is a seriously useful tool as well. He’s planning to annotate screencasts with it. And I found a great use for it here.

That short video features Bob Coffey, the senior climber at our YMCA. When I made and posted the video, I wasn’t quite sure how senior Bob was so I didn’t say. Yesterday I remembered to ask. Turns out he is 79.

It’s painful to add new information to a video. Opening up the raw file (if you even kept it around), adding a caption, recompressing, reuploading — it’s too much overhead, and unless there’s a compelling need you’re just not going to bother.

Of course you can update the textual wrapper, and alter the title or description. But in this case, I didn’t want to that. The information is much more effective when inserted midstream. After he’s scampered halfway up the wall, the popup annotation saying “Bob Coffey is 79 years old” makes the point more subtly and powerfully.

The point, by the way, is that we can do more, physically, at all ages, than we think. I’ve known a few people over the years who have redefined what’s possible, and it’s always an inspiring thing to see.

To complement my series on client-side calendar publishing, I’ve been looking for a way to push ICS files from Exchange. Why? A couple of local organizations with calendars I’d like to include in my calendar syndication project are running Exchange. It’s true that individuals within those organizations can use Outlook 2007 to publish calendars to the Internet. But companies like to manage these processes centrally. If the city wants its recreation center to publish summer activities, the city’s IT department should be able to do that whether or not the rec center’s desktop machine is alive.

Several folks suggested that the WebDAV interface to Exchange is the way to go, and after noodling on that for a while I’ve come up with a solution that seems to work. (I’ve tested it on a server here in my lab, and against my account on a production server.) I’m planning to release it as a CodePlex project, but as a Microsoft employee I have to cross a few t’s and dot a few i’s to get that done.

Meanwhile, since I’ve never administered Exchange, I have some questions for those who have. What I’ve got is a 100-line IronPython script that can be run on a scheduled basis. You give it a hostname and an account name, it reads events from that account’s calendar, and emits an ICS file.

First question: How does this fit in with your workflow? If you’re a town government, how would you like to manage your public calendar from Exchange? For example, would employees in various departments share one account that corresponds to that calendar? Would you rather divvy things up and then syndicate a set of calendars?

Second question: How will you want to publish the ICS file? My script only creates it. To publish the file to the Net, it could be copied to a directory on a webserver, or it could be uploaded to lots of places in lots of ways. It’s good to have choices, but it’s also good to have a sensible default, and I’m not sure what that should be.

Third question: Would you deploy this thing? I hope the small footprint will help. Being wary myself of solutions that haul in boatloads of requirements, I’ve whittled them down. You only need IronPython, plus of course the .NET Framework which it requires.

Originally, I was also using a few things from standard Python: command-line arguments, regular expressions. But it struck me that requiring all of standard Python for just those things was overkill, so I switched to their IronPython/.NET equivalents.

IronPython itself, resting as it does on the .NET foundation, is a tiny installation. Even so, it’ll be a new and unfamiliar thing to most people. Would an Exchange admin be willing to install IronPython on a production server and allow it to do the things this script does?

My guest for this week’s Interviews with Innovators is John Buckman, a serial entrepeneur with a passion for the world-changing possibilities of online communication. I was a customer of his first company, Lyris, whose email list manager I once deployed for a client. A few years later we met at SXSW, where John showed me his new project, Magnatune, and explained why he created it.

In this interview we discuss Lyris and Magnatune, plus some recent interests: BookMooch, an online book exchange, and Wreck a Movie, a “Web platform that is designed to harness the power of passionate Internet communities for creating short films, documentaries, music videos, Internet flicks, full length features, mobile films and more.”

This platform for crowdsourced film production, which is an outgrowth of the collaboration that led to Star Wreck, is now being used to create a new film called Iron Sky. You gotta love the tagline: “In 1945 the Nazis fled to the moon. In 2018 they are coming back.” As John says, who wouldn’t want to see Nazi invaders from the moon?

His ideas about how to do crowdsourcing closely parallel those of Yochai Benkler, as channeled by Bruce Sterling in his SXSW 2007 keynote, which includes a digression on this theme that begins thusly:

Socially motivated commons-based peer production: How to do it. You don’t just open up a website and invite comments. It actually has to be engineered with some thought and care. And I want to explain how this guy [Yochai Benkler] thinks it’s done. First you have to divvy up the work, because there’s a lot of it, and nobody wants to do it, and you’re not paying them, and you’re not ordering them, because you’re not the market and you’re not the state, so you can’t pay them and you can’t draft them. So you have to divide it up. It has to be granular, modular, and integratable. Granular means that even if I contribute for 5 minutes, I’m going to contribute something of merit.

Software is part of the answer. You need a framework within which to define tasks, parcel them out, and gather back results. Amazon’s Mechanical Turk is the best-known commercial example, but there are noncommercial precedents like distributed proofreaders and LibriVox. It’ll be interesting to see how Wreck a Movie follows and extends those examples.

John affirms Yochai Benkler’s point about granularity, and stresses the importance of precise task definitions.

If you say, nebulously, “Help us get press,” then nobody does anything. If you very specifically say this is the message, post to these press outlets, and name three URLs, that’s a well-defined thing. You get much stronger response. People need direction.

Also, I read recently that membership in volunteer organizations is down, and has been for a decade, but the amount of time people put into volunteer tasks is up. Why? People are less interested in open-ended commitments, instead they’re interested in knocking off specific tasks, and feeling a sense of accomplishment for finishing them.

As Bruce Sterling says, you have to engineer that scenario with thought and care. It’s a kind of engineering that is both social and technical, and some of the leading practitioners are game designers. For more on this theme, listen to Ned Gulley’s reflections on the MATLAB programming contest he’s been running for a number of years.

When David Stutz left Microsoft, he wrote a parting essay that invoked a new kind of Internet-oriented operating system characterized by “software that runs above the level of a single device.” Tim O’Reilly echoed that phrase here, and often used it to help explain what he meant by Web 2.0.

The recently-announced LiveMesh is a nice example of software that runs above the level of a single device. It runs symmetrically on all your computing devices, in the part of the cloud that’s associated with your devices, and in other parts of the cloud where services you transact with are running. This entire constellation is the LiveMesh platform which, as Ray Ozzie recently explained to investors, is an answer to this question:

What would an OS look like in a world of multiple devices, in a world where instead of the computer being at the center, you are at the center?

The platform’s connective tissue, as discussed in my interview with Ray Ozzie, is FeedSync, a synchronization system based on the same simple technology that powers the blogosphere: XML feeds of items, in RSS or Atom formats. Whether they represent big chunks of information like documents and media files, or small scraps of information like calendar events and status messages, LiveMesh objects are made up of feeds. All these objects synchronize across your mesh of devices and services using the same openly-specified FeedSync mechanism.

That openness is another key characteristic of an Internet operating system. So it’s nice to see that FeedSync isn’t only being applied in the context of LiveMesh. This week’s Perspectives interview, with Barbara Willett of Mercy Corps and Nigel Snoad of Microsoft Humanitarian Systems, details Mercy Corps’ use of FeedSync to collect, synthesize, and share information about the management of agricultural development programs in Afghanistan.

In this case, the synchronized data sources are humble Access databases for which Nigel and his team have developed a FeedSync adapter. They’ve also built adapters for spreadsheets and — what should be very interesting to Ken Banks and Joel Selanikio — for SMS messaging systems.

I love stories about pragmatic solutions that find new ways to use existing, simple, and widely-deployed technologies. This is clearly one of those. But it also illuminates an aspect of the LiveMesh platform that hasn’t yet been widely noticed or appreciated. One of its keystones, FeedSync, is an open and general-purpose building block that can be used by anyone, for any purpose.

If the Mercy Corps solution interests you, Nigel says that the toolkit he and his team built for them will be openly released in a few weeks. Watch the FeedSync blog for details.

The word drafting has many meanings but the one I’m interested in here comes from bicycling. When you ride closely behind another rider, you’re drafting. The leader pushes the headwinds out of the way, and the follower doesn’t have to push so hard.

Blogging can work that way too. I thought of this when I reallized that one of the benefits of subscribing to James Fallows is that I’m drafting on his interest in the fledgling air taxi industry. My interest in that topic is more than casual. I’ve interviewed DayJet founder Ed Iacobucci, for example. But James Fallows is way more deeply invested than I am, having written the seminal book on the topic, Free Flight.

I can do a pretty good job of tracking developments on that front by scanning the news, or better yet by subscribing to searches for terms like DayJet and Eclipse 500. But the best way is to draft on a blogger who is authoritative on the topic.

I don’t need to see every news story about DayJet, and pushing them all out of the way in order to focus on the ones that really matter is like pushing a headwind. But James Fallows is already motivated to push that headwind, so I can just draft on him. That way I get just the right air taxi newsfeed, with a dollop of expert analysis on top.

Happily, the analogy breaks down in a couple of ways. A cyclist can only draft on one other cyclist, and it’s a one-way relationship. The follower can’t simultaneously lead. With blogging, I can draft on many peoples’ interests, and many people can draft on mine, and sometimes the leader/follower relationship is reciprocal — I draft on you for topic A, and you draft on me for topic B.

For our purposes here, we can define blogging broadly to include the conventional format, but also microblogging formats like del.icio.us crumbtrails and Twitter tweets. Drafting, in the sense I mean here, can happen in any publish/subscribe medium.

As part of my project in community calendar syndication, I would like to find a way to push an Exchange calendar to a web-accessible ICS file. Although that isn’t a native function of Exchange, I’m sure it can be accomplished by way of the Exchange API, as an add-in or a scheduled server process. For maximum breadth, I guess the relevant API would be Collaboration Data Objects (CDO) rather than the newer Exchange Web Services.

I asked some folks on the Exchange team and none were aware of a component that does this. So I’ve set up an Exchange server in my lab, rolled up my sleeves, and am ready to dive in and see what can be done. But it never hurts to ask. Have I overlooked an existing off-the-shelf solution?

This post is part three of a series in which I’ll summarize what I know about publishing calendars openly on the web, for free, using popular calendar applications including Outlook, Google Calendar, and Apple iCal.

Apple iCal

If you have a .Mac account you can publish your calendar there, but
.Mac isn’t free, and the purpose of this series is to showcase free
calendar publishing options.

The solution here is to find a free service that uses the same
protocol, and the same kind of server, as the .Mac service uses.
The protocol is called WebDAV, and the server is a
special-purpose web server commonly used for calendar publishing.

It’s possible that your ISP already offers a WebDAV server you can
use, for no additional charge. But for many people it would be ideal if there were a free service available for this purpose. One such service is iCal Exchange.

Here’s the signup page:

After completing the form you’ll land here:

For private sharing you can create passwords and use the private URL, but our goal here is public sharing so the public URL — in this case, http://icalx.com/public/judell — is the one you’ll use.

Now switch to iCal, select Calendar, select Publish, and switch the Publish option from the default — .Mac — to Private Server. Paste the public URL you just created into the Base URL, and enter your iCal Exchange credentials in the Login and Password fields.

Now click Publish. Here’s the outcome:

The web address for your calendar is the public web address that iCal Exchange gave you, plus the name you gave your calendar (in this case, Jon), plus the .ics extension. For this example, it adds up to webcal://icalx.com/public/judell/Jon.ics.

This solves half of the problem. Your calendar is now published in a way that enables individuals to subscribe to it. It’s also available for syndication by online services like http://elmcity.info/events.

But the other half of the problem remains unsolved. In parts one and two of this series, we saw that the free calendar hosting options offered by Microsoft and Google provide links to hosted calendar viewers.

There are a variety of other hosted viewers, but as yet I’ve not found one that’s free, and can render any public calendar given a public URL like the one shown here. If such a service does exist, I’m hoping this entry will help me find it.

My guest on Innovators this week is Greg Wilson. We share common interests in collaboration and Python, but neither of those topics was the focus of this conversation. Instead, we discussed Greg’s unique and somewhat curmudgeonly take on high-performance computing. In his view, the HPC industry has focused on achieving bigger and faster computation at the expense of human productivity, verifiable correctness, and reproducibility.

I claim no expertise in that field, but Greg is an expert, so I wondered what he’d think about the approach discussed in one of my recent Perspectives shows, Cluster computing for the classroom. On that show, Kryil Faenov — Microsoft’s general manager for Windows HPC — describes a system that enables professors to define computational models that students can check out, tweak, and then run against large data on a compute cluster.

From a human productivity standpoint Greg likes that approach. But he’d prefer to see more attention paid to verifying the correctness of the models, and to ensuring that code and the data are managed in ways that make experiments reliably reproducible.

Disclosure: While working at Los Alamos National Laboratory back in 2000, Greg commissioned me to write a report on Internet Groupware for Scientific Collaboration.

This post is part two of a series in which I’ll summarize what I know about publishing calendars openly on the web, for free, using popular calendar applications including Outlook, Google Calendar, and Apple iCal.

Google Calendar

You’ll need a Google account. If you use Gmail you already have one. Start the calendar program by clicking the Calendar link at the top of the Gmail page.

To publish your calendar in ICS (aka ICAL) format, open the drop-down menu for your calendar’s name under the My Calendars heading, and select Calendar Settings.

The first tab on the ensuing page is called Calendar Details. If you scroll to the bottom you’ll see two sections containing sets of hyperlinked icons. The sections are labeled Calendar Address and Private Address.

You don’t actually have to make your calendar public in order to share both its ICS (ICAL) and HTML formats. You could use the second set of private links to publish (and otherwise communicate) those formats without exposing the contents of your calendar to the Google search engine. But if the goal is to advertise your calendar as widely as possible, you’ll want to do that. So, visit the second tab on this page, labeled Share this Calendar, and check Make This Calendar Public:

Now your ICS feed is active at an URL that looks like this:

http://www.google.com/calendar/ical/yourname%40gmail.com/
public/basic.ics

To capture your version of this link, right-click the ICAL icon in the Calendar Address section, and use your browser’s link-capture method: Copy Shortcut (IE), Copy Link Location (Firefox), Copy Link (Safari). You can paste the link into a web page that you publish, or into a web form or an email that transmits it to another site to which you want to syndicate your calendar.

Similarly, the web view of your calendar is active at an URL that looks like this:

http://www.google.com/calendar/embed?
src=yourname%40gmail.com&ctz=America/New_York

To capture your version of this link, right-click the HTML icon in the Calendar Address section and do as above. This link leads to a Google-hosted page for viewing the calendar.

If your web hosting circumstances allow you to use an HTML feature called IFRAME, you can instead embed the calendar in one of your own pages. The HTML code to do that is provided in the Embed This Calendar section.