Web widget and mashup

Draft

Definition

Web widget

“A web widget is a portable chunk of code that can be installed and executed within any separate HTML-based web page by an end user without requiring additional compilation. They are akin to plugins or extensions in desktop applications. Other terms used to describe a Web Widget include Gadget, Badge, Module, Capsule, Snippet, Mini and Flake.” (Web widget, Wikipedia, retrieved 19:58, 14 May 2007 (MEST)).

“Client-side web applications, or widgets, are typically self-contained applications for displaying and updating remote data, packaged in a way to allow a single download and installation on a client machine or mobile device.” (Widgets 1.0 Requirements, W3C Working Draft 9 February 2007)

“Widget: A software component used by a mash-up assembler within a mash-up maker to create a new mash-up. Widgets encapsulate disparate data, UI (user interface) mark-up, and UI behavior into a single component that generates DHTML code fragments. Widgets generally respond or act upon user interaction or data events.”(QedWiki FAQ, retrieved 19:25, 16 May 2007 (MEST)).

Notice: This article also deals a bit with desktop widgets because they have a similar purpose, i.e. they typically provide easy access to frequently used functions (e.g. the clock) and increasingly pull information from the Internet and provide some visual information. Some applications can exist in both versions. E.g. you can read Gmail in your browser or install it on the desktop (beta).

Mashup

“These Web applications are a type of a situation application that yield a new utility by seamlessly combining content from two or more sources or disparate components with new behavior to form a new, integrated Web application. A mash-up is the application artifact resulting from the assembly of information-rich widgets.” (QedWiki FAQ, retrieved 19:25, 16 May 2007 (MEST)).

Important note: Some writing about Mashups below doesn't feel right. There is a confusion between simply importing and combining contents and widgets within a webtop (or similar) and real mashup that combines contents from services into a new kind of widgets (e.g. typically what people do with Google Maps). Have to go over this at some point - Daniel K. Schneider 11:00, 7 June 2007 (MEST)

See also: AJAX, Universal Widget API (Netvibe's API), Pageflakes API and all other articles in the category Rich internet applications

What are web widgets and mash-ups ?

(1) Web widgets or web gadgets are pieces of code that can be easily embedded in webpages. They represent smaller or larger applications that usually also rely on some server, e.g. interface with a news site (through RSS), rich internet application (RIA) such as "office 2.0" (web 2.0) applications. Widgets are largely combinations of HTML or XHTML, CSS, and Javascript. In particular, simple "content" widgets are increasingly popular, e.g. on 19:58, 14 May 2007 (MEST) Google lists over 6000 and Widgetbox over 10000. Most widgets are probably based on RSS feeds. Some widgets also exist as Desktop widgets or can be converted from a web widget to a desktop widget.

(2) End users can run this kind of widgets on several "platforms", i.e. various kinds of web widgets are available in several ways:

• on webtops (easy drag/drop or some "add" button),
• insert widget code intro webtops or personal blogs (fairly easy)
• insert widget code into HTML pages

The same kind of widgets also can be installed on a client computer. However we wouldn't call them web widgets anymore, but desktop widgets (computers) or mobile widgets (mobile phones, PDAs, etc.).

• Extensions to web browsers that Daniel K. Schneider would qualify as widgets.
• Widgets "docked" to a desktop of an OS (Windows, MacOSX, Linux).

(3) Web widgets can be combined into a web mashup according to the following formula:

Web Mashup = API [1] + API [2] + API [N]

A web mashup is a website or web application that uses content from more than one source to create a completely new service. Content used in mashups typically comes from a third party via a public interface (API), an RSS feed or sometimes screen scraping. webtop are an example of sophisticated mashups, i.e. individual users can configure a "start page"

“We know we don't have a corner on creativity. There are creative people all around the world, hundreds of millions of them, and they are going to think of things to do with our basic platform that we didn't think of. So the mashup stuff is a wonderful way of allowing people to find new ways of applying the basic infrastructures we're propagating. This will turn out to be a major source of ideas for applying Google-based technology to a variety of applications.” (Vint Cerf interview with J.C. Perez, retrieved 19:58, 14 May 2007 (MEST))

So web widgets can be of different kinds. They work within a kind of environment (a HTML page, a webtop, etc.) and they use different technologies to talk to a server. Some are cut/past end-user ready, some run in specific environments (e.g. webtops), some are meant to mashed up by developers. A web widget that runs on one "platform" does not run on an other (unless there are appropriate versions). There is a strong relation with so-called rich internet applications. Some of these can be included as widgets, or more precisely the web service behind these can be accessed through widgets.

The same is true for mashups. It's not really clear what they are. Major categories could distinguish between different kinds of "mash-up assemblers" (IBM):

• Mashups made by and for individual users (e.g. a webtop used as personal learning environment.
• Mashups built for others (e.g. a mashup built by a teacher for his class).
• In-between mashups where a mash-up enabler designs what array of mash-ups (in a given "mashboard" provided by an organization) can be created from a collection of widgets.

Using and building widgets

(just some for now ...)

How to use widgets

Basically, an end-user has several choices (depending on the technology)

(1) Get a widget insertion code from a widget site

• On these sites, there is at least "get the code" button that will display some short JavaScript code which you then can insert into a web page, a blog, a webtop, etc. Unfortunately it's not so easy to find widgets of interest to educators, since the most popular widgets focus on specific information or games ...

• Some of these web site have "add to" buttons, that will directly import a widget your desktop (e.g. MS Live), your webtop (Netvibes, iGoogle, Pageflakes, etc.), your social website (MySpace, Blogger, etc.).

• To quickly build your own good looking home page, use a generator, then insert JavaScript code. E.g. web 2.0 generator (note: this site was meant to be a bit of a joke, but it does generate nice looking pages ...)

(2) Use a mashup maker, i.e. a service that supports assembly of widgets

Webtops and similar software provide users with a larger set of widgets from which they then can compose an "Ajax start page". See:

• See webtop, which usually are mashup-up tools for individual use. Also read about personal learning environments that includes a story on how to build such a thing with a webtop.
• An increasing amount of social software services are configurable with extra widgets.
• Mashup tools for organizations (e.g. a teacher) that will allow you to create an half-baked environment for a group of people (e.g. learners). Without testing (!), Daniel K. Schneider believes that QEDWiki from IBM might be a good contender. "Enterprise mashups" ought to be more powerful, e.g include a safe data repository.

(3) Use browser, application or OS-dependent widgets

• Modern desktop applications like web browsers or desktops themselves also allow users to insert widgets that interact with the Internet. Typical examples are:
• Yahoo! Widget Engine
• Google Desktop
• KlipFolio

How to create widgets easily

This is for people with a little HTML/XML/JS knowledge. Web Widgets often but not always use Adobe Flash or JavaScript programming languages. The widget code itself is usually small and connects to some server application.

• Some services have widget creation kits for a certain number of widgets. E.g. iGoogle allows you to create photos, "daily me", you tube channel, personal list...

• Services like [1] dapper that allow you grab "any content from the web". Requires some technical skills.

Mashup software for developers

According to Eric Armstrong, to create a mashup, you need:

• Web page GUI components and a GUI Builder (likely an AJAX tool).
• A communication mechanism, e.g. REST
• Data formats, e.g. microformats, RSS, JSON
• One or more data sources you can access
• Optionally, a data repository you can interact with

Programmers may use published APIs (see for example the list in Webmashup.com). To build the mashup itself, developers may use a so-called AJAX development kit, but recently mashup development kits came into existence (see links below). Some of these are relatively simple to use others less it seems. I.e. it seems to me that there is some grey zone between mashup builders meant for end-users, for web designers in organizations, for people with some scripting knowledge and for "real programmers".

To program webservices plus associated libraries/widgets is another issue.

Widget architecture

For the moment there are several published APIs but none can be called a standard.

The W3C proposal

W3C proposes design goals and requirements for a specification that would standardize the way client-side web applications (widgets) are to be scripted, digitally signed, secured, packaged and deployed in a way that is device independent.

According to this widget 1.0 requirement working draft, a widget-architecture is usually composed of the following distinct parts:

A host runtime environment

The software that hosts instances of a widget. Host runtime environments are either directly built on, or provide similar functionality to, a common web browser. Examples include Yahoo!'s Widget Engine, Microsoft Windows Vista Sidebar, and Apple's Dashboard.

A media type

A pre-registered Media Type that associates the package with a certain runtime environment on the end-user's computer. For example, Opera's current 'application/x-opera-widgets' media type.

A packaging format

A self-contained structured resource that encapsulates the contents of a widget for the purposes of distribution and deployment. For example, Apple's Bundle format and related directory structures as specified in the Dashboard Reference.

To standardize: a format, e.g. a zip

A manifest

A distinguished resource of the package that describes how the widget and host runtime environment should be configured when the widget is instantiated. The manifest may also define the relationship between resources in the package. The manifest resource usually takes the form of an XML document that declares properties and behaviors of various components of the widget and the host runtime environment. For example, the 'config.xml' resource bundled with an Opera Widgets package.

To standardize: The abstract internal structure (if any), including required resources and directories. This looks a bit like IMS Content Packaging

APIs

A set of programming interfaces that provide functionality specific to widgets. Current Widget APIs range extensively in the level of functionality they provide an author; see for example Microsoft's API for Windows Vista in the Sidebar Reference and Apple's APIs for widgets in the Dashboard Reference.

To standardize: The APIs that authors can use to programmatically access:

• • Access preferences of each instance of a widget.
  • Access runtime configuration properties and other relevant platform properties like localization information.
  • Access to services particular to the device on which the widget has been instantiated (eg. camera, and the Short Message Service (SMS)).
  • Access metadata values that the author declared in the manifest.
  • Request that the host runtime environment check if an updated version of the widget is available for download.
  • Capture events and access properties that are particular to each instance of a widget.

Resources

Images, text, graphical user interface markup, style sheets, executable scripts and other possible parts of the widget. Resources may be organized in directories and may have directories versions tailored for localization purposes.

W3C widgets 1.0 requirement

Desktop widgets according to the The W3C widgets 1.0 requirements document

In the appendix of this working draft one can find some information on popular host runtime environments. Unfortunately webtop APIs as Netvibes and Pageflakes are not included, but future versions of this document may do so - Daniel K. Schneider 01:42, 22 May 2007 (MEST).

The following table lists the packaging formats currently used by various vendors to bundle widgets and that are supported by host runtime environments.

Runtime	Container Format	File Extension	Media Type
Yahoo! Widgets Engine	Zip, directory, proprietary flat-file	.widget	application/vnd.yahoo.widget
Microsoft Sidebar	Zip, Cab, directory	.gadget	application/x-windows-gadget
Google Desktop	Zip	.gg	app/gg
Opera Browser	Zip	.zip	application/x-opera-widgets
Apple Dashboard	Zip, Apple bundle	.wdgt or .zip	application/x-macbinary
AOL Modules	Zip	.html	text/html

The next table lists the formats of the manifest files various vendors are using as part of packaging widgets. It also identifies what markup language authors are required to use when declaring the user interface of widgets.

Runtime	Manifest Format	Manifest File	UI Markup
Yahoo! Widgets Engine	Proprietary XML Yahoo! Widgets Reference	*.kon	Proprietary XMLWidgets Reference
Microsoft Sidebar	Proprietary XMLMicrosoft Gadgets	gadget.xml	HTML + CSS
Google Desktop	Proprietary XMLGoogle Gadgets	gadget.gmanifest	Proprietary XMLGoogle Gadgets
Opera Widgets	Proprietary XMLOpera Config	config.xml	HTML + CSS
Apple Dashboard	Proprietary XML Apple pList	Info.plist	HTML + CSS
AOL Modules (any capable User Agent)	Microformat AOL ModuleT	index.html	HTML + CSS

Web widgets APIs

There are different kinds of APIs, e.g.

• Public open APIs (like Google APIs, eBay, Netvides, etc.)
• Public closed APIs
• APIs that only work with given mashup platforms

Web 2.0 API's are not standardized and require various programming knowledge. Some organizations also provide client libraries to make things easier. E.g. for the Google APIs there is a Zend Google Data Client Library (PHP). Other libraries exist for Java, .NET, Python, JavaScript (via JSON).

Netvibes

• See Universal Widget API

Software Links

This section is quite messy. Since I just started playing with this tech, I don't have a good enough "map" of these - Daniel K. Schneider 19:25, 16 May 2007 (MEST).

Widget management sites

(widgets to download, needs to be sorted out at some point. Also some websites include both widgets and more stand-alone web 2.0 applications. Anyhow, the difference is not really clear to me ...)

• Widgetbox. A directory and syndication platform for web widgets for blogs and other web pages. Claim to work with TypePad, WordPress, Blogger, MySpace as well as most other blogs, sidebars or websites.

• programmableweb

• Widgetoko

• Yourminis

• LabPixies.com. Good Widgets that you can directly add to several popular environments (e.g. desktop or webtops).

• NeoBinaries (online resource for web based applications)

• Google Gadgets

• Snipperoo (free registration required).

• WidSets (for your mobile phone)

• TagWorld Widgets (free registration required)

• widgetlabs.com

Desktop

• Microsoft Live http://get.live.com/

• Yahoo widgets (works with widgets 4 application, Win/MacOS)

• Bubbles

Wiget builder kits

Forms Creation

• WyaWidget from Realtime Applications is a really nice Widget Creator with which you can create fill-in forms in minutes. See an example on my Pageflakes test site (go to the student sample page)
  • Can use it with pageflakes, NetVibes, your HTML and others, really easy to use !
  • Disclaimer: I don't know how safe/good/compliant this is with various webtops.

Mashup builder kits

End-user mashups

• Most popular mash-up makers are webtops

• Schmapplets to share places on the web with maps, photos, articles and reviews. (This is just an example of a tool that allows to create mashups for a given domain). (Windows/MacOSX).

Easy mashups

• Data Mashups. Create personalized portals in minutes for your customers or staff (Beta, minimal service is free).

• Teqlo

Global Enterprise mashup tools

We focus free or open source products here. See also Enterprise MashUps (blog entry) and development kits below. It's not very clear what such a tool is, since there are two dimensions to it: "web 2.0" things, service-oriented architecture.

• QEDWiki from IBM is a lightweight mash-up maker written in PHP 5 and hosted on a LAMP, WAMP, or MAMP stack. (Only available for trials !). Daniel K. Schneider thinks that this kind of technology is more or less what a teacher would need since it supports assembly, wiring of components and sharing. In other words, it allows teachers to create a situational application fitting both teacher's and learner's needs.
  • QEDWiki, IBM, and "Situational Applications, retrieved 19:25, 16 May 2007 (MEST).
  • QEDWiki (Wikipedia)

• Bungee. On-demand, web-based development environment that enables developers to build and deploy web apps that utilize the large variety of APIs and web services out on the Internet. See Richard MacManus' Bungee Labs - Next Generation Web Development Platform article.

• Kapow sells various products. OpenKapow is a free application that can (quote) create RSS feeds, REST services or clips from almost any web site. You have to download RoboMaker and then create "Robots" that will do the mashup. There is a larger list of existing Robots you may use. Java-based Windows/Linux, on 19:25, 16 May 2007 (MEST) download included a JAVA VM. Kapow's (full) mashup server interfaces with IBM QEDWiki and Yahoo Pipes.

• Yahoo pipes, a kind of RSS remixer (yes, like in Music)
  • Yahoo Pipes by Anil Dash, retrieved 19:25, 16 May 2007 (MEST).

• Others: JackBe, Backbase, Nexaweb

Tools for manual mashup creation

• Dapper. Dapper make it easy to extract and reuse content from any website. It can create various kinds of formats (XML, RSS, Google maps, HTML, Netvibes.) etc. from various kinds of input formats (Dapp XML, RSS, Google Gadget, ...). A certain number of prebuilt Dapp's are also available.

Web 2.0 (mashup) APIs

Indexes and repositories

• programmableweb APIs Keeping you up to date with the latest on mashups and the new Web 2.0 APIs. Good directory !

• Webmashup.com The open directory of mashups & web 2.0 APIs

APIs (some)

• Netvibes' UWA
• Google. By far the most popular API (globally) seems to be Google Maps
• Flickr
• del.icio.us

Lower Widget/Mashup development kits

There are several kinds of development kits and the difference is not always really clear to Daniel K. Schneider. The hottest trend may be trough the web development and integrated mashup development kits described [#Mashup builder kits|above].

• AJAX kits of various kinds. Some of these now turn into "WebOS Application Builders". (e.g. Morfik).

• Some web operating systems (WebOS), e.g. eyeOS can be installed on your own server. Others at least provide APIs to create widgets for webtops.

• Libraries provided by service providers, e.g. Google APIs

• Traditional portalware also may soon include web 2.0ish functionality (and some actually do). E.g. Portaneo POSH.

Mashup Examples

• Saga Earth An interactive game that challenges users to find countries on a map of the world, then displays videos from that country.
  • Saga Earth page on Programmable web.

• Skillsfeed a microlearning platform for course authors and students. It provides fast access to small chunks of learning content composed of Wiki text, multiple choice tests, movies from YouTube, GoogleVideo, Grouper, etc. and Slideshare presentations.
  • SkillsFeed / Programmable Web

• YouTorials Find tutorials on YouTub]
  • YouTorials / Programmable Web

References and text links

Tutorials

• Raymond Yee, Mashup Guide Blog for a Book in Progress about Remixing Data and Services on the Web, includes draft chapters (in PDF)

• AJAX Desktop Tutorial - Overview (The purpose of this tutorial for AJAX programmers is to explain, step-by-step, how to code your own AJAX desktop/homepage).

• NewWebPlatform User submitted tips, tutorials and scripts for Web 2.0 API and developement platforms.

Vision / Short Overviews

• Khasanshyn, Renat (2007). Three Trends Influencing Enterprise 2.0, Blog Entry.

• Kirkpatrick, Marshall (2006), PostApp launches WidgetBox, a marketplace for widgets, TechCrunch. [2]
• MacManus, Richard (2006). Widgets are the new black, Read/Write Web. HTML

• Comparison of widget engines (Wikipedia) desktop widgets.

• Widget engine (Wikipedia), desktop widgets.

• Mix, Match, And Mutate, BusinessWeek article.

• Build Your Own Web 2.0 Application Using Fluff and Hot Air

• Mashing Up the Web for Learning (wiki by Gerry Paille). See also his del.icio.us links

• Lamb, Brian (2007). Dr. Mashup; or, Why Educators Should Learn to Stop Worrying and Love the Remix. EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 12–25 HTML (good overview of educational potential including some refs.)

Specifications / APIs

• Widgets 1.0 Requirements, W3C Working Draft 9 February 2007 HTML. Latest version: http://www.w3.org/TR/widgets-reqs/

• Open Social "a new universe of social applications all over the web". A openweb API called Open Social, which is being spearheaded by Google and others (oct. 2007, rather cool if this can take off).

Various

• Hacking Web 2.0 applications with firefox (some good information about security).

• programmableweb. ProgrammableWeb is where you can keep-up with what's new and interesting with mashups, Web 2.0 APIs, and the new Web as Platform.

• MashupCamp.com

• NewWebPlatform User submitted tips, tutorials and scripts for Web 2.0 API and developement platforms.