Build your own interactive network

Similar to the map builder, there is a web-based version of the network builder under development. The web based version is useful as it provides immediate feedback on a new interactive visualization — due to browser security restrictions the visualizations will not run from a local computer without a webserver. Nevertheless, the scale and size of network data also demand an offline alternative for networks.

I am very happy to announce the first version of such an option: a Sigma.js export option for Gephi. The code we have written for this has been tested, but it should be considered an ‘alpha’ version. We very much welcome feedback and information on any bugs you discover.

To create an interactive network visualization with this offline approach, you will need to have Gephi installed. It is free and available for Linux, Mac, and Windows, and can be downloaded from the Gephi website.

Once installed, open Gephi and go to Tools->Plugins. Click “Available Plugins” and find “SigmaExporter” in the list. If you don’t see it, please click the “Reload Catalog” button and see if it then appears. Place a checkmark next to it and click the “Install” button at the bottom of the window. Accept the license agreement (GNU GPL v3), and click to install even though it hasn’t been signed (I’ll need update the plugin to address this later).

You must then exit Gephi and open it a second time. From that point onward, you will have a new menu option in the File->Export menu. Open any network and visualize it the way you like within Gephi. When you are ready to share your network online, click File->Export->Sigma.js template… . Close a suitable location and provide the necessary information (anything marked with * should be filled in at a minimum). Click ok and your network will be built. You will not be able to preview the network on your local machine without a webserver installed (this is why a web version is in the works). Upload the “network” folder in the location you specified to any webserver of your choice and share the link with all your friends. We’d love for you to share the link with us too and provide feedback.

If changes are needed, re-run the export wizard a second time and re-upload. The only files that change are config.json and data.json. Advanced users may also wish to just open and edit config.json in a text editor after formatting the JSON with JSONLint or another tool.

Build your own interactive map

After initial testing in our closed beta period, the InteractiveVis team is very happy to open the Interactive Map Builder to everyone. All you need is some interesting data at a country level — or use our literacy data to start. Information about the data format is available in the Preparing data… post, and the actual wizard to produce a map is at: http://api.oii.ox.ac.uk/InteractiveVisBuilder/.

Please try it out, let us know yours thoughts, and send feedback (here as comments or by email — there are prizes available for helpful feedback!. Once you build a map, you will be able to preview it temporarily on our server, but you will need to download the visualization as a zip file, extract it, and post it to your own web server to have a permanent version. There are no specific requirements for the webserver. If you are within the University of Oxford, you will find information on accessing your personal webspace on the ITS website.

Start Building »

US election zoom-in

Update: 6 November 2012 – Featured in the Guardian.

Interactive world maps are great, but the excitement over the US Presidential Election left us wanting to get a bit more detail. Data collected by Mark Graham, Adham Tamer, Ning Wang, and I (Scott Hale) created the opportunity to try this. I built a new US State Map template based on this Wikipedia map, and the result follows. Mark has a much more detailed discussion of the data on his blog under Obama wins the election! (on Twitter). Enjoy! The US state map option will be added to the map creation wizard in due course.

Network visualization lunchtime talk

Scott Hale, of the InteractiveVis project team, will lead an introductory talk on network visualization next Wednesday, 7 November at Oxford’s IT Services. All from the university are welcome to this free event.

The talk will cover basic steps to collect, clean, and visualize network data using two desktop software solutions: Gephi and NodeXL. There are also some online solutions to capture network data including: 140kit to capture Twitter data and NameGen to download a network your (own) Facebook friends.

The InteractiveVis project aims to help you share these visualizations more easily. The talk will cover existing desktop software methods to prepare static images for posting on a web, but will then turn to how to create interactive network diagrams like those below easily. The InteractiveVis project team is working finish testing and deploy a step-by-step process now.

Demo 1: Visualization of followers at @OIITwitter

Background information about this visualization

Demo 2: Visualization of UK Central Government

Preparing data to create your own interactive map

Behind every visualization is data. After some extensive internal testing we are preparing to release a wizard to create map visualizations next week. This wizard will require the data in spreadsheet form as a Google Spreadsheet. (We’d welcome suggestions on future formats and sources). An example of the data for the current visualization on literacy is available as a Google Doc for you to view and adapt to your own data. The first two columns are required along with at least one additional column of data. The first column is a country code (e.g. ES for Spain) while the second is the label to display when this country is clicked (e.g. Spain or España). Any additional columns are custom columns to hold data to shade countries by or to display as a bar graph or text label in the right-hand panel. There is a row for each country for which data is available. If all data points are unavailable for a given country, it is safe to delete the row for the country. Lastly, it is optional to add a “world” row which is optionally shown when the visualization first loads.

So, open the spreadsheet on Google, save it to your account, and change the data to your own in preparation for creating your own map visualization next week! Lots of data that could be displayed is at the CIA World Facebook or at the World Bank if you need inspiration.

Data

Resulting Visualization (updated and slightly different from image below)

Updated with IE-compatibility, conference appearance, wizard coming soon

Internet Explorer Compatibility

The Interactive Visualizations are standards compliant, but not all browsers are. We are relying on the Sigma.js framework to display network visualizations. This library loads networks from an XML (gexf) file and displays them on a canvas object. Since the library’s release, however, XML parsing has not worked correctly on Internet Explorer. While this browser is a decreasing part of the market, we want to support it and have therefore written a new import plugin. We now parse the XML file to JSON first and simply load this JSON file. This has made the file size smaller, the load time faster, and increased compatibility to work with Internet Explorer as well as other major browsers (Firefox, Opera, Chrome, Safari). We’d particularly welcome feedback from Internet Explorer users on the visualizations — please remember to let us know your browser version. For users already using Sigma, code for the new JSON parser is available in the OII branch of the project. Simply run gexf2json.py on a gexf file and then load the results through the new paseJson plugin for Sigma. Code for the entire InteractiveVis project is also available on github.

Internet, Politics, Policy Conference / Wizard

The InteractiveVis project had a very successful demo session at the Internet, Politics, Policy Conference last week. We received a lot of interest and also some great feedback. We’re working to finish a web-interface / wizard to let anyone upload their own data and produce visualizations like these. Incorporating some of the helpful feedback received via the blog and email has put us a little behind on this, but it is now our top priority. I’ll update the blog with more details on the wizard soon.

Feedback Needed: Interactive map demo

skip to the demo…

As with the network diagrams discussed earlier, geographic data can often benefit from the ability to be explored in an interactive manner. The World Bank has many statistics on a per country basis as do other organizations. The CIA World Fact Book also has many statistics like this useful listing of literacy broken down by gender.

After converting the above data to tabular form, it was possible to produce a map with each country shaded according to one attribute (such maps are called choropleth maps). These can be produced as static images with desktop software including the commercial application arcGIS or the open-source alternative qGIS. The CIA literacy data, however, has three statistics per country: total literacy, male literacy, and female literacy. With static images, it could be possible to show these three statistics on three separate maps or to use shading with a bar chart on top of each country. Neither of these solutions is ideal, however. Separate maps are sometimes difficult to compare, while the use of bars for every country quickly makes the map too crowded. On the earlier static image of this data shown below, we compromised by including the break down by gender for only some countries.

Literacy and Gender

The static image, however, is less than ideal. Exact numbers and gender breakdowns are only available for a small number of countries that the authors selected. In addition, the colours of the smallest countries are harder to see. So, whether it’s literacy, internet usage, income, tweets, Wikipedia edits, or another statistic, interactivity is a key way to allow the user to get exact numbers, view smaller countries, and see a break down of additional information for every country. The additional information in this example is male and female literacy shown with bars, but it could be a line plot showing change over time or text information.

So, without further ado please find below a demo visualization of what we’d like to allow users to produce and customize with a future web-based wizard. We’ve tested this visualization in the latest versions of Safari, Firefox, Opera, and Chrome on Windows, Linux, and Mac, but we consider it to be very much a work-in-progress. We’re releasing it to get feedback that we can incorporate into the project. The more specific you can be in describing any suggestions or issues the better. Please at a minimum include your operating system (Windows, Mac, Linux) and browser version.

So, please leave feedback! What works, what’s missing, and what would you ideally change? In addition, what options need to be supported for the right side pane besides bar charts? Leave a comment below or send an email.

Demo 3: Visualization of Literacy and Gender

Feedback Needed: Interactive network diagram demos

This post lays out some of the importance of interactivity for network diagrams and we think is well worth a read (or at least a skim!); however, if you wish to skip to the demos, these are linked at the end of the post.

Visualizing network data is a prime example of a research activity where both desktop tools and online libraries exist, but are not well connected with one another. Desktop software like NodeXL and Gephi provide the ability to collect and visualize network data (users on Twitter, videos on YouTube, etc.). These applications, however, provide limited facilities for hosting, sharing, and optimizing visualizations for the web. Numerous technologies, frameworks and platforms exist for displaying network data online, but they often depend on bespoke data formats or a high degree of technical expertise.

The visualization below shows the Twitter network of the Oxford Internet Institute (@oiioxford), and was produced without a single line of code needing to be written. Data was collected through NodeXL and visualized with Gephi.

Twitter Network of @OIIOxford

However, when trying to share this visualization online, the software we had used to create it left us wanting.  The only option readily available was to export as an image and post it to a blog, department website, or gallery. There are numerous practical reasons that this solution was undesirable. For example, it would have been necessary to export at an extremely high resolution to preserve the legibility of the smallest nodes, but this would have created an impractical file size. Also, navigating such a large image file in a web-browser would likely prove awkward and difficult to manage for users, which would limit the intelligibility of the visualization. However, perhaps a more pertinent issue concerns the general usefulness of static network visualizations. When viewed in Gephi or NodeXL, the easy accessibility of node and edge attributes encouraged the data to be explored. These attributes are flattened and reduced to purely visual indicators when a static visualization is created.

Network visualizations like this one seem a prime candidate for interactivity, and indeed great work has already been done with the Sigma.js JavaScript library to enable the rendering of network graphs with HTML5 canvas. This library allows all newer browsers as well as tablet devices and phones to display the visualization without any extra plugins (e.g. Flash) or software (e.g. Java) needing to be installed.

Sigma.js does, however, require knowledge of JavaScript and HTML to produce an interactive network diagram. The amount of knowledge required grows further if the creator of a visualization wishes to provide an infobox with more node information, an ability to search for a specific node, or to change fonts and colours. The library also could be improved in some subtle, but important, ways: we’d like to add the ability to detect when the user positions the cursor over a specific edge, properly handle redrawing on tablets when the visualization is zoomed, and provide the option for edges to be a blend of the colours of the two nodes they connect. We will work to make these changes and provide a hosted wizard to allow a user to easily create and customize the visualization (what attributes of a node should be shown on click, what title and explanatory text should be included, should any edge information be shown on hover, should a search function be included, etc.).

Before the wizard is released, however, we are very excited to release two demo visualizations as examples of what could be produced and customized with the wizard. We’ve tested these visualizations in the latest versions of Safari, Firefox, Opera, and Chrome on Windows, Linux, and Mac, but we consider these to be very much work-in-progress. We’re releasing them to get feedback that we can incorporate into the project. The more specific you can be in describing any suggestions or issues the better. Please at a minimum include your operating system (Windows, Mac, Linux) and browser version.

So, please leave feedback! What works, what’s missing, and what would you ideally change? Leave a comment below or send an email.

Thanks so much!

Demo 1: Visualization of followers at @OIITwitter

Demo 2: Visualization of UK Central Government

Existing tools and OII InteractiveVis approach

From the beginning of this project, we have been keen to build upon the best of the existing tools and not “reinvent the wheel.” At the same time, however, we have set ourselves some unique parameters to allow the widest use of visualizations produced with the tools we develop. In particular, we want our visualizations to be native HTML5 visualizations, run as standalone / offline visualizations, work on tablets and other touch-screen interfaces. At the same time, we want to make the process of going from datafile to visualization as easy and accessible as possible. We feel confident this combination of factors isn’t available yet, but there are some existing building blocks we can use. This blog post will look at each of the parameters mentioned above and then turn to look at existing visualization tools (Many Eyes, Google Charts, Simile Widgets) and libraries (Sigma.js, D3, Miso, Raphael).

Parameters

First, we wanted to use only native web technologies. While Java, Flash, and other technologies are powerful (and currently more common) ways to show and present data, these technologies require plugins that not all devices support. In particular, the Apple iPhone and iPad have not supported Flash since their launch, while Adobe will halt new installations of Flash on Android devices and will not support Flash for the newest version of Android. Tablets and phones, however, could be a great medium to explore data interactively with touch. These devices, along with desktop computers running modern web browsers, support web technologies (HTML5, CSS, HTML, SVG, Canvas) that can allow a rich, cross-platform experience for interactive visualization.

More than just running in a web browser, however, these technologies power the newest, emerging ebook formats. So, visualizations developed with native web technologies should also be compatable with ebook formats once ebook reading applications implement the newest standards. To be the most versatile, however, visualizations should be entirely self contained so that once downloaded (or bundled with an ebook) they can work even when the device is offline.

Finally, and most importantly for this project, we want to simplify the process of creating these visualization by allowing the creation and customization of the visualization in a standard web browser.

Existing tools and libraries

All of the existing tools we know of lack at least one of the three parameters above. Many Eyes, for instance, relies on Java and StatPlanet uses Flash. While Google Charts and Google Maps use native web technologies, they require an active Internet connection.

Finally, there are some libraries that use native web technologies and work offline, but these currently require the creator of a visualization to be able to know some programming or scripting languages. Among the emerging libraries are three that this project is actively building upon: Sigma.js, Raphael, and Miso. Each of these and some of the alternatives will be detailed in subsequent blogs posts with prototype visualizations.

On the horizon

While our project aims to move from datafile to interactive visualization online, there is still a lot of preprocessing to get data into the correct format. For network data, this will often involve desktop programs or scripting languages for data capture and layout (Gephi, NodeXL, R). While many of these desktop tools will always be necessary, there are some workflows that could be executed entirely within a web browser. 140kit by OII MSc student Devin Gaffney aims to ease the collecting of data from Twitter, while a proposal by the authors of Gephi aims to allow network layout in the browser as well. Additionally, work by project member Joshua Melville and his supervisor Dr. Bernie Hogan has resulted NameGen, a Facebook application which exports network data in common file formats and uses Gephi and Sigma.js to layout and render Facebook ego-networks entirely in the browser.

Beyond Books slide presentation

The OII InteractiveVis team has been busy at work and anticipates an exciting release of two demo visualizations extremely soon. In the meantime, there is lots to share, and I wanted to share one item now (can you tell we’re more into coding than blogging? sorry, we’re working on getting more up on this blog).

At the end of June we had the opportunity to present to at a Learning Technology one-day conference hosted by the Oxford University Computing Services entitled Beyond Books. It was a great opportunity to see ideas about the direction of ebooks and get feedback on the work we’re doing to enable interactive visualization that should be compatible with emerging ebook standards. We also took the opportunity to showcase the ability of HTML5 in general; so, the slides we presented we’re written entirely in HTML5 based on some earlier slides created by individuals in the Google Chrome team. I also learned at the conference about impress.js, an open-source HTML5 alternative to pretzi.

Without further introduction, the slides are below. They layout some of the existing tools (a blog post on these is forthcoming), and some of the parameters within which we are developing (native web technologies only, standalone/offline capability for ebooks).

Clicking the image below will launch the slides. Use the left and right arrow keys on your keyboard to switch slides. The presentation requires a modern browser: ideally Firefox, Chrome, Opera, or Safari

Click this image to see the slides