I have something to say about that…

Distributed and syndicated content: what’s wrong with this picture?

You know those AMP URLs you get from Google search results and which often pop up on Twitter?

Instead of https://www.rt.com/sport/… you’ll get https://www.google.co.uk/amp/s/www.rt.com/document/…

Screenshot of AMP's RT article, headline: Meet Achilles the Cat, deaf animal psychic
What you’re seeing is Google’s AMP project hosting content for Russia Today. This lets Google load the page during the search results, so that when you click on the link on the search page, the text appears immediately.  (This is solving a big problem, by the way.  That shorter loading time can make the web a far more enjoyable experience.)

Facebook’s Instant Articles and Apple News operate similarly but without the benefit of being on the web or using real URLs — a much worse starting point.

The web relies heavily on the “origin policy”, which amongst other things, helps browsers manage permissions (e.g., access to your location, camera, microphone, etc.), attribute bad actions (phishing attacks), and assist you with things like passwords and filling out forms.  This core aspect of web architecture ties permissions and security settings to a particular origin, like rt.com. Distributing or syndicating content removes that context by hosting one site’s content within a different site, which can confuse users and stop browsers from keeping the web safe.

In the W3C Technical Architecture Group we have been thinking about this issue.  While we understand the value these approaches provide, they also pose serious issues. Fundamentally, we think that it’s crucial to the web ecosystem for you to understand where content comes from and for the browser to protect you from harm. We are seriously concerned about publication strategies that undermine them.

We have published this finding to explain our thoughts in more detail.

This post originally appeared on the W3C TAG blog.

Where terrorists go to chat; government and the end-to-end encryption problem

One reason we form governments is to protect our communities. At the same time, our economy and human rights depend on private and encrypted online services. How do we move forward when these two agendas clash?

What’s prompting this post

Following this week’s explosion in the Manchester Arena, we in the UK are struggling to come to terms with the loss of children, the unsettling reminders of our vulnerability, and the stark contrast in our communities coming together in the aftermath.

We are having the to-be-expected conversations about why this happened, what we can learn, and how we protect ourselves. We are reexamining what we expect of our government. It’s part of how we heal as a country, how we pick ourselves back up.

Some of the discussion inevitably turns to encryption and how terror plots are organised — in the UK, abroad; face to face, over the internet. Quickly we run into the encryption question: end-to-end encrypted services can’t be decrypted in between the users’ devices, which makes it difficult for authorities to identify a conspiracy.

Home Secretary Amber Rudd outlined the problem in her comments after the Westminster attack:

“It used to be that people would steam open envelopes, or just listen in on phones, when they wanted to find out what people were doing, legally, through warrantry — but in this situation we need to make sure that our intelligences services have the ability to get into situations like encrypted WhatsApp.”

We have seen this conversation come up again and again, during the debates for the Investigatory Powers Act in 2015 and the (ultimately dropped) Communication Bill of 2012. It also resurfaced a few weeks ago, after the Westminster attack.

It feels like a discussion at a stalemate; I’m seeing government asking for the problem to be solved, and technologists rolling their eyes at the implications that “government wants to outlaw maths.”

Having been on both sides of this discussion, I want to explain the miscommunications I see happening and outline the (few) options I think we have to proceed.

The source of the conflict

There are two conflicting pressures pushing us towards this impasse.

Problem 1: The democracy problem

In the UK, we ask (and pay our taxes for) our government to keep us safe. We expect it to be in every party manifesto on which we elect the next government. We authorise it through a large percentage of our government’s budget. We, often through our press, actively get upset when our government doesn’t keep us safe, and we launch inquiries and hold leaders accountable when they fail.

Our police and national security machinery are constantly trying to keep up with the changing ways criminals act. The rise in end-to-end encryption on messaging services has complicated their jobs — and when they hear us asking to be kept safe, they have pointed to this as an obstacle.

So they’re asking us as the tech industry to “fix it”. If we don’t, they can’t do their jobs properly — which is what we, as citizens, have asked them to do.

Problem 2: The technology problem

In a completely different vein, the we — the tech community — are building an internet on which our society and economy can flourish. We are fighting a whole industry of criminals who are trying to undermine this — as we all know, we need to protect ourselves against phishing, malware, unauthorised intrusions, man-in-the-middle attacks… Our infrastructure is vulnerable in a lot of ways. As I’m fond of repeating, we initially set up the protocols in the internet and web stacks to optimise for sharing — we’re only recently retrofitting security to it.

Continue with reading

The evergreen web

You know those old browsers in TVs, exercise bikes, kiosks and the like that can’t browse the web anymore? Have you ever noticed how strange it is that they become dusty and increasingly hard to use, when the browsers in your mobile phone or laptop carry on very well?

It happens because no one keeps them up to date. As web technologies (and therefore, websites) evolve around them, they get further away from being able to handle what a site serves them. And as a result, they become increasingly less useful.

A black-and-white old browser with an error message: "Unable to load https://theguardian.com"
Photo from an exercise bike’s defunct browser, from Peter O’Shaughnessy of @samsunginternet

I’ve edited a finding with the W3C Technical Architecture Group about that.

In The Evergreen Web, we write:

Constant evolution is fundamental to the Web’s usefulness. Browsers that do not stay up-to-date place stress on the ecosystem. These products potentially fork the web, isolating their users and developers from the rest of the world.

Browsers are a part of the web and therefore they must be continually updated. Vendors that ship browsers hold the power to keep the web moving forwards as a platform, or to hold it back.


Data on the Web Best Practices — draft standard in Proposed Rec stage

For the past two years, we — the W3C Data on the Web working group — have been working on a standards document for publishing data on the web. Things like:

  • Provide descriptive metadata
  • Use machine-readable standardised data formats
  • Provide bulk download
  • Make data available through an API
  • Use web standards as the foundation for your API (REST etc)

The best practices aren’t especially groundbreaking — but just think, when you’re trying to use the data that an app or government has provided and this stuff is missing… Isn’t it frustrating?!?

We wanted to make sure the definitive guide to how to put data on the web was available for those who wanted to “do it right”.

So… a) for all of you developers. It’s out there. Enjoy.
b) for W3C members, the spec is in PR until 15 Jan 2017. Responses welcomed until then.

Best Practices for Data on the Web:

Announcement of publication for PR:

My OvertheAir keynote: Trump, Brexit and us as developers

I gave the opening keynote at OvertheAir yesterday, covering President-elect Trump, Brexit and what it all means for us as developers. Topics like:

  • Data protection laws. Will your app from London be able to handle users in another country? Or will you need to do something special to be compliant with their laws? Will it matter where you host data about your users?
  • The importance of informed and empowered users. We need to build services that make clear what data is going where. And I think we REALLY need to standardise private browsing mode. Everyone should know what it does when they turn it on… but it varies widely from browser to browser!
  • Keeping transactions secure. If everything depends on economic growth, and economic growth depends on secure, reliable transactions… Security and strong encryption will be crucial to our future.
  • Fake news. We (the web community) — well, we didn’t invent fake news. But we did create ways for it to be distributed on a mass scale. Therefore, we have some responsibility here — we should work towards fixing it.

There is still a lot that isn’t settled, on the political/governmental fronts, but it’s useful to keep an eye on the facts we have and the questions we’ll need answers to as things unfold. Lots to do ahead — and lots to build.

Photo by @documentally courtesy of Nexmo.

Photo of Hadley at a podium, in front of a slide that says State of the Web

Video: my keynote at ViewSource

I gave the opening keynote this morning at Mozilla’s ViewSource conference in Berlin. View Source has gathered an amazing group of web developers to explore new frontiers in the open web.

I talked about how the open web lets us build it as we like — we get to make the rules. And there are lots of rules left for us to shape. The web is not yet a finished product.

The outline of my talk is below, if you aren’t in the mood for a video.

Text: The rules are fluid, because we make them. We have the ability to shape the web. What should it be? -Hadley Beeman
Quote originally published on @viewsource’s twitter feed at https://twitter.com/viewsourceconf/status/775606536861278208

Outline of my View Source keynote

Continue with reading

Video: Making things open (OSCON 2015)

I gave the opening keynote at OSCON 2015 in Portland, Oregon. I was speaking on behalf of the UK government and the Government Digital Service.

Summary of the talk from the conference site:

Openness is good for Government on many levels — open data, open standards, open source, open markets. Where we set the way we work in Government, it’s important to let industry determine the technical standards we work with. We can’t do everything ourselves and by making our own code and data open we have an opportunity to gain from innovation as well as support other services to be developed by suppliers of all sizes.

Make things open, it makes things better.

My memories of the 7/7 bombings

Ten years ago today… my morning commute to work was disrupted. There was much confusion at Waterloo Station. The tube station was closing, and everyone was milling about near the trains.  There were rumours of trouble in multiple parts of the tube network, but no one seemed to understand the disruption.

I hopped on a bus and called an old friend, who I knew had been up all night working on her PhD on nuclear terrorism.  I thought she might interpret the news better than I could.

“They’re reporting ‘a power surge,'” she told me as my bus wound through Holborn towards Farringdon.  It’s hard to tell what’s actually happening.  Be careful just in case.”

I got to work — one of the few who did, it turned out — and was asked to make a list of my team and work out who was still alive. While the sirens raced past our office buildings en route to various hospitals, I made the phone calls. I never want to do that again.

(They were all alive, fortunately. But I counted those endless sirens, thinking of those teams across the capital — and families — who were finding that some of their members weren’t.)

I remember phoning my mother and waking her up.  “You’re going to see that there’s something going on in London,” I told her. “I just want you to know, when you see that, that I’m at work and I’m okay.”  She blearily thanked me and apparently went straight to watch the news… which was already reporting explosions.  I got a somewhat teary voicemail from her a number of hours later — when the phone networks were no longer clogged — thanking me and telling me she loved me.

The City was evacuated in the afternoon, and I headed across London on foot towards friends in South Kensington (since I lived too far away to get back to my home). All the Londoners I encountered had a surreal quality of shock at the events and a heightened, startled awareness of each other… We weren’t just obstacles in each other’s journeys anymore — we talked. We nodded to each other. We shared our worried looks and our stoic laughter. As we all tried to work out how to get home, it felt like we actually saw each other for the first time. We understood we were in it together.

When I got to Hyde Park Corner, a couple of brave TfL bus drivers had picked up their routes — all the more courageous when we didn’t yet know what had happened to cause the explosions, nor whether it was truly finished. I stepped onto a bus and was humbled to see the driver, as confused and stricken as the rest of us, determined to do his part: London was on the move, and he could help us get home. What a gesture of solidarity. I thanked him profusely.

Coincidentally, I found one of the South Ken friends at the back of that bus, and we went on to their flat to make margaritas with his wife. There was lots to talk about that night, and to be grateful for. And to mourn. We’d each experienced it differently, but we’d been through it together — and London is never more amazing than when it finds a reason to pull together.

Uses for open data

I’m often asked these days why people would bother with open data. (Here, I’m using LinkedGov’s definition of open data.)  I thought it would be useful to write down and gather some feedback, see if we can refine these categories further.

Thus far, it seems, the uses are boiling down to four categories:

1.  Transparency

Broadly speaking, this means getting a better view of what is going on inside government or the public sector.  This audience covers both the non-public sector and the public sector itself.


  • Infrastructure:  Transport timetables, traffic information or road potholes for a journey planner app
  • Accountability:  Financial and budget statements for armchair auditors
  • Media:  Potential headlines and stories for journalists
  • Sharing information resources:  Formal research available to inform academic and professional enquiries (for example, data from NHS clinical studies informing projects hosted by universities or industry). This group also includes management and demographic statistics, like the number of people in a particular benefits programme
  • Status and progress updates: performance data, such as the number of outcomes met in a specific project
  • News: announcements about public sector activities, grant opportunities and new ways to interact with government
  • Community information:  local planning applications, crime statistics or upcoming events which impact a neighbourhood

2.  Delivering services to/on behalf of government

Open data allows commercial and third sector organisations to have a closer relationship with customers and funding sources in government and the public sector.


  • Delivering front-line services on behalf of a governmental or public body:  As an example, the train operating companies might benefit from greater access to forecasts of passenger activity from Transport for London.
  • Marketing to government:  If a photocopier sales department can see which public sector offices are likely to need a new photocopier soon, they can target their marketing appropriately.

3.  Improving commercial activities outside of government

Many existing business models could benefit significantly from greater access to public data.  A few examples:

  • Smoothing commercial transactions. A tool for selecting the ideal import tariffs or a faster route of calculating tax could provide significant savings for a commercial goods company.
  • Enhancing an existing offering.  A tour operating company could plan more accurately (or prompt their clients to plan better) with weather data from the Met Office.
  • Targeting marketing.  Census data and council tax bands, for example, could help a new company work out where its target market is, helping them to concentrate their comms efforts in the most efficient place

4.  Efficiency

Much of the public sector could benefit from better access to their data and the information contained within it.  Examples include:

  • Procurement:  Comparing costs and existing contracts when looking at procurement for something new.
  • Evidence base: Better informed policy development and decision-making
  • Reducing the load: Less enquiries from the public (specifically requests under the Freedom of Information Act) and from within the public sector (for example, parliamentary questions from ministers to civil servants in their department).

What are your thoughts?  How can we refine this model and make it more complete?

We the people vs Facebook, Google et al.

A theme in this morning’s news items struck me:

It’s interesting to me that these issues are based in the same quandry: how do we, as a society, deal with placing the control of our content in the hands of a few big providers?

The writers and the publishers – a contract

User-generated content comes out of a relationship: the writers (us) write things, generate data through web activities, and create links to people, while the hosts (Facebook and Google, here) gather the information and do neat things with it.  They share our posts with our friends, connect us with ads that might interest us, and host our status updates and regulate who sees what we are up to.

The first two links are public retaliations for what the plaintiffs feel is a betrayal of trust by Google and Facebook.  They put their trust in these two tools to safeguard their content. They are unhappy that Google and Facebook changed the rules (or perhaps violated their side of the agreement) with the users by changing the defaults on what information is public.

This, to me, is an age-old “breach of contract” question.  Have Google and Facebook in fact violated the terms of service, to which they agreed when each user opened an account with them?  And if so, what do they owe us?

Making amends

The next story is about Facebook, having heard the outcry (well represented by the aforementioned lawsuit) and attempting to re-establish good will.  Though they aren’t admitting that they have done anything wrong, they appear to be trying to regain some of the trust they lost in November and December by offering users more control over who sees posts from the various applications they use.  (The example cited in the Facebook blog explanation: I’ll let the Someecards app post to my close friends only, but My Causes can post to everyone including the boss.)

As the Facebook announcement says, “Facebook is designed to give you control over the information you share.”  I think they are hoping that even greater control will result in a stronger feeling of contract and trust between the users and their tools.

Be careful what you say…

“The danger is publicly telling people where you are. This is because it leaves one place you’re definitely not… home.”

Pleaserobme.com is a tongue-in-cheek reminder that all information posted on the web is public.  Also that most posts can be added to other bits of content for more context than we might intend.

Pleaserobme.com takes basic posts to Twitter from the location-based app Foursquare, which announces where a user is when they check in at that location.  As the Pleaserobme site says, “The danger is publicly telling people where you are. This is because it leaves one place you’re definitely not… home.”

There are a number of ways to work out where someone lives, not the least of which is that many homes are being added to Foursquare as check-in destinations. Sure it’s nice to know where your friends are, but this could be problematic!

(Side note: when I added a new location to Foursquare on Tuesday, it offered me the choice to have that location be private among my friends.  It appears that they are already trying to counter this problem.)

But the idea is that, by announcing on Twitter that I have checked in at a location that isn’t home, then all my valuables at home are open for the taking.  Obviously, that’s not good.

As a content-generator in this relationship, I have to be aware of what information I am releasing to my hosting platforms (Facebook, Twitter, Foursquare, Google, etc.) and how that information can be compiled.

Are we making progress?

We can talk at length about the generational change in individual data, and how kids today will grow up happily sharing every last bit of their lives on the Web.  (I’m not convinced of this, by the way- I think they will grow out of a lot of their exhibitionism.  Caution and desire for privacy often comes with age.)

But these stories represent, to me, an ongoing push-me-pull-you tension of expectations and service provision, as the capabilities and they way they’re used continually race ahead of each other.  I think our society and laws will continue to swing back and forth on privacy issues as we re-establish our norms and our expectations for companies that hold our content.

*Photo from http://www.flickr.com/photos/38057014@N05/3542597760/