Sorry, we don't support your browser. Some features may not function or display as we intended.
Please upgrade to at least Internet Explorer 8 (preferably version 10) or the latest version of Chrome/Safari/Firefox.

Developer Documentation

Help with using our API

Introduction

This page describes the current version of our production API, which was deployed on 26th of June 2013. This release included only backward-compatible changes. See our Change Log for more details of what changed.

1 - Linked Data API

1.1 URI Dereferencing

Following the standard practices for Linked Data, we distinguish between a 'real-world' resource and documents about that resource. Identifiers (URIs) for the resources follow the pattern:

http://opendatacommunities.org/id/{...}

When you look them up you get redirected to the corresponding document about that thing. The document URLs follow the pattern:

http://opendatacommunities.org/doc/{...}

For example, in our Local Authorities Dataset, the URI for Cheshire East Unitary Authority is

http://opendatacommunities.org/id/unitary-authority/cheshire-east

If you put this into your browser you get redirected, with an HTTP status code of 303 ("See Other"), to an HTML page about Cheshire East

http://opendatacommunities.org/doc/unitary-authority/cheshire-east

In cases where a URI identifies something that is essentially a document (an 'information resource') then we respond with a 200, as their URI and document page URL are one and the same. This includes Datasets as well as ontology terms and concept schemes.

1.2 Resource Formats

You can specify what format you want the resulting document to be in. By default you get HTML in a human-readable form, but you can also ask for the document in one of several RDF formats: RDF/XML, N-triples, Turtle or JSON-LD.

There are two ways to specify which format you want: you can append a format extension to the document page's URL or you can use an HTTP Accept header with the resource's URI or document page's URL.

Format Extensions Accept Headers
RDF/XML .rdf application/rdf+xml
n-triples .nt, .txt, .text application/n-triples, text/plain
Turtle .ttl text/turtle
JSON-LD .json application/ld+json, application/json

1.3 Example: Dereferencing URIs with Ruby

Here's an example of dereferencing a URI using the RestClient library. Similar approaches can be taken in other languages. This assumes you already have Ruby set up on your system. Also, if you don't already have it, you'll need to install the gem:

$ gem install rest-client.

... and require it in your script.

require 'rest-client'.

1.3.1 Specifying the format in an accept header - in this case RDF/XML

If you're using the accept header, you can directly request the URI. This involves two requests, because doing an HTTP GET on the resource identifier gives you a 303 redirect to the appropriate document page. RestClient looks after that for you.

RestClient.get 'http://opendatacommunities.org/id/unitary-authority/cheshire-east', :accept=>'application/rdf+xml'

You can also request the document page directly:

RestClient.get 'http://opendatacommunities.org/doc/unitary-authority/cheshire-east', :accept=>'application/rdf+xml'

1.3.2 Specifing the format as an extension - in this case JSON

If using an extension, you must request the document page directly (as '.json' is not part of the URI)

RestClient.get 'http://opendatacommunities.org/doc/unitary-authority/cheshire-east.json'

1.4 Example: Dereferencing URIs with cURL

Here's an example of dereferencing a URI using the widely available cURL command line program.

1.4.1 Specifying the format in an accept header - in this case Turtle

If you're using the accept header, you can directly request the URI. This involves two requests, because doing an HTTP GET on the resource identifier gives you a 303 redirect to the appropriate document page. cURL looks after that for you if you use the -L option.

curl -L -H "Accept: text/turtle" http://opendatacommunities.org/id/unitary-authority/cheshire-east

You can also request the document page directly

curl -H "Accept: text/turtle" http://opendatacommunities.org/doc/unitary-authority/cheshire-east

1.4.2 Specifing the format as an extension - in this case N-triples

If using an extension, you must request the document page directly (as '.nt' is not part of the URI)

curl http://opendatacommunities.org/doc/unitary-authority/cheshire-east.nt

2 - Other Resource APIs

Alongside the URI dereferencing we offer the following additional ways of accessing data in the system. Please be sure to read the Options and Limits section, for some background information which applies to all these APIs, such as details on data formats and pagination.

Some examples of accessing the data from our APIs using different languages follow at the end of this section.

2.1 Individual Datasets

Dataset identifiers take the form http://opendatacommunities.org/data/{dataset-short-name}, where {dataset-short-name} is a URI section that uniquely identifies the dataset. The short name can contain lower-case letters, numbers, slashes, and hyphens.

Dereferencing a dataset identifier responds with HTTP status code 200 and provides metadata about the dataset, including a link to where the dataset contents can be downloaded.

e.g. http://opendatacommunities.org/data/homelessness-decisions

Please also see the Use of Named Graphs section, for how the dataset data and metadata is stored in the database.

2.2 Themes

Datasets are grouped into Themes. A list of all themes is available at: http://opendatacommunities.org/themes.

Information about a particular theme can be accessed by dereferencing the theme's URI. e.g.

http://opendatacommunities.org/def/concept/themes/homelessness

2.3 Collections of Datasets

A list of all datasets is available at: http://opendatacommunities.org/data, paginatable with page and per_page.

Lists of datasets in a single theme are available at: http://opendatacommunities.org/themes/{theme-slug}, where {theme-slug} is the part of the theme URI after the final slash. e.g. homelessness

2.4 Individual Resources

As well as using URI dereferencing to access information about individual resources, you can use the following URL pattern:

http://opendatacommunities.org/resource?uri={resource-uri}

This is especially useful for resources for which we have information in our database, but which aren't in the opendatacommunities.org domain (i.e. so you can't dereference them in our site).

e.g. http://opendatacommunities.org/resource?uri=http://data.ordnancesurvey.co.uk/id/postcodeunit/SW1A1AA

If using a format extension to request a particular format for the resource, the extension is added immediately after '/resource', for example to get a JSON-LD version of the above postcode

http://opendatacommunities.org/resource.json?uri=http://data.ordnancesurvey.co.uk/id/postcodeunit/SW1A1AA

2.5 Collections of Resources

Collections of resources can be retrieved from /resources by supplying filters. For now, we just support filters for dataset and type_uri.

Filter parameter Expected value Behaviour
dataset The short name of a dataset (see above). Filters the results to only include resources in the named graph of that dataset.
type_uri The URI of a resource type. Filters the results to only include resources of the type identified by that URI.

e.g. http://opendatacommunities.org/resources?dataset=local-authorities&type_uri=http%3A%2F%2Fopendatacommunities.org%2Fdef%2Flocal-government%2FLondonBoroughCouncil

2.6 Options and Limits

2.6.1 Formats

Resources accessed via our resource APIs can be accessed in the same choice of formats as for URI dereferencing (via both format extensions or HTTP Accept headers).

2.6.2 Pagination

For any APIs which return collections of things, the list can be paginated using page (default 1) and per_page (default 1000) query-string parameters. The maximum allowable page size will initially be set to 1000, but we may consider increasing this (as well as the default) in the future.

2.6.3 Response Size Limits

All requests to our APIs are subject to the response size limits.

2.7 Example: Using Ruby to get a filtered list of resources

2.7.1 Basic Example

Here we use Ruby to retrieve a list of all the observation resources in the Homelessness Descisions dataset as N-triples.

The short name for that dataset is homelessness-decisions, and the URI for the observation type is http://purl.org/linked-data/cube#Observation, so the URL we need to call is as follows. (See the Collections of Resources section).

http://opendatacommunities.org/resources?dataset=homelessness-decisions&type_uri=http%3A%2F%2Fpurl.org%2Flinked-data%2Fcube%23Observation

If you visit that URL in your browser, you'll see a paginated list of the resources. We want to get it in N-triples format, so we'll add the .nt extension. (See the Formats section).

The following Ruby code assigns a string of N-triples into the ntriples_data variable. Note that as the maximum page size is 1000, and there are over 1000 resoures of that type in the dataset, we'll need to make multiple requests.

We use the RestClient here, which you can install with $ gem install rest-client.

require 'rest-client' url = "http://opendatacommunities.org/resources.nt" ntriples_data = "" page = 1 done = false while !done puts "requesting page #{page}..." response = RestClient.get url, {:params => { :page => page, :per_page => 1000, :dataset => "homelessness-decisions", :type_uri => "http://purl.org/linked-data/cube#Observation" } } if response.length > 0 ntriples_data += response page += 1 else puts "no more data" done = true end end puts "data:" puts ntriples_data

2.7.2 Extension: parsing the n-triples into an array of statements.

The ruby-rdf library is useful for parsing various rdf formats. Install it with $ gem install rdf. The following code reads our string of ntriples data into an array of RDF::Statements.

require 'rdf' statements = [] RDF::Reader.for(:ntriples).new(ntriples_data) {|r| r.each {|s| statements << s}} puts "parsed #{statements.length} triples"

Note: If you're doing a lot of work with RDF in Ruby, you might want to look at using Swirrl's open-source SPARQL ORM for Ruby, Tripod.

2.8 Example: Using JavaScript to get a filtered list of resources

Here we use jQuery to retrieve a list of all the observation resources in the Homelessness Descisions dataset as JSON-LD.

The short name for that dataset is homelessness-decisions, and the URI for the observation type is http://purl.org/linked-data/cube#Observation, so the URL we need to call is as follows. (See the Collections of Resources section).

http://opendatacommunities.org/resources?dataset=homelessness-decisions&type_uri=http%3A%2F%2Fpurl.org%2Flinked-data%2Fcube%23Observation

If you visit that URL in your browser, you'll see a paginated list of the resources. We want to get it in JSON format, so we'll add the .json extension. (See the Formats section).

The following HTML page uses JavaScript to request the data as JSON and add it to the results array. Note that as the maximum page size is 1000, and there are over 1000 resoures of that type in the dataset, we'll need to make multiple requests.

<!DOCTYPE html> <html> <head> <script src="http://code.jquery.com/jquery-1.9.1.min.js"></script> </head> <body> <script type="text/javascript"> var perPage = 100; var typeUri = "http://purl.org/linked-data/cube#Observation"; var dataset = "homelessness-decisions"; var baseUrl = "http://opendatacommunities.org/resources.json?" baseUrl += "per_page=" + perPage.toString(); baseUrl += "&dataset=" + dataset; baseUrl += "&type_uri=" + encodeURIComponent(typeUri); var page = 1; var results = []; function callAjaxPaging() { console.log("trying page: " + page.toString()); url = baseUrl + "&page=" + page.toString(); $.ajax({ dataType: 'json', url: url, success: function(pageOfData) { results = results.concat(pageOfData); console.log("got " + results.length.toString() + " so far"); if (pageOfData.length == perPage) { // this page was full. There might be more. page += 1; console.log("trying next page"); callAjaxPaging(); } else { // no more pages. alert('finished with ' + results.length.toString() + " results"); } } }); } alert('press OK to begin'); callAjaxPaging(); </script> </body> </html>

2.9 Example: Using cURL to get the list of datasets in a theme

Here we use the cURL command line program to get a list of datasets in the housing theme, as JSON-LD.

The slug for the housing theme is housing, so the URL we need to call is as follows. (See the Collections of Datasets section).

http://opendatacommunities.org/themes/homelessness

We'll use the Accept header to tell the server we want the response as JSON.

curl -H "Accept: application/json" http://opendatacommunities.org/themes/homelessness

3 - SPARQL

3.1 Introduction to SPARQL

The most flexible way to access the data is by using SPARQL. Pronounced "sparkle", SPARQL stands for Sparql Protocol and RDF Query Language. It's a query language, analagous to SQL for relational databases, for retrieving and manipulating data from triple-stores like ours. We support SPARQL 1.1 query syntax.

To submit a SPARQL query from your code, issue an HTTP GET request to our endpoint:

http://opendatacommunities.org/sparql?query={URL-encoded query}

For example, to run this simple query...

SELECT * WHERE {<http://opendatacommunities.org/id/unitary-authority/cheshire-east> ?p ?o}

...and get the results as JSON, you could GET the following URL (note the .json extension):

http://opendatacommunities.org/sparql.json?query=SELECT+%2A+WHERE+%7B%3Chttp%3A%2F%2Fopendatacommunities.org%2Fid%2Funitary-authority%2Fcheshire-east%3E+%3Fp+%3Fo%7D

See below for details of SPARQL Results Formats and examples in a variety of languages (at the end of this section).

3.2 SPARQL Results formats

As with other aspects of our API, to get the data in different formats, you can use either format extensions or HTTP Accept headers.

The available formats depend on the type of SPARQL query. A SPARQL query can be one of four main forms: SELECT, ASK, CONSTRUCT or DESCRIBE.

Query Type Format Extension Accept Headers
SELECT xml .xml application/xml, application/sparql-results+xml
json .json application/json, application/sparql-results+json
text .txt, .text text/plain
csv .csv text/csv
ASK json .json application/json, application/sparql-results+json
xml .xml application/xml, application/sparql-results+json
text .txt, .text text/plain
CONSTRUCT RDF/XML .rdf application/rdf+xml
N-triples .nt, .txt, .text text/plain, application/n-triples
Turtle .ttl text/turtle
DESCRIBE RDF/XML .rdf application/rdf+xml
N-triples .nt, .txt, .text text/plain, application/n-triples
Turtle .ttl text/turtle

3.3 SPARQL Results Pagination

We will accept page and per_page query-string parameters for paginating the results of SELECT queries.

For requests made through the website (i.e. HTML format), the page size is defaulted to 20.

For requests to our sparql endpoint for data formats (i.e. non-HTML), there will be no defaults for these parameters (i.e. results are unlimited).

For SELECT queries, for convenience you can optionally pass the pagination parameters and we will use them to apply LIMIT and OFFSET clauses to the query. For other query types (i.e. DESCRIBE, CONSTRUCT, ASK), pagination like this doesn't make so much sense, so those parameters will be ignored.

Please also refer to the Response Size Limits section below, and the examples at the end of this section.

3.4 SPARQL Errors

If you make a SPARQL request with a malformed query in a data format (i.e. non-HTML), then we will respond with HTTP status 400, with a helpful message in the response.

Additionally, please note the Response Size Limits, which apply to all API calls, as well as the Errors section.

3.5 JSON-P

If you're requesting SPARQL results as JSON, you can additionally pass a callback parameter and the results will be wrapped in that function. This is useful for getting around cross-domain issues if you're running JavaScript on older browsers. (Please also see the Cross-Origin Resource Sharing section).

For example:

http://opendatacommunities.org/sparql.json?callback=myCallbackFunction&query=SELECT+%2A+WHERE+%7B%3Chttp%3A%2F%2Fopendatacommunities.org%2Fid%2Funitary-authority%2Fcheshire-east%3E+%3Fp+%3Fo%7D

Or to make a JSON-P request with jQuery, you can omit the callback parameter from the url and just set the dataType to jsonp.

queryUrl = 'http://opendatacommunities.org/sparql.json?query=SELECT+%2A+WHERE+%7B%3Chttp%3A%2F%2Fopendatacommunities.org%2Fid%2Funitary-authority%2Fcheshire-east%3E+%3Fp+%3Fo%7D' $.ajax({ dataType: 'jsonp', url: queryUrl, success: function(data) { // callback code here alert('success!'); } });

3.6 Use of Named graphs

3.6.1 Dataset Data

The data for each dataset is contained within a separate named graph. The dataset itself has a URI, for example

http://opendatacommunities.org/data/local-authorities

The web page for the dataset lists the named graph that contains the dataset, in this case

http://opendatacommunities.org/graph/local-authorities

The graph name for the dataset is contained in the dataset metadata, using a predicate called http://publishmydata.com/def/dataset#graph and can be obtained by a query like this:

SELECT ?graph WHERE { <http://opendatacommunities.org/data/local-authorities> <http://publishmydata.com/def/dataset#graph> ?graph }

3.6.2 Dataset Metadata

The metadata we store about the dataset itself (that is returned by dereferencing its URI), is stored its own separate graph, for example:

http://opendatacommunities.org/data/local-authorities/metadata

We also use named graphs for each concept scheme and ontology.

3.7 Parameter Substitution

You can parameterize your SPARQL by including %{tokens} in your queries, and providing values for the tokens on the url query string.

http://opendatacommunities.org/sparql?query=URL-encoded-SPARQL-query?token1=value-for-token1&token2=value-for-token2

Note that the following tokens are reserved and cannot be used as parameters for substitution.

  • controller
  • action
  • page
  • per_page
  • id
  • commit
  • utf8
  • query

3.8 Example: Using Ruby to request data from the SPARQL Endpoint

This sample Ruby makes a request to our SPARQL endpoint for all resources of type Unitary Authority (as JSON) and then puts the result in a Hash.

require 'rest-client' require 'json' query = 'SELECT * WHERE {?s a <http://opendatacommunities.org/def/local-government/UnitaryAuthority>}' url = "http://opendatacommunities.org/sparql.json" results_str = RestClient.get url, {:params => {:query => query}} results_hash = JSON.parse results_str results_array = results_hash["results"]["bindings"] puts "total number of results: #{results_array.length}"

Note: If you're doing a lot of work with RDF in Ruby, you might want to look at using Swirrl's open-source SPARQL ORM for Ruby, Tripod.

3.8 Example: Using JavaScript to request data from the SPARQL Endpoint

This example HTML page uses jQuery to make a request to our SPARQL endpoint for all resources of type Unitary Authority (as JSON) and log the results.

<!DOCTYPE html> <html> <head> <script src="http://code.jquery.com/jquery-1.9.1.min.js"></script> </head> <body> <script type="text/javascript"> var query = "SELECT * WHERE {?s a <http://opendatacommunities.org/def/local-government/UnitaryAuthority>}"; var url = "http://opendatacommunities.org/sparql.json?query="; url += encodeURIComponent(query); $.ajax({ dataType: 'json', url: url, success: function(data) { alert('success: ' + data.results.bindings.length + ' results'); console.log(data); } }); </script> </body> </html>

Note: See the Cross-Origin Resource Sharing section for a note about accessing data from from other domains.

4 - General

4.1 Response Size Limits

For all requests to our API, if the request issues a request to the database which causes more than 10MB of data to be returned, we will respond with HTTP status code 400, with the a message in the response body including the phrase Response too large. Note that full pre-canned dumps of all datasets will be available (in zipped n-triples format) at URLs defined in the dataset metadata.

4.2 Errors

Error type HTTP status code Notes
Response too large 400 We will include a text message in the response body including the phrase "Response too large."
SPARQL Syntax Error 400 We will include a text message in the response body with details of the error.
Resource Not Found 404 Returned if you request a resource or URL that doesn't exist
Not Acceptable 406 Returned if you request a non-supported data format
Unexpected Errors 500
Query Timeouts 503 The timeout for requesting data from our database will initially be set to 10 seconds.

4.3 Cross-Origin Resource Sharing (CORS)

Our web server is configured to allow access from all domains (by adding the following line to our nginx configuration):

add_header Access-Control-Allow-Origin "*";

This means that if you're writing JavaScript to request data from our server in to a web page hosted on another domain, your browser should check this header and allow it.

A Note about Browser Support for CORS:

Modern browsers (such as recent versions of Internet Explorer Firefox, Chrome and Safari) have full CORS support. It is not supported in Internet Explorer 6 and 7. Versions 8 & 9 of Internet Explorer offer limited support. If you need to support older browsers, consider making requests for data via SPARQL, with JSON-P.

4.4 Discontinued Datasets

A dataset can be marked as 'discontinued'. This approach is most often used in cases where a dataset uses an outdated vocabulary or outdated naming convention for URIs. This is similar to the concept of deprecation (in computer software).

A discontinued dataset is assigned a type of http://publishmydata.com/def/dataset#DeprecatedDataset as well as the usual http://publishmydata.com/def/dataset#Dataset. The discontinued-status is indicated on the list of datasets in the user interface and on the individual dataset page.

Optionally, a discontinued dataset may be replaced by another dataset. In this case a link to the new dataset appears on the dataset web page and the dataset metadata contains the triple:

<{discontinued dataset URI}> <http://purl.org/dc/terms/isReplacedBy> <{new dataset URI}> .

The contents of discontinued datasets are still available via SPARQL queries and other APIs. This allows us to update the way that data is represented without breaking external applications that use it. Discontinued datasets will generally be removed completely after some period of time. Information about planned deletion will be provided in the 'Further Information' section on the dataset web page.

4.5 API Keys

4.5.1 Obtaining an API key

You can register for an API key to obtain unthrottled access to our API, analytics, and announcements about upcoming API changes. Upon registering you will be logged in and taken to your profile page which displays your unique key. You should then use the API key with all requests to our API.

4.5.2 Using the API Key

To use the API key, supply it in your requests to our API (including for SPARQL queries) via either:

  • the X-PublishMyData-APIKey HTTP header, or
  • the api_key query string parameter

Note that some corporate or goverment networks may strip non-standard HTTP headers, so in these situations we recommend using the query string parameter instead.

4.5.3 Supplying the API Key in an HTTP header

For example, using the cURL command line tool to supply the API key in an HTTP header.

curl -H "X-PublishMyData-APIKey: <your-api-key-here>" <request-url>

Replace <your-api-key-here> with your actual API key. e.g.

curl -H "X-PublishMyData-APIKey: 1a3e2562d2466ecb9462ddfd723a4597" http://opendatacommunities.org/data/additional-affordable-dwellings.json

4.5.4 Supplying the API key in a query string parameter

For example, using the cURL command line tool to supply the API key in the custom HTTP header.

curl http://opendatacommunities.org/data/affordable-housing?api_key=<your-api-key-here>

Replace <your-api-key-here> with your actual API key. For example:

curl http://opendatacommunities.org/data/additional-affordable-dwellings.json?api_key=1a3e2562d2466ecb9462ddfd723a4597
  • This page is available as
  • HTML