A3 Hypermedia REST-API:

There and back again

Beginning: Fuzzy User Stories:

a2 features
innovative text collaboration
all that we dream of

Beginning: Challenging Technical Requirements:

distributed or external resources
versioning with branches/merges
complex semantic statements
polices how to updated statements if resource has new version
easy to extend/customize
scalable/good performance

Realisation: prototype used in productions

Realisation: conceptual open questions

good scalablity / performance ?
forever cachable Resources?
many References/backreferences? (Users, likes, images, comments, ...)
distributed or external resources?

Realisation: "hypermedia" api?

Server API provides machine and human readable defintion how to interact
Client explores API follwoing hypermedia links

Our API instead:

mixed with "Developers knows how to interact with the API"
missing features (like order properties, search, usable traversal, ...)
not used features (like generated UI based on resource sheets, ...)

Realisation: API for implementation not for Users

search queries instead instead of hypermedia api links (traversal)
needless coupling frontend/backend
complex data structure/api
no standards for data formats/types/semantic
missing/complex documentation
support substanced admin application
many many requests needed

users: our frontend, third party (Daten werden bleiben, implentierung wird wechseln) (mehrere anfragen von scientis, externen developers (resource type/sheet names are generated from python interface) (frontend need to know alot how to use the API)

Resource Data

{
"content_type": "adhocracy_mercator.resources.mercator.IMercatorProposalVersion",
"path": "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/VERSION_0000005/"
"data":
{
    "adhocracy_core.sheets.title.ITitle":
    {
        "title": "KITCHEN ON THE RUN. COOK - EAT - SHARE!"
    },
    "adhocracy_core.sheets.metadata.IMetadata":
    {
        "creation_date": "2015-04-11T10:35:04.593416+00:00",
        "deleted": "false",
        "creator": "https://.../api/principals/users/0005151/",
        ...
    },
    "adhocracy_core.sheets.comment.ICommentable":
    {
        "comments":
                ["https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000001/",
                 "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000002/",],
                 "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000003/",],
                "post_pool": "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/"
            }
        },
    ...
    }

Get comments with search:

versions = get \
    /process/resource_type=adhocracy_core.resources.merator.IProposalVersion&tag=last&elements=content
comment_versions_search = get \
    versions[0]."adhocracy_core.sheets.comment.IComment".post_pool + /resource_type=ICommentVersion\
    &elements=content&tag=last&adhocracy_core.sheets.comment.IComment:comments=\
    + versions[0]
comment_versions = comment_versions_search."adhocracy_core.sheets.pool.IPool".elements

=> works, only 2 requests, but frontend needs to know alot, backend implementation is no very scalable

Get comments with traversal:

process = get \
     /process/
item = process.elements[0]
version = get \
    item."adhocracy_core.sheets.tag.ITags".LAST
comment_versions_paths = proposalversion."adhocracy_core_sheets.comment.IComment".comments
comment_versions = [get comment_version_path[0] if it is last version, ...] How to do this??

=> works somehow, many request needed, frontend needs to know less, backend implementation is better scalable

Restart: API for Users (frontend, third-party, scientists)

hypermedia REST API (partly done)
simplify data structure / API
decouple frontend / backend (partly done)
better scalability / performance
distributed or external resources
autogenerated forms in frontend (partly done)
autogenerated documentation for human / machines
use standards

Simplify data structure

no resource sheets (not in backend implementation)
no 'data', field
native json types (string, bool, integer, float)
...

{
"title": "KITCHEN ON THE RUN. COOK - EAT - SHARE!"

"creation_date": "2015-04-11T10:35:04.593416+00:00",
"deleted": False,
"creator": "https://.../api/principals/users/0005151/",

"comments":
   ["https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000001/",
    "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000002/",],
    "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000003/",],

"comments_post_pool": "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/"
...

Simplify API

backreference only 'last' versions of resources
calculate 'comments_count', 'rates_count', ... in backend

{
"title": "KITCHEN ON THE RUN. COOK - EAT - SHARE!"

"creation_date": "2015-04-11T10:35:04.593416+00:00",
"deleted": False,
"creator": "https://.../api/principals/users/0005151/",

"comments_count": 1,
"comments": [
    "https://../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/comment_0000000/VERSION_0000003/"
    ],
"comments_post_pool": "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/"
...

Simplify API

reference to backrefernce listing instead of listing inline (example "comments", "likes")
remove post_pool property use reference to listing instead (example "comments", "likes")
pools may use batching to limit large listings (example users/comments/rates pools)

...
"comments_count": 1,
"comments": "https://.../api/mercator/KITCHEN_ONTHERUN.COOK-EAT-SHARE2589/comments/\
    ?resource_type=ICommentVersion&tag=last&adhocracy_core.sheets.comment..."
...

Simplify API: get comments with traversal

process = get /process/
item  = get process.elements[0]
proposal_version = get proposal.LAST
comment_versions = proposal_version.comments

or more improved:

process = get /process/
comment_versions  = process.elements[0].comments

=> easy

Simplify API

direct links to image scales

image = get proposalversion.image_thumbnail

no 'first_version_path'
simple API for Versioning?
...

Use Standard: JSON-Linked-Data

"JSON": Basic data format
"Linked Data": Graph of machine interpretable data across different Web sites.
augment exiting API possible

Use Standard: JSON-Linked-Data

References (single, set, list)
multiple graphs, sub-graph, embedded resources
Semantic Vocabularies to describe data for machines and humans
Semantic Web toolchain and Vocabularies usable:

metadata, persons, organisations, locations,
thesauri,  comments, ratings, actions, social activity/event stream,
blockchain, ...

Signed resources (to prove data is not modified)
Translation of resource properties

Resource Represenation: Compact

implicit mapping properties to semantic vocabularies, property types (@context link)

{
"@context": "https://api/contexts/ProposalVersion.jsonld",
"@type": ["ItemVersion", "ProbposalVersion"]
"@id": "https://.../api/mercator/Welcome_Back1194/",

"title": "KITCHEN ON THE RUN. COOK - EAT - SHARE!"

"creation_date": "2015-04-11T10:35:04.593416+00:00",
"deleted": False,
"creator": "https://.../api/principals/users/0005151/",
...

get @context:

{
"@context": {
    "ProposalVersion": "https://api/vocab#ProposalVersion",
    "title": {@id: "http://purl.org/dc/term/title/"
              @type: "http://ww.w3.org/2001/XMLSchema#string"},
    "deleted": {@id: "https://api/vocab#deleted"
                @type:"http://www.w3.org/2001/XMLSchema#bool"},
    "creation_date": {@id: "http://purl.org/dc/term/created/"
                      @type: "http://ww.w3.org/2001/XMLSchema#date"},
    "creator": {@id: "http://purl.org/dc/term/created/"
                @type: @id"},

}

property types describes the allowed values, resource types do not define allowed propoerties

Resource Represenation: Compact with namespaces

explicit mapping to vocabularies with namespace (example 'dc' = DubilCore)

{
"@context": "https://api/contexts/ProposalVersion.jsonld",
"@type": ["ItemVersion", "ProbposalVersion"]
"@id": "https://.../api/mercator/Welcome_Back1194/",

"dc:title": "KITCHEN ON THE RUN. COOK - EAT - SHARE!"

"dc:created": "2015-04-11T10:35:04.593416+00:00",
"deleted": False,
"dc:creator": "https://.../api/principals/users/0005151/",
...

get @context:

{
"@context": {
    "ProposalVersion": "https://api/vocab#ProbposalVersion",
    "dc": "http://purl.org/dc/term/title/"
    "dc:title": {@type: "http://ww.w3.org/2001/XMLSchema#string"},
    "deleted": {@id: "https://api/vocab#deleted"
                @type:"http://www.w3.org/2001/XMLSchema#bool"},
    "dc:created": {@type: "http://ww.w3.org/2001/XMLSchema#date"},
    "creator": {@type: @id},

}

Resource Representaion: Extended

Normalised data representation with semantic vocabularies and property types

{
"@type": ["https://api/vocab#ItemVersion", "https://api/vocab#ProbposalVersion"]
"@id": "https://../api/mercator/Welcome_Back1194/",

"http://purl.org/dc/term/title/": [{
          @type: "http://ww.w3.org/2001/XMLSchema#string",
          @value: "KITCHEN ON THE RUN. COOK - EAT - SHARE!"}],
"https://api/vocab#deleted": [{
          @type:"http://www.w3.org/2001/XMLSchema#bool",
          @value: False}],
"http://purl.org/dc/term/created/": [{
          @type: "http://ww.w3.org/2001/XMLSchema#date",
          @value: "2015-04-11T10:35:04.593416+00:00"}],
...

Use Standard: HYDRA

JSON-LD vocabulary for Hypermedia REST-APIs
Data Model with resource classes and depending properties
Collection/Partial-Collection types to list resources
extendable
Hydra console: browse API, generic operations

json-ld alone is only for semantic/value types. It has a open world concept != closed world like other standards. Thats why HYDRA adds classes (RDF-Schema like data model).

Describes client / server interaction:

operations: how to read, create, replace, delete, search resources
properties: required/readable/writable, expected data values
references may define operations for target resource
search query templates
server responses
UI information: order properties

General API-Documentation:

{
  "@context": "https://api/contexts/api.jsonld",
  "@type": "ApiDocumentation",
  "@id": "http://api/vocab",
  "title": ....
  "description": ...
  "supportedClass": [
    ... Classes known to be supported
  ],
  "possibleStatus": [
    ... Statuses that should be expected
  ]
}

...
"@type": "Class",
"@id": "https://api/vocab#Process",
"title": "Participation process",
"supportedProperty": [
  ... properties known to be supported by the class
  {
    "@type": "SupportedProperty",
    "property": "https://api/vocab#deleted",
    "required": true,
    "range": bool,
  }]
"supportedOperations": [
  ... possible operations
  {
    "@type": "edit_process",
    "method": "PUT",
    "expects": {
      "@id": "Process",
      "supportedProperty": [
          ... override property definitions
          ]
    "returns": "Process",
    "possibleStatus": [ ]... Statuses that should be expected
   }]

Properties with references as value may define the target operations:

...
"@id": "http://api/vocab#comments",
"@type": "Link",
"title": "Comments",
"range": "CommentsCollection",
"supportedOperation": [
  {
    "@type": "CreateCommentOperation",
    "method": "POST",
    "expects": "http://api/vocab/#Comment",
    "returns": "http://api/vocab/#Comment",
  }
  {
    "@type": "RetrieveCommentsOperation",
    "method": "GET",
    "expects": null,
    "returns": "CommentsCollection",
  }]
 ...

If no operations are available for the target resource, the link property is not shown.

The bright future:

hypermedia rest api (done, discussion and maybe extensions needed)
simplify data structure / API (mostly done, Versioning?)
decouple frontend / backend (done)
better scalability / performance (done, much less requests)
distributed or external resources (done, conceptually)
autogenerated forms frontend possible (done)
autogenerated documentation for human / machines (done)
use standards (done)

How to get there?

improve exiting API to make current projects work
add new API based on JSON-LD/Hydra
rewrite frontend for new projects

Appendix: Hypermedia REST API Links

JSON-Linked-Data (W3C specs)
HYDRA/JSON-Linked Data (W3C specs )
Collection+JSON
JSON HYPER-Schema (no operations)
JSON API
HAL
SIREN (extended HAL, with Resource classes)

JSON-LD and Why I Hate the Semantic Web
"JSON for Linked Data" slides
Code examples for all standards
How to Build Hypermedia APIs with JSON-LD and Hydra
Hydra console

Space	Forward
Right, Down, Page Down	Next slide
Left, Up, Page Up	Previous slide
P	Open presenter console
H	Toggle this help