Immersive Twins

Philippe Coval <purl.org/rzr> #LGM2020

lgm2020-full-opti.png

$ whois Phil Coval

  • Software Engineer from Rennes, France
  • OpenSource Contributor, Mozilla Rep (IoT)
  • Involved in industry's OSS:
    • Tizen (Intel), Yocto, IoTivity (Samsung)
  • Contact: https://purl.org/rzr @RzrFreeFr
    • Presentations, Videos, Social…
    • Currently available for cooperation

From "flat" reality to…

  • Validating concepts
    • 1950: "Experience Theater", Sensorama
    • 1980: Cyberspace : SciFi William Gibson
    • 1993: Doom FPS (native)
  • 3D online
    • 1995: VRML, X3D (browser plugin)
    • 2003: SecondLife (native)
    • 2006: WebGL (GPU in browser)
  • Devices & ubiquity
    • 2014: Cardboard (on mobile)
    • 2016: Pokemon Go (AR/ Overlay)
    • 2017: OpenXR: devices layer
    • 2018: WebXR (devices + apps)

Openness

  • OpenXR: to "Solve AR/VR Fragmentation"
    • Standard from Khronos Group
    • API for applications and Device layer
    • FLOSS: Monado, OpenHMD
  • WebXR: Web Browser API
    • to abstract devices for apps:
      • HMD, sensors (6 DoF), controllers
    • Cross devices/browsers:
    • API: navigator.xr

Play WebXR apps

  • From a VR/AR headset
    • supporting Firefox reality
    • or device's web-browser
    • Mobile with Card-box (or similar)
    • use orientation sensors
  • From a regular browser:

Build WebXR apps

  • Many frameworks:
    • WebGL: JS bindings to OpenGL (GPU)
    • Three.js: Scene on WebGL
    • A-Frame: based on Three.js
      • HTML for 3D Scene : "<a-box />"
    • Babylon.js: based on WebGL
  • glTF: GL Transmission Format
    • Compress assets (JPEG of 3D)
    • JSON structure
    • Tools: Blender3D

Web as a platform

  • Not only flat (HTML)
    • 3D Native (WebGL)
    • Immersive WebXR (Supersede WebVR)
    • Dynamic (JS), programmable (Services)
  • Transversal:
    • between 3D worlds
    • among people, cultures (social)
    • or even connected devices

Back to real reality

  • IoT: connected devices on the Internet
    • API for Sensors and actuators
  • Web Of Things: devices on the WWW
  • Digital Twin:
    • "Live" Replica of a physical entity (model)
  • Web Of Twins: A DT experiment on the web
    • Bind WoT to XR
    • Control WebThings using A-Frame

Robot and its "twin"

📺 📺

Mozilla WebThings

  • Mozilla's SmartHome platform
    • Gives control back to users
    • UI Dashboard: automation using rules
  • Extensible with addons
    • for new protocols or services
  • Simplified W3C WoT Description (in JSON)

Different perspectives

📺 📺

From Reality to XR:

  • IO: Sensors (or actuators) are served
    • WebThings API (RESTful or WebSockets)
  • Webthings are connected to gateway (in LAN)
    • Can be shared to the Internet
      • using a token (JWT)
  • WebXR application listen to webthing changes
    • update "twin" node accordingly
  • FLOSS: https://purl.org/aframe-webthing

Light example

📺 📺 📺

Mozilla's VR hubs

📺 📺 📺

Resources

Q&A or Extra demos ?

Extra: DT Robot demo

📺 📺

Video playback

📺 📺 📺 📺 📺 📺

More

Playlist

📺 📺 📺 📺

Created by Philippe Coval