project2017–2019full-stack engineeralgorithm engineer

Mixed-reality surgical navigation

One CT scene in mixed reality, shared by every headset and registered to the body.

Put two people in HoloLens headsets around one CT scan and each of them is really looking at a private copy. One of them turns the model. The other never finds out. From 2017 to 2019 I worked at Visual3D in China on a mixed-reality system for surgical navigation, which is that problem again with a real body and a robot in the room. I started as a full-stack engineer and wrote the back end, including the server that every device in this story talks through. I also built the medical scenes (and the operations on them) and published them to the headsets. Colleagues wrote the HoloLens client that shows them. In my second year, as an algorithm engineer, I designed and built the registration system, the part that puts the scene on the body. I also trained two segmentation networks there (and had a hand in the rendering).

Drawing of the whole setup: two HoloLens headsets, an iPad, a laptop showing a recorded view, a CT rendering of a chest, a cube covered in QR codes marking the world frame, a motion sensor, a robot arm, and a navigation tracker with its probe and workstation, each device with its own small axes. Blue lines run from every device to one tower server in the middle.
The whole room as I drew it in 2020, with every device's line to my server drawn in blue.

One scene for every device

I'll go one device at a time, each with the problem that brought it in. Sharing is the easy part. A cube with QR codes on its faces marks a world frame, each headset scans it, and the CT scene is aligned to the cube, so both headsets agree where the model sits. That holds until somebody moves it. One headset pans or rotates the scene, the other isn't aware of the change, and now you need a real-time communication system. Every device connects to one server, and a change goes up once and comes back down to everybody else. I had the server combine whatever came in and broadcast it at a fixed rate, so a room full of clients all editing at once couldn't block it.

Two headsets and the QR cube. The CT scene is drawn twice, rotated and faded; the first headset is labelled view changed and the second not aware of the change.
One headset changes the view and the other has no idea.
The same two headsets with a server added below. Arrows from the server labelled sync and update scene go to the CT scene and to the second headset.
Add a server and the second headset gets the update too.

Then the controls. Working the scene from inside a HoloLens is clumsy (some of the time, anyway), so an iPad comes in as a controller. A message like "control headset 1" then has to reach exactly that headset, which is why I gave the server an account system. Then recording. The headset's own recording is slow and the video poor, so a desktop recorder asks the server for a preview, and the headset's head pose reaches it through the server.

That server wasn't my last. Teeport, which I wrote later at SLAC, came out the same shape. It's a real-time hub too, only the clients are optimizers and the problems they solve instead of headsets.

Putting the scene on the body

For navigation the scene has to sit on the real body, which means knowing where the body is in the world frame, and a headset can't tell you that. So a navigation device comes in. You touch points on the body with a tracked probe and pair them with the same points in the CT, which ties the CT to the tracker. A QR code stuck on the tracker ties the tracker to the headsets. The registration system rests on two methods. If you know the pose of one marker, that pose alone gives you the whole transform. If all you have is points, you pair them up and fit the transform by least squares, solving the normal equations. The probe step is the second kind. Each registration is one arrow in the drawing below, and the scene is only right when every arrow along the way is.

Then the robot. Take a puncture, say. A robot is better at it than a hand as long as it knows the route, and the route is drawn in CT coordinates. With a tracked marker on its arm, a pose match ties the robot to the tracker, the tracker is already tied to the CT, and so the robot knows the route. The last problem is that bodies move. Registering again takes time, so a motion sensor (tied to the headsets by its own QR code) watches the body. When the pose changes, the sensor sends a scene update through the same server, the way a headset's edit goes.

Registration chain between six coordinate frames: motion sensor to HoloLens by scanning its QR code, HoloLens to world by scanning the QR cube, CT aligned to world, CT to navigation by point pairs with a tracked probe, navigation to HoloLens by scanning the QR code on the tracker, robot to navigation by pose match, and from those, robot to CT, so the robot knows the route Motion sensorHoloLensWorld RobotNavigationCT M · watches the bodyH · one per headsetW · the QR cube R · marker on the armN · tracks the probeC · scan and route scan its QR scan the cube point pairs pose match scan the QRon the tracker align the sceneto the cube so robot to CT, and the robot knows the route
Nobody measures the dashed arrow; it follows from the pose match and the point pairs.

The back end that shipped

In the drawings the server is one tower, and that's how I first wrote it. It was a single server with a WebSocket to every client. Redis held the scene in memory while people worked on it and PostgreSQL kept it afterwards. The models just sat on local disk. Then it grew. Each time it did I redrew the plan, six drawings from that one box to the last. Scene work and database work went to two servers, with a layer of APIs between them and the storage. Accounts got a login server of their own. A discovery server came next, and clients talked to it over UDP to find the rest. Then came a PACS server, so a scene could refer to its scans (DICOM series) in a PACS, and a license server for login to check against.

The last drawing is what shipped, and I wrote every service in it. It's flatter, too. Each job has its own service now, and a client first asks service discovery (still over UDP) where they all are. Keeping everybody's scene the same, the job the one server was written for, now runs through a message queue. An edit reaches the channel server, which puts it on NATS, and a scene state update server takes it off and writes the scene in Redis.

The back end as it shipped. Clients ask service discovery over UDP for the service URLs. Behind it are nine services. A scene edit goes to the channel server, which puts it on the NATS message queue; NATS forwards it to the scene state update server, which writes the scene in Redis. The scene state update server loads and saves scenes through the scene state API server, which also writes to Redis and goes through the scene server. The scene server uses the account, file and PACS servers. Account and file records are in PostgreSQL; the file server keeps models on disk, and the license server keeps licenses there; the PACS server reads DICOM series from PACS storage. Account, file and PACS all verify with the license server. Microservices ClientClientClientService discoveryChannelSceneNATSScene stateScene stateAccountFileLicensePACSRedisPostgreSQLDiskPACS join, leave, editmanage scenesmessage queueupdate serverAPI serverregister, log inupload, downloadlicensingreads DICOMthe scene in memoryaccount and file recordsmodels and licensesDICOM series over UDP, for the service URLs into the queueforwardwrite the scene load, saveassignrelationshipscene fileDICOM info account, file and PACSverify with it
The last drawing, the one that shipped, with the path of a scene edit in orange.

The server was JavaScript. Each client was in whatever its platform wanted, C# on the headsets and C++ on the navigation and robot workstations. It was a lot of code. When I counted in 2020, about three quarters of all the repositories I had ever worked on (SLAC and personal ones included) were Visual3D's. What I had to show for it in 2018 was three demos, and two of them used bone models.

Two networks I trained

The second year I also trained segmentation models, and the networks themselves aren't mine. TractSeg for tractography and nnU-Net for brain tumours both come from the MIC-DKFZ group, and both are U-Nets underneath (Ronneberger and colleagues). nnU-Net is basically a U-Net with a dataset auto-adapter, which saves you normalizing the data yourself. I trained and ran them. The tumour model had the bigger training set of the two. It trained for a bit over two weeks on a single V100. After that, a new scan took about a minute.

Axial MRI slice of a brain with reference fibre tracts drawn over it in green, red and blue, two bundles running front to back on either side of the ventricles.
Reference tracts on one axial slice.
The same axial slice with the tracts TractSeg predicted, the same two bundles in the same colours, slightly smoother.
What TractSeg predicted for the same slice, a little smoother.
Axial MRI slice of a brain with a large tumour on one side and the reference labels overlaid, a teal core inside a white ring and an orange region spreading behind it.
Reference tumour labels on another brain.
The same slice with the nnU-Net prediction overlaid, the teal core, white ring and orange region almost matching the reference.
The nnU-Net prediction for that slice.

Numbers are from the Work Overview slides I made in September 2020, and the Dice score is from my résumé. How the server broadcasts and the two ways to register come from my 2018 job talk at SLAC. The room drawings (their device pictures are product photos) and MRI slices are from the 2020 slides, and the registration diagram is redrawn from them. The back-end diagram is redrawn from the last of my architecture drawings, which I kept with those slides.