Pixel eXtraction via Location tracking (PXL) is an additional Unity package built on Microsoft's Mixed Reality Toolkit 3 framework for extended reality (XR) applications.
The code fetches an image, creates a button behind the image that detects user touch via Unity cues, and converts 3D input to pixel space and estimates the corresponding pixel position. Images displayed as sprites aren't interactive by default, and this package aims to address that issue.
Make sure that your Unity project has MRTK 3 and required dependencies all installed before proceeding.
If you are new to Unity, follow Microsoft's setup instructions here: https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/getting-started/setting-up/setup-new-project
Make sure that you have your Unity scene configured for MRTK 3 as well using the instructions at this link: https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/getting-started/setting-up/setup-new-scene
Download this repo and unzip it. Navigate to the package manager under "Window" -> "Package Manager" in Unity. Select "Add package from disk". Find where you unzipped your package, and click the "manifest.json" file.
This package comes with various scripts that work together to get the pixel coordinate.
Start by finding the starter prefab "Interactive Image System" under Packages/PXL/Assets/Prefabs. Place that prefab into your scene.
| Script | Purpose |
|---|---|
| ImageGetter.cs | ImageGetter.cs is the script that makes the web request for your image. By default it is attached to the child game object under the Interactive Image System prefab. Specify the URL in the public field. You can also specify where you want the image and how you want it positioned in the other public fields "Image Position" and "Image Rotation". This class fetches the image in the Start() method. This class also will scale your image as higher quality images can be quite large in a Unity scene. |
| ButtonManager.cs | ButtonManager.cs is the script that creates the interactive panel behind the image. In the private method Notify(), the specific pixel that was touched is calculated (stored in the Vector2 "touchedPixel"). You don't have to do anything with this script. |
| HandTracker.cs | HandTracker.cs is the script that gets the hand position when the interaction is triggered. Attach this script to the Camera offset game object in the hierarchy (under MRTK XR Rig). |
Feel free to customize these scripts to fit your needs like changing how you fetch the image if you need some sort of authentication, want to change how the image is scaled, or if you need the pixel value for a certain application.
This code is provided under a BSD-3-Clause license and is not maintained. Feel free to alter this code.
See more under the License tab of this repo.