GPUFrame

A GPUFrame represents a buffer on the GPU, often the result of a Frame that has been resized with the Resizer.

The GPUFrame exposes its underlying pixel buffer to the CPU via getPixelBuffer (a copy), or without copying via the platform accessors: getPixelByteBuffer() (a direct java.nio.ByteBuffer) on Android, getPixelData() (an NSData) on iOS.

The Resizer reuses one output buffer, so only one GPUFrame of a Resizer can be alive at a time: it must be disposed (or used in a use { } block) before the next Resizer.resize call.

resizer.resize(frame).use { resized ->
val pixels = resized.getPixelBuffer()
// run your ML model...
}

Properties

Link copied to clipboard
abstract val channelOrder: ChannelOrder?

The channel ordering of this GPUFrame.

Link copied to clipboard
abstract val dataType: DataType?

The scalar data type of this GPUFrame.

Link copied to clipboard
abstract val height: Int

The height of this GPUFrame, in pixels.

Link copied to clipboard
abstract val pixelLayout: PixelLayout?

How this GPUFrame's channels are laid out in memory:

Link copied to clipboard
abstract val width: Int

The width of this GPUFrame, in pixels.

Functions

Link copied to clipboard
open override fun close()

Same as dispose, to use a GPUFrame with use { }.

Link copied to clipboard
abstract fun dispose()

Disposes this GPUFrame, giving its buffer back to the Resizer so it can resize the next Frame. Calling it more than once is a no-op.

Link copied to clipboard
abstract fun getPixelBuffer(): ByteArray

Gets the pixels of this GPUFrame: width * height * 3 values of dataType (1 byte for DataType.INT8 and DataType.UINT8, 2 for DataType.FLOAT16, 4 for DataType.FLOAT32, in the platform's native byte order, little-endian on every supported device), arranged as pixelLayout describes.

Link copied to clipboard

Gets a direct ByteBuffer over this GPUFrame's mapped Vulkan output memory, without copying (the equivalent of VisionCamera's zero-copy getPixelBuffer() ArrayBuffer), in the native byte order. It can be passed as is to ML runtimes such as LiteRT (TensorFlow Lite) or ONNX Runtime.

Link copied to clipboard
fun GPUFrame.getPixelData(): NSData

Gets an NSData over this GPUFrame's shared Metal output buffer, without copying (the equivalent of VisionCamera's zero-copy getPixelBuffer() ArrayBuffer). Its bytes can be passed as is to ML runtimes, e.g. to a Core ML MLMultiArray(dataPointer:...).