Overview

The evaluation network for LENS operates differently to the training network. During training, we use 2D temporally flattened images of event streams to learn the network weights.

During evaluation, however, we use the Sinabs spiking network to introduce constant leak integrate and fire (IAF) neurons into the LENS network.

In this case, we move away from static temporal representations of place to timebased activation of a spiking network. This works for both the onboard evaluation network with the SPECK™ using the development kits and the generated query images.

Time-based frame conversion

Generating events from static images for timebased evaluation is automatically done and based on the pixel intensity of query images:

Spikes are generated by comparing a uniformly random tensor against the input activation:

\[\begin{split}I_{t,h,w} = \begin{cases} 1 & \text{if } R_{t,h,w} < X_{h,w} \\ 0 & \text{otherwise} \end{cases}\end{split}\]

Here:

  • \(I_{t,h,w}\) is a boolean tensor with spatial dimensions \(h\) and \(w\) over \(t\) time windows, where 1 indicates a spike and 0 indicates no spike.

  • \(R_{t,h,w} \in [0, 1]\) is a uniformly random tensor.

  • \(X_{h,w}\) is the original input tensor.

Inference is simulated with \(\Delta t = 0.001\,\text{s}\) between spikes, i.e. 1000 timesteps will correspond to 1-second of inference time.

Constant leak integrate and fire neuron

In Sinabs, constant leak integrate and fire neurons are added as activation layers inbetween the linear LENS network weights to provide event-driven spiking activity.

# Define the inferencing forward pass
self.inference = nn.Sequential(
    self.conv,
    nn.ReLU(),
    nn.Flatten(),
    self.feature_layer.w,
    nn.ReLU(),
    self.output_layer.w,
)

nn.ReLU() layers are replaced with spiking layers during model conversion:

from sinabs.from_torch import from_model

self.sinabs_model = from_model(
                    self.inference.to(self.device),
                    input_shape=input_shape,
                    num_timesteps=self.timebin,
                    add_spiking_output=True
)

The subthreshold membrane potential dynamics are governed by the leaky integrator equation:

\[\tau \dot{v} = -v_{\text{leak}} + R \cdot (I_{\text{syn}} + I_{\text{bias}})\]

where:

  • \(\tau\) is the membrane time constant.

  • \(v\) is the membrane potential.

  • \(v_{\text{leak}}\) is the leak component of the membrane potential.

  • \(R\) is a constant resistance factor used to match units between potential and current.

  • \(I_{\text{syn}}\) is the weighted sum of all input synaptic currents.

  • \(I_{\text{bias}}\) is a constant bias current applied to the neuron.

Sequence matching

A crucial feature of LENS is sequence matching, which highlights sequential information in a distance matrix of queries and references whilst diminishing off-axis matches during evaluation.

Note

We recommend reading the original SeqSLAM paper that first introduced sequence matching.

Evaluation metrics

There are two main evaluation metrics used in LENS: Recall@K and the Precision-Recall curve. During evaluation, the distance matrix generated by LENS is compared against the ground truth binary matrix to identify the number of true positives (TP), false positives (FP).

Precision and recall is defined as:

\[\begin{split}precision = TP/(TP + FP)\\ recall = TP/GTP\end{split}\]

where GTP is the number of ground truth positives.

Recall@K analyzes the top K matches found for each query, where K = 1 only considers the top match whereas K = 25 considers the top 25 matches. The Precision-Recall curve measures the balance between the accuracy of correctly identifying place matches against the total number of correct matches.

Note

We highly recommend the VPR Tutorial for a complete guide on evaluation metrics.