Bevy Picking

Clicking a point on your screen very different than clicking on a point inside your scene. We have to translate the 2D screen coordinates into a 3D ray and find where it hits something.

That's what picking is all about. Bevy tells us our PointerLocation and gives them to one of the picking backends. They then tell us what we are hitting.

Backends are plugins that take the PointerLocation components and produce PointerHits events that contain a Pointer and the entities they are hitting.

Bevy uses 3 key plugins to enable picking which are all enabled by default if you have the bevy_picking feature enabled:

Plugin	Description
`InteractionPlugin`	Generates pointer events and handles event bubbling
`PickingPlugin`	Sets up the core picking infrastructure and `PickingSettings`
`PointerInputPlugin`	Mouse and touch events for picking pointers

Additionally, depending on what it is you want to interact with in your game, you will want at least one picking backend which is the specific implementation that feeds data to these 3 more generic plugins.

Pointer

Pointers are spawned and despawned depending on user input. Mouse and Touch pointers are automatically spawned.

Every event that Bevy fires has 3 common elements:

#[derive(Clone, PartialEq, Debug, Reflect)]
pub struct Pointer {
  /// The pointer that triggered this event
  pub id: PointerId,
  /// The [`NormalizedRenderTarget`] associated with the pointer, usually a window.
  pub target: NormalizedRenderTarget,
  /// The position of the pointer in the `target`.
  pub position: Vec2
}

A pointer's target is a can be one of:

A window
An image
GPU Texture Views
None (for cameras that don't render to a color target)

These are embedded in a more concrete event:

#[derive(EntityEvent, Message, Clone, PartialEq, Debug, Reflect)]
pub struct PointerPress {
  /// The entity this pointer event happened for.
  pub entity: Entity,
  /// The pointer that triggered this event
  pub pointer: Pointer,
  /// Pointer button pressed to trigger this event.
  pub button: PointerButton,
  /// Information about the picking intersection.
  pub hit: HitData,
  /// Number of consecutive presses, starting at `1`.
  pub count: u8,
}

Picking backends

Backends are there to handle all the hard parts for us. They have only one job: Read PointerLocation components and produce PointerHits events.

The PointerHits event contains information about what entities a pointer is currently hitting:

pub struct PointerHits {
  pub pointer: PointerId,
  pub picks: Vec<(Entity, HitData)>,
  pub order: f32,
}

Each pick pairs an Entity with its HitData, which includes the camera, the depth of the hit, and optional backend-specific data that a backend can attach via HitData::extra and read back later with HitData::extra_as.

Backends that need a ray into the scene can read the RayMap resource, which bevy_picking builds for every camera and pointer, handling viewports and DPI for you.

These PointerHits are then used by the generic picking plugins to produce higher level pointer events such as PointerPress or PointerDrag, which we can react to with either an Observer or MessageReader.

Observers give us three important abilities above message readers, they allow:

Attaching event handlers to specific entities
Events that bubble up the entity hierarchy
Events of different types that can be called in a specific order.

An app can have multiple picking backends active at once.

Pointer events

The pointer events that bevy_picking defines fall into a few broad categories:

Type	Description
Hovering and movement
`PointerOver`	Fires when a pointer crosses into the bounds of the target entity. Unlike `PointerEnter`, this event bubbles up to all of the target's ancestors without restriction
`PointerEnter`	Fires when a pointer crosses into the bounds of the target entity. Unlike `PointerOver`, this event only bubbles up to ancestors that were not previously hovered by the pointer (like `mouseenter` on the web)
`PointerMove`	Fires while a pointer is moving over the target entity
`PointerLeave`	Fires when a pointer crosses out of the bounds of the target entity. Unlike `PointerOut`, this event only bubbles up to ancestors that are no longer hovered by the pointer (like `mouseleave` on the web)
`PointerOut`	Fires when a pointer crosses out of the bounds of the target entity. Unlike `PointerLeave`, this event bubbles up to all of the target's ancestors without restriction
Clicking and pressing
`PointerPress`	Fires when a pointer button is pressed over the target entity
`PointerRelease`	Fires when a pointer button is released over the target entity
`PointerClick`	Fires when a pointer sends a pointer pressed event followed by a pointer released event, with the same target entity for both events.
Scrolling
`PointerScroll`	Fires while a pointer is scrolling over the target entity
Dragging and dropping
`PointerDragStart`	Fires when the target entity receives a pointer pressed event followed by a pointer move event
`PointerDrag`	Fires while the target entity is being dragged
`PointerDragEnd`	Fires when a pointer is dragging the target entity and a pointer released event is received
`PointerDragEnter`	Fires when a pointer dragging the `dragged` entity enters the target entity
`PointerDragOver`	Fires while the `dragged` entity is being dragged over the target entity
`PointerDragDrop`	Fires when a pointer drops the `dropped` entity onto the target entity
`PointerDragLeave`	Fires when a pointer dragging the `dragged` entity leaves the target entity.
Cancellation
`PointerCancel`	Fires when a pointer is canceled, and its current interaction state is dropped

Internally, Bevy is using a HoverMap which maps pointers to the entities they are hovering over. This is what determines which events to send and what state the components should be in.

Hovering is defined as a pointer hitting an entity (as reported by a picking backend) with no entities between it and the pointer blocking interactions. By default an entity blocks everything behind it, but setting Pickable::should_block_lower to false lets the pointer hover entities behind it as well.

Ignoring entities

Picking can be disabled for individual entities by adding Pickable::IGNORE.

To make mesh picking entirely opt-in, set MeshPickingSettings::require_markers to true and add MeshPickingCamera and Pickable components to the desired camera and target entities.

Configuring picking

Two resources let you tune picking at runtime. PickingSettings can disable picking entirely, or turn off individual stages like input collection or hover updates:

use core::time::Duration;

PickingSettings {
  is_enabled: true,
  is_input_enabled: true,
  is_hover_enabled: true,
  is_window_picking_enabled: true,
  multi_click_interval: Duration::from_millis(500),
}

PointerInputSettings controls the built-in mouse and touch input independently:

PointerInputSettings {
  is_touch_enabled: true,
  is_mouse_enabled: true,
}

Because pointers aren't tied to any specific hardware, you can spawn your own pointer entity with a PointerId and drive it by writing PointerInput messages. This is how you would add a gamepad or VR cursor, for example.

As for the built-in backends, the sprite and UI backends are already enabled by DefaultPlugins, but MeshPickingPlugin has to be added to your app explicitly.

Picking sprites

Sprites have a built-in backend that is already enabled. Sprites will not trigger pointer events by default. Instead we have to add a Pickable component to enable it.

fn sprite() -> impl Scene {
  bsn! {
    Sprite { color: GREEN }
    Transform::from_xyz(0., 0., 0.)
    Pickable
    on(change_color_on_hover)
  }
}

fn main() {
  App::new()
    .add_plugins(DefaultPlugins)
    .add_systems(Startup, (setup_camera, sprite.spawn()))
    .run();
}

We then attach an observer to the sprite which lets us react to pointer events on it:

fn change_color_on_hover(
  hover: On<PointerOver>,
  mut sprites: Query<&mut Sprite>,
) {
  let mut sprite = sprites.get_mut(hover.entity).unwrap();

  sprite.color = YELLOW.into();
}

Picking UI

Just like sprites, UI elements have a built-in picking backend that is already set up with the bevy_picking feature. The only difference is that UI elements are already pickable by default, so we don't need to add a Pickable component.

fn button() -> impl Scene {
  bsn! {
    Button
    Node {
      width: px(150),
      height: px(65),
      border: UiRect::all(px(5)),
      justify_content: JustifyContent::Center,
      align_items: AlignItems::Center
    }
    BorderColor::all(Color::WHITE)
    BackgroundColor(Color::BLACK)
    Children [
      Text::new("Button")
      TextColor(Color::srgb(0.9, 0.9, 0.9))
      TextShadow::default()
    ]
  }
}

fn layout() -> impl Scene {
  bsn! {
    Node {
      width: percent(100),
      height: percent(100),
      align_items: AlignItems::Center,
      justify_content: JustifyContent::Center,
    }
    Children [
      @button()
      on(change_button_color_on_hover)
    ]
  }
}

fn main() {
  App::new()
    .add_plugins(DefaultPlugins)
    .add_systems(Startup, (setup_camera, layout.spawn()))
    .run();
}

Note how we had to be careful to put the observer on the button specifically and not the layout node. Container nodes still occupy space even when they have no visible background, so they are pickable by default just like the button. Putting the observer on the full-screen layout node would make it react to pointer events across the whole screen, not just the button.

Next, we can hook up an observer that changes the background color by inserting a new component of the same type (which overrides the previous one):

fn change_button_color_on_hover(
  hover: On<PointerOver>,
  mut commands: Commands,
) {
  commands
    .entity(hover.entity)
    .insert(BackgroundColor(GREEN.into()));
}

Picking meshes

To pick meshes in 3D space, we can use one of the built-in backends: MeshPickingPlugin.

fn main() {
  App::new()
    .add_plugins((DefaultPlugins, MeshPickingPlugin))
    .add_systems(Startup, (setup_camera, cube.spawn()))
    .run();
}

For this example we can add a system to spawn the camera and a cube at the center of the screen:

fn cube() -> impl Scene {
  bsn! {
    Mesh3d(asset_value(Cuboid::from_length(5.)))
    MeshMaterial3d<StandardMaterial>(asset_value(Color::WHITE))
    Transform::from_xyz(0.0, 0.5, 0.0)
    on(rotate_on_drag)
  }
}

fn setup_camera(mut commands: Commands) {
  commands.spawn((
    MainCamera,
    Transform::from_xyz(8.0, 16.0, 8.0).looking_at(Vec3::ZERO, Vec3::Y),
    PointLight {
      shadow_maps_enabled: true,
      intensity: 10_000_000.,
      range: 100.0,
      shadow_depth_bias: 0.2,
      ..default()
    },
  ));
}

The MeshPickingPlugin is the backend that will produce our PointerHits whenever our mouse interacts with a mesh.

We are going to react to these events with an observer that we setup on the cube we spawned. The observer will call our rotate_on_drag function when it matches the event we care about, in this case the PointerDrag event.

fn rotate_on_drag(
  drag: On<PointerDrag>,
  mut transforms: Query<&mut Transform>,
) {
  let mut transform = transforms.get_mut(drag.entity).unwrap();
  transform.rotate_y(drag.delta.x * 0.02);
  transform.rotate_x(drag.delta.y * 0.02);
}

When received by an observer, each of these events carries a pointer: Pointer field, which contains additional metadata about the pointer event.

Because the events are no longer generic in 0.20, you can write a handler that works for any pointer event using the PointerEvent trait:

fn on_pointer_event<E: PointerEvent>(event: On<E>) {
  // `event.pointer()` returns the `Pointer` metadata common to every event.
  info!("pointer {:?} at {}", event.pointer().id, event.pointer().position);
}

The picking pipeline

To get a whole picture of how picking works in Bevy, we can look at the overall pipeline:

We gather inputs and update pointers, generating PointerInput events
After inputs are generated they are collected and have the PointerLocation of each pointer updated
The picking backends read these PointerLocation components and produce PointerHits
The data from the backends is used to determine what each pointer is hovering over, producing a HoverMap. If one entity is in front of another, usually only the topmost one will be hovered.
Finally, higher level events are generated

Most of the action happens inside the pointer_events function which dispatches interaction events to the target entities. The generate_hovermap and update_interactions systems build the hover state that pointer_events reads.

Ordering of picking events

Within a single frame, the order of operations is:

PointerOut -> PointerLeave -> PointerDragLeave
PointerDragEnter -> PointerEnter -> PointerOver

Then any of the following can happen in any order:

For each movement: PointerDragStart -> PointerDrag -> PointerDragOver -> PointerMove
For each button press: PointerPress or PointerClick -> PointerRelease -> PointerDragDrop -> PointerDragEnd -> PointerDragLeave
For each pointer cancellation: PointerCancel

Between frames things are managed by the interaction state machine:

When a pointer moves over the target: PointerOver -> PointerEnter -> PointerMove -> PointerLeave -> PointerOut
When a pointer presses buttons on the target: PointerPress -> PointerClick -> PointerRelease
When a pointer drags the target: PointerDragStart -> PointerDrag -> PointerDragEnd
When a pointer drags something over the target: PointerDragEnter -> PointerDragOver -> PointerDragDrop -> PointerDragLeave
When a pointer is canceled: No other events will follow the PointerCancel event for that pointer.