Bevy Picking | Tainted Coders

Often a game feature will want to have a player interact with some kind of object in the world, usually through clicking, dragging and dropping. Or inside of the UI knowing when the mouse is hovering over a button and when it is clicked.

This behavior is often called picking and is generic enough that bevy provides a built-in plugin to handle it for you: bevy_picking.

Picking is built around the concept of a Pointer which is a abstract representation of user input at a specific screen location.

Pointers can exist on 3 things:

Windows
Images
GPU Texture Views

Bevy's DefaultPlugins includes the DefaultPickingPlugins if you have the bevy_picking feature enabled which contains:

Plugin	Description
`InteractionPlugin`	Generates pointer events and handles event bubbling
`PickingPlugin`	Manages the picking state and generates higher level events
`PointerInputPlugin`	Mouse and touch events for picking pointers

Additionally, depending on what it is you want to interact with in your game, you will want at least one picking backend which is the specific implementation that feeds data to these 3 more generic plugins.

Picking backends

The 3 plugins listed above are there to handle all the hard parts for us. A picking backend only has one job: read PointerLocation components and produce PointerHits events.

An app can have multiple picking backends active at once.

A PointerHit event contains information about what entities a pointer is currently hitting:

pub struct PointerHits {
    pub pointer: PointerId,
    pub picks: Vec<(Entity, HitData)>,
    pub order: f32,
}

These PointerHits are then used by the generic picking plugins to produce higher level Pointer<E> events which we can react to with either an Observer or MessageReader.

Observers give us three important abilities above message readers, they allow:

Attaching event handlers to specific entities
Events that bubble up the entity hierarchy
Events of different types that can be called in a specific order.

Pointer events

The Pointer<E> events that bevy_picking defines fall into a few broad categories:

Type	Description
Hovering and movement
`Over`	Fires when a pointer crosses into the bounds of the target entity
`Move`	Fires while a pointer is moving over the target entity
`Out`	Fires when a pointer crosses out of the bounds of the target entity
Clicking and pressing
`Press`	Fires when a pointer button is pressed over the target entity
`Release`	Fires when a pointer button is released over the target entity
`Click`	Fires when a pointer sends a pointer pressed event followed by a pointer released event, with the same target entity for both events.
Dragging and dropping
`DragStart`	Fires when the target entity receives a pointer pressed event followed by a pointer move event
`Drag`	Fires while the target entity is being dragged
`DragEnd`	Fires when a pointer is dragging the target entity and a pointer released event is received
`DragEnter`	Fires when a pointer dragging the `dragged` entity enters the target entity
`DragOver`	Fires while the `dragged` entity is being dragged over the target entity
`DragDrop`	Fires when a pointer drops the `dropped` entity onto the target entity
`DragLeave`	Fires when a pointer dragging the `dragged` entity leaves the target entity.

Internally, Bevy is using a HoverMap which maps pointers to the entities they are hovering over. This is what determines which events to send and what state the components should be in.

Hovering is defined as a pointer hitting an entity (as reported by a picking backend) and no entities between it and the pointer blocking interactions.

Ignoring entities

Picking can be disabled for individual entities by adding Pickable::IGNORE

To make mesh picking entirely opt-in, set MeshPickingSettings::require_markers to true and add MeshPickingCamera and Pickable components to the desired camera and target entities.

Picking sprites

Sprites have a built-in backend that is already enabled. Sprites will not trigger pointer events by default. Instead we have to add a Pickable component to enable it.

use bevy::color::palettes::basic::{GREEN, YELLOW};

fn spawn_sprite(mut commands: Commands) {
  commands
    .spawn((
      Sprite::from_color(GREEN, Vec2::new(100., 100.)),
      Transform::from_xyz(0., 0., 0.),
      Pickable::default(),
    ))
    .observe(change_color_on_hover);
}

We then attach an observer to the sprite which lets us react to pointer events on it:

fn change_color_on_hover(
  hover: On<Pointer<Over>>,
  mut sprites: Query<&mut Sprite>,
) {
  let mut sprite = sprites.get_mut(hover.entity).unwrap();

  sprite.color = YELLOW.into();
}

Picking UI

Just like sprites, UI elements have a built-in picking backend that is already set up with the bevy_picking feature. The only difference is that UI elements are already pickable by default, so we don't need to add a Pickable component.

fn spawn_button(mut commands: Commands) {
  commands
    .spawn(Node {
      width: percent(100),
      height: percent(100),
      align_items: AlignItems::Center,
      justify_content: JustifyContent::Center,
      ..default()
    })
    .with_children(|parent| {
      parent.spawn(button()).observe(change_button_color_on_hover);
    });
}

fn button() -> impl Bundle {
  (
    Button,
    Node {
      width: px(150),
      height: px(65),
      border: UiRect::all(px(5)),
      justify_content: JustifyContent::Center,
      align_items: AlignItems::Center,
      ..default()
    },
    BorderColor::all(Color::WHITE),
    BackgroundColor(Color::BLACK),
    children![(
      Text::new("Button"),
      TextColor(Color::srgb(0.9, 0.9, 0.9)),
      TextShadow::default(),
    )],
  )
}

Note how we had to be careful to put the observer on the button specifically and not the layout node. Some UI nodes will be invisible but still be pickable by default. So its very easy to interact with them by accident.

Next, we can hook up an observer that changes the background color by inserting a new component of the same type (which overrides the previous one):

use bevy::color::palettes::basic::GREEN;

fn change_button_color_on_hover(
  hover: On<Pointer<Over>>,
  mut commands: Commands,
) {
  commands
    .entity(hover.entity)
    .insert(BackgroundColor(GREEN.into()));
}

Picking meshes

To pick meshes in 3D space, we can use one of the built-in backends: MeshPickingPlugin.

fn main() {
  App::new()
    .add_plugins((DefaultPlugins, MeshPickingPlugin))
    .add_systems(Startup, (setup_camera, spawn_cube))
    .run();
}

For this example we can add a system to spawn the camera and a cube at the center of the screen:

use bevy::color::palettes::basic::SILVER;

fn setup_camera(mut commands: Commands) {
  commands.spawn((
    MainCamera,
    Transform::from_xyz(8.0, 16.0, 8.0).looking_at(Vec3::ZERO, Vec3::Y),
    PointLight {
      shadows_enabled: true,
      intensity: 10_000_000.,
      range: 100.0,
      shadow_depth_bias: 0.2,
      ..default()
    },
  ));
}

fn spawn_cube(
  mut commands: Commands,
  mut meshes: ResMut<Assets<Mesh>>,
  mut materials: ResMut<Assets<StandardMaterial>>,
) {
  let cube = Cuboid::from_length(5.);

  commands
    .spawn((
      Mesh3d(meshes.add(cube)),
      MeshMaterial3d(materials.add(Color::from(SILVER))),
      Transform::from_xyz(0.0, 0.5, 0.0),
    ))
    .observe(rotate_on_drag);
}

The MeshPickingPlugin is the backend that will produce our PointerHits whenever our mouse interacts with a mesh.

We are going to react to these events with an observer that we setup on the cube we spawned. The observer will call our rotate_on_drag function when it matches the event we care about, in this case the Pointer<Drag> event.

fn rotate_on_drag(
  drag: On<Pointer<Drag>>,
  mut transforms: Query<&mut Transform>,
) {
  let mut transform = transforms.get_mut(drag.entity).unwrap();
  transform.rotate_y(drag.delta.x * 0.02);
  transform.rotate_x(drag.delta.y * 0.02);
}

When received by an observer, these events will always be wrapped by the Pointer<E> type, which contains additional metadata about the pointer event.

The picking pipeline

To get a whole picture of how picking works in Bevy, we can look at the overall pipeline:

We gather inputs and update pointers, generating PointerInput events
After inputs are generated they are collected and have the PointerLocation of each pointer updated
The picking backends read these PointerLocation components and produce PointerHits
The data from the backends is used to determine what each pointer is hovering over, producing a HoverMap. If one entity is in front of another, usually only the topmost one will be hovered.
Finally, higher level events are generated

Most of the action happens inside the pointer_events function which dispatches interaction events to the target entities.

Ordering of picking events

Within a single frame, the order of operations is:

Out -> DragLeave
DragEnger -> Over

Then any of the following can happen in any order:

For each movement: DragStart -> Drag -> DragOver -> Move
For each button press: Press or Click -> Release -> DragDrop -> DragEnd -> DragLeave
For each pointer cancellation: Cancel

Between frames things are managed by the interaction state machine:

When a pointer moves over the target: Over -> Move -> Out
When a pointer presses buttons on the target: Press -> Click -> Release
When a pointer drags the target: DragStart -> Drag -> DragEnd
When a pointer drags something over the target: DragEnter -> DragOver -> DragDrop -> DragLeave
When a pointer is canceled: No other events will follow the Cancel event for that pointer.