Using Gemini for game art

2026-05-17 · gamedev, ai, art

My long time favorite game is Heroes of the Might and Magic II and III. I play both of them since their creation in the late 1990s and early 2000s, and I still enjoy them. I always wanted to create new units for those games, but my art skills are really bad. I managed to create smaller games with my brother where I supplied the programming and he did the art side, but this was always very limited in scope.

I really like the open source project fheroes2 which reimplements the engine in pretty readable C++ code and I wanted to create a simple mod for it adding a new unit based on the existing one.

Sprite Editor - Generate Monster

To do that, I built (well Claude did) a sprite editor which sends selected frames from the original sprite sheet to Gemini with custom background color and margin to ensure it is possible to slice it properly afterwards.

How the sprite editor works

The frames are packed into a single sheet first. Each sprite sits on a magenta background, picked because no game sprite uses that color, so it is easy to find and strip out again afterwards. The margin around each one leaves Gemini room to add wings or a cape beyond the original silhouette.

gemini_client.py › defaults 
GRID_GAP = 2  # pixels between frames (at native resolution)
DEFAULT_BG_COLOR = (255, 0, 255)  # Magenta — distinct from any game sprite color
DEFAULT_UPSCALE = 4
DEFAULT_MARGIN = 4  # pixels of padding around each sprite (at native resolution)

The sheet then goes to Gemini as an image-to-image edit. The prompt pins the result to the same size and frame layout so everything still lines up when it is sliced back, and the default safety filters are turned off, because skeletons, demons and dragons trip them constantly.

gemini_client.py › generate 
size_prompt = (
    f"{prompt} "
    f"Keep the exact same layout, spacing, and frame positions. "
    f"The output must be exactly {sheet.width}x{sheet.height} pixels."
)

# Loosen safety filters — fantasy monster sprites (skeletons, demons,
# dragons) trip the default thresholds too aggressively.
safety_settings = [
    types.SafetySetting(category=c, threshold="BLOCK_NONE")
    for c in (
        "HARM_CATEGORY_HARASSMENT",
        "HARM_CATEGORY_HATE_SPEECH",
        "HARM_CATEGORY_SEXUALLY_EXPLICIT",
        "HARM_CATEGORY_DANGEROUS_CONTENT",
        "HARM_CATEGORY_CIVIC_INTEGRITY",
    )
]

config = types.GenerateContentConfig(
    response_modalities=["IMAGE"],
    safety_settings=safety_settings,
)
if system_instruction:
    config.system_instruction = system_instruction

Slicing the sheet back into frames is the fiddly part. Gemini does not hand back a clean magenta background, it blends the edges towards it, so a plain color match leaves a colored fringe around every sprite. The editor matches the background by hue instead, which catches those blended pixels and clears them to transparent.

gemini_client.py › mask 
def _compute_bg_mask(cell_pixels: np.ndarray, bg_color: tuple[int, int, int]) -> np.ndarray:
    """Return a boolean mask of pixels that match bg_color (background + blended fringe).

    Uses hue-window detection when bg is saturated (catches Gemini's blended edges
    that drift towards the bg hue), falls back to RGB distance for low-saturation
    backgrounds (grey/white/black) where hue is unstable.
    """
    r, g, b = bg_color
    bg_h, bg_s, _bg_v = colorsys.rgb_to_hsv(r / 255.0, g / 255.0, b / 255.0)

    if bg_s < 0.15:
        # Low-saturation bg: hue is undefined, use RGB distance instead.
        rgb = cell_pixels[:, :, :3].astype(np.int16)
        target = np.array(bg_color, dtype=np.int16).reshape(1, 1, 3)
        return np.max(np.abs(rgb - target), axis=2) < 40

    rgb_img = Image.fromarray(cell_pixels[:, :, :3], "RGB")
    hsv = np.array(rgb_img.convert("HSV")).transpose(2, 0, 1).astype(np.float32)
    hue_deg = hsv[0] / 255.0 * 360.0
    sat = hsv[1] / 255.0

    target_deg = bg_h * 360.0
    diff = np.abs(hue_deg - target_deg)
    diff = np.minimum(diff, 360.0 - diff)  # circular distance
    return (diff < 30.0) & (sat > 0.2)

This process works pretty well, especially with manual tuning of the prompt and few attempts.

The gameplay video above shows the result of this process with additional high resolution and low resolution sprites generated this way.

The new units and Gemini workflow live on the FK/Azure-Dragon branch of my fheroes2 fork.