SparseControlNetModel

SparseControlNetModel is an implementation of ControlNet for AnimateDiff.

ControlNet was introduced in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.

The SparseCtrl version of ControlNet was introduced in SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models for achieving controlled generation in text-to-video diffusion models by Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai.

The abstract from the paper is:

The development of text-to-video (T2V), i.e., generating videos with a given text prompt, has been significantly advanced in recent years. However, relying solely on text prompts often results in ambiguous frame composition due to spatial uncertainty. The research community thus leverages the dense structure signals, e.g., per-frame depth/edge sequences, to enhance controllability, whose collection accordingly increases the burden of inference. In this work, we present SparseCtrl to enable flexible structure control with temporally sparse signals, requiring only one or a few inputs, as shown in Figure 1. It incorporates an additional condition encoder to process these sparse signals while leaving the pre-trained T2V model untouched. The proposed approach is compatible with various modalities, including sketches, depth maps, and RGB images, providing more practical control for video generation and promoting applications such as storyboarding, depth rendering, keyframe animation, and interpolation. Extensive experiments demonstrate the generalization of SparseCtrl on both original and personalized T2V generators. Codes and models will be publicly available at this https URL.

Example for loading SparseControlNetModel

import mindspore as ms
from mindone.diffusers import SparseControlNetModel

# fp32 variant in float16
# 1. Scribble checkpoint
controlnet = SparseControlNetModel.from_pretrained("guoyww/animatediff-sparsectrl-scribble", mindspore_dtype=ms.float16)

# 2. RGB checkpoint
controlnet = SparseControlNetModel.from_pretrained("guoyww/animatediff-sparsectrl-rgb", mindspore_dtype=ms.float16)

# For loading fp16 variant, pass `variant="fp16"` as an additional parameter

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetModel

Bases: ModelMixin, ConfigMixin, FromOriginalModelMixin

A SparseControlNet model as described in SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models.

PARAMETER	DESCRIPTION
in_channels	The number of channels in the input sample. TYPE: `int`, defaults to 4 DEFAULT: 4
conditioning_channels	The number of input channels in the controlnet conditional embedding module. If concat_condition_embedding is True, the value provided here is incremented by 1. TYPE: `int`, defaults to 4 DEFAULT: 4
flip_sin_to_cos	Whether to flip the sin to cos in the time embedding. TYPE: `bool`, defaults to `True` DEFAULT: True
freq_shift	The frequency shift to apply to the time embedding. TYPE: `int`, defaults to 0 DEFAULT: 0
down_block_types	The tuple of downsample blocks to use. TYPE: `tuple[str]`, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")` DEFAULT: ('CrossAttnDownBlockMotion', 'CrossAttnDownBlockMotion', 'CrossAttnDownBlockMotion', 'DownBlockMotion')
only_cross_attention	TYPE: `Union[bool, Tuple[bool]]`, defaults to `False` DEFAULT: False
block_out_channels	The tuple of output channels for each block. TYPE: `tuple[int]`, defaults to `(320, 640, 1280, 1280)` DEFAULT: (320, 640, 1280, 1280)
layers_per_block	The number of layers per block. TYPE: `int`, defaults to 2 DEFAULT: 2
downsample_padding	The padding to use for the downsampling convolution. TYPE: `int`, defaults to 1 DEFAULT: 1
mid_block_scale_factor	The scale factor to use for the mid block. TYPE: `float`, defaults to 1 DEFAULT: 1
act_fn	The activation function to use. TYPE: `str`, defaults to "silu" DEFAULT: 'silu'
norm_num_groups	The number of groups to use for the normalization. If None, normalization and activation layers is skipped in post-processing. TYPE: `int`, *optional*, defaults to 32 DEFAULT: 32
norm_eps	The epsilon to use for the normalization. TYPE: `float`, defaults to 1e-5 DEFAULT: 1e-05
cross_attention_dim	The dimension of the cross attention features. TYPE: `int`, defaults to 1280 DEFAULT: 768
transformer_layers_per_block	The number of transformer blocks of type [~models.attention.BasicTransformerBlock]. Only relevant for [~models.unet_2d_blocks.CrossAttnDownBlock2D], [~models.unet_2d_blocks.CrossAttnUpBlock2D], [~models.unet_2d_blocks.UNetMidBlock2DCrossAttn]. TYPE: `int` or `Tuple[int]`, *optional*, defaults to 1 DEFAULT: 1
transformer_layers_per_mid_block	The number of transformer layers to use in each layer in the middle block. TYPE: `int` or `Tuple[int]`, *optional*, defaults to 1 DEFAULT: None
attention_head_dim	The dimension of the attention heads. TYPE: `int` or `Tuple[int]`, defaults to 8 DEFAULT: 8
num_attention_heads	The number of heads to use for multi-head attention. TYPE: `int` or `Tuple[int]`, *optional* DEFAULT: None
use_linear_projection	TYPE: `bool`, defaults to `False` DEFAULT: False
upcast_attention	TYPE: `bool`, defaults to `False` DEFAULT: False
resnet_time_scale_shift	Time scale shift config for ResNet blocks (see ResnetBlock2D). Choose from default or scale_shift. TYPE: `str`, defaults to `"default"` DEFAULT: 'default'
conditioning_embedding_out_channels	The tuple of output channel for each block in the conditioning_embedding layer. TYPE: `Tuple[int]`, defaults to `(16, 32, 96, 256)` DEFAULT: (16, 32, 96, 256)
global_pool_conditions	TODO(Patrick) - unused parameter TYPE: `bool`, defaults to `False` DEFAULT: False
controlnet_conditioning_channel_order	TYPE: `str`, defaults to `rgb` DEFAULT: 'rgb'
motion_max_seq_length	The maximum sequence length to use in the motion module. TYPE: `int`, defaults to `32` DEFAULT: 32
motion_num_attention_heads	The number of heads to use in each attention layer of the motion module. TYPE: `int` or `Tuple[int]`, defaults to `8` DEFAULT: 8
concat_conditioning_mask	TYPE: `bool`, defaults to `True` DEFAULT: True
use_simplified_condition_embedding	TYPE: `bool`, defaults to `True` DEFAULT: True

Source code in mindone/diffusers/models/controlnet_sparsectrl.py

class SparseControlNetModel(ModelMixin, ConfigMixin, FromOriginalModelMixin):
    """
    A SparseControlNet model as described in [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion
    Models](https://arxiv.org/abs/2311.16933).

    Args:
        in_channels (`int`, defaults to 4):
            The number of channels in the input sample.
        conditioning_channels (`int`, defaults to 4):
            The number of input channels in the controlnet conditional embedding module. If
            `concat_condition_embedding` is True, the value provided here is incremented by 1.
        flip_sin_to_cos (`bool`, defaults to `True`):
            Whether to flip the sin to cos in the time embedding.
        freq_shift (`int`, defaults to 0):
            The frequency shift to apply to the time embedding.
        down_block_types (`tuple[str]`, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`):
            The tuple of downsample blocks to use.
        only_cross_attention (`Union[bool, Tuple[bool]]`, defaults to `False`):
        block_out_channels (`tuple[int]`, defaults to `(320, 640, 1280, 1280)`):
            The tuple of output channels for each block.
        layers_per_block (`int`, defaults to 2):
            The number of layers per block.
        downsample_padding (`int`, defaults to 1):
            The padding to use for the downsampling convolution.
        mid_block_scale_factor (`float`, defaults to 1):
            The scale factor to use for the mid block.
        act_fn (`str`, defaults to "silu"):
            The activation function to use.
        norm_num_groups (`int`, *optional*, defaults to 32):
            The number of groups to use for the normalization. If None, normalization and activation layers is skipped
            in post-processing.
        norm_eps (`float`, defaults to 1e-5):
            The epsilon to use for the normalization.
        cross_attention_dim (`int`, defaults to 1280):
            The dimension of the cross attention features.
        transformer_layers_per_block (`int` or `Tuple[int]`, *optional*, defaults to 1):
            The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for
            [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.CrossAttnUpBlock2D`],
            [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`].
        transformer_layers_per_mid_block (`int` or `Tuple[int]`, *optional*, defaults to 1):
            The number of transformer layers to use in each layer in the middle block.
        attention_head_dim (`int` or `Tuple[int]`, defaults to 8):
            The dimension of the attention heads.
        num_attention_heads (`int` or `Tuple[int]`, *optional*):
            The number of heads to use for multi-head attention.
        use_linear_projection (`bool`, defaults to `False`):
        upcast_attention (`bool`, defaults to `False`):
        resnet_time_scale_shift (`str`, defaults to `"default"`):
            Time scale shift config for ResNet blocks (see `ResnetBlock2D`). Choose from `default` or `scale_shift`.
        conditioning_embedding_out_channels (`Tuple[int]`, defaults to `(16, 32, 96, 256)`):
            The tuple of output channel for each block in the `conditioning_embedding` layer.
        global_pool_conditions (`bool`, defaults to `False`):
            TODO(Patrick) - unused parameter
        controlnet_conditioning_channel_order (`str`, defaults to `rgb`):
        motion_max_seq_length (`int`, defaults to `32`):
            The maximum sequence length to use in the motion module.
        motion_num_attention_heads (`int` or `Tuple[int]`, defaults to `8`):
            The number of heads to use in each attention layer of the motion module.
        concat_conditioning_mask (`bool`, defaults to `True`):
        use_simplified_condition_embedding (`bool`, defaults to `True`):
    """

    _supports_gradient_checkpointing = True

    @register_to_config
    def __init__(
        self,
        in_channels: int = 4,
        conditioning_channels: int = 4,
        flip_sin_to_cos: bool = True,
        freq_shift: int = 0,
        down_block_types: Tuple[str, ...] = (
            "CrossAttnDownBlockMotion",
            "CrossAttnDownBlockMotion",
            "CrossAttnDownBlockMotion",
            "DownBlockMotion",
        ),
        only_cross_attention: Union[bool, Tuple[bool]] = False,
        block_out_channels: Tuple[int, ...] = (320, 640, 1280, 1280),
        layers_per_block: int = 2,
        downsample_padding: int = 1,
        mid_block_scale_factor: float = 1,
        act_fn: str = "silu",
        norm_num_groups: Optional[int] = 32,
        norm_eps: float = 1e-5,
        cross_attention_dim: int = 768,
        transformer_layers_per_block: Union[int, Tuple[int, ...]] = 1,
        transformer_layers_per_mid_block: Optional[Union[int, Tuple[int]]] = None,
        temporal_transformer_layers_per_block: Union[int, Tuple[int, ...]] = 1,
        attention_head_dim: Union[int, Tuple[int, ...]] = 8,
        num_attention_heads: Optional[Union[int, Tuple[int, ...]]] = None,
        use_linear_projection: bool = False,
        upcast_attention: bool = False,
        resnet_time_scale_shift: str = "default",
        conditioning_embedding_out_channels: Optional[Tuple[int, ...]] = (16, 32, 96, 256),
        global_pool_conditions: bool = False,
        controlnet_conditioning_channel_order: str = "rgb",
        motion_max_seq_length: int = 32,
        motion_num_attention_heads: int = 8,
        concat_conditioning_mask: bool = True,
        use_simplified_condition_embedding: bool = True,
    ):
        super().__init__()
        self.use_simplified_condition_embedding = use_simplified_condition_embedding

        # If `num_attention_heads` is not defined (which is the case for most models)
        # it will default to `attention_head_dim`. This looks weird upon first reading it and it is.
        # The reason for this behavior is to correct for incorrectly named variables that were introduced
        # when this library was created. The incorrect naming was only discovered much later in
        # https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131
        # Changing `attention_head_dim` to `num_attention_heads` for 40,000+ configurations is too backwards breaking
        # which is why we correct for the naming here.
        num_attention_heads = num_attention_heads or attention_head_dim

        # Check inputs
        if len(block_out_channels) != len(down_block_types):
            raise ValueError(
                f"Must provide the same number of `block_out_channels` as `down_block_types`.`block_out_channels`:"
                f"{block_out_channels}. `down_block_types`: {down_block_types}."
            )

        if not isinstance(only_cross_attention, bool) and len(only_cross_attention) != len(down_block_types):
            raise ValueError(
                f"Must provide the same number of `only_cross_attention` as `down_block_types`. `only_cross_attention`:"
                f"{only_cross_attention}. `down_block_types`: {down_block_types}."
            )

        if not isinstance(num_attention_heads, int) and len(num_attention_heads) != len(down_block_types):
            raise ValueError(
                f"Must provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`:"
                f"{num_attention_heads}. `down_block_types`: {down_block_types}."
            )

        if isinstance(transformer_layers_per_block, int):
            transformer_layers_per_block = [transformer_layers_per_block] * len(down_block_types)
        if isinstance(temporal_transformer_layers_per_block, int):
            temporal_transformer_layers_per_block = [temporal_transformer_layers_per_block] * len(down_block_types)

        # input
        conv_in_kernel = 3
        conv_in_padding = (conv_in_kernel - 1) // 2
        self.conv_in = nn.Conv2d(
            in_channels,
            block_out_channels[0],
            kernel_size=conv_in_kernel,
            padding=conv_in_padding,
            pad_mode="pad",
            has_bias=True,
        )

        if concat_conditioning_mask:
            conditioning_channels = conditioning_channels + 1

        self.concat_conditioning_mask = concat_conditioning_mask

        # control net conditioning embedding
        if use_simplified_condition_embedding:
            self.controlnet_cond_embedding = nn.Conv2d(
                conditioning_channels,
                block_out_channels[0],
                kernel_size=3,
                pad_mode="pad",
                padding=1,
                has_bias=True,
            )
        else:
            self.controlnet_cond_embedding = SparseControlNetConditioningEmbedding(
                conditioning_embedding_channels=block_out_channels[0],
                block_out_channels=conditioning_embedding_out_channels,
                conditioning_channels=conditioning_channels,
            )

        # time
        time_embed_dim = block_out_channels[0] * 4
        self.time_proj = Timesteps(block_out_channels[0], flip_sin_to_cos, freq_shift)
        timestep_input_dim = block_out_channels[0]

        self.time_embedding = TimestepEmbedding(
            timestep_input_dim,
            time_embed_dim,
            act_fn=act_fn,
        )

        self.down_blocks = []
        self.controlnet_down_blocks = []

        if isinstance(cross_attention_dim, int):
            cross_attention_dim = (cross_attention_dim,) * len(down_block_types)

        if isinstance(only_cross_attention, bool):
            only_cross_attention = [only_cross_attention] * len(down_block_types)

        if isinstance(attention_head_dim, int):
            attention_head_dim = (attention_head_dim,) * len(down_block_types)

        if isinstance(num_attention_heads, int):
            num_attention_heads = (num_attention_heads,) * len(down_block_types)

        if isinstance(motion_num_attention_heads, int):
            motion_num_attention_heads = (motion_num_attention_heads,) * len(down_block_types)

        # down
        output_channel = block_out_channels[0]

        controlnet_block = nn.Conv2d(output_channel, output_channel, kernel_size=1, has_bias=True, pad_mode="same")
        self.controlnet_down_blocks.append(controlnet_block)

        for i, down_block_type in enumerate(down_block_types):
            input_channel = output_channel
            output_channel = block_out_channels[i]
            is_final_block = i == len(block_out_channels) - 1

            if down_block_type == "CrossAttnDownBlockMotion":
                down_block = CrossAttnDownBlockMotion(
                    in_channels=input_channel,
                    out_channels=output_channel,
                    temb_channels=time_embed_dim,
                    dropout=0,
                    num_layers=layers_per_block,
                    transformer_layers_per_block=transformer_layers_per_block[i],
                    resnet_eps=norm_eps,
                    resnet_time_scale_shift=resnet_time_scale_shift,
                    resnet_act_fn=act_fn,
                    resnet_groups=norm_num_groups,
                    resnet_pre_norm=True,
                    num_attention_heads=num_attention_heads[i],
                    cross_attention_dim=cross_attention_dim[i],
                    add_downsample=not is_final_block,
                    dual_cross_attention=False,
                    use_linear_projection=use_linear_projection,
                    only_cross_attention=only_cross_attention[i],
                    upcast_attention=upcast_attention,
                    temporal_num_attention_heads=motion_num_attention_heads[i],
                    temporal_max_seq_length=motion_max_seq_length,
                    temporal_transformer_layers_per_block=temporal_transformer_layers_per_block[i],
                    temporal_double_self_attention=False,
                )
            elif down_block_type == "DownBlockMotion":
                down_block = DownBlockMotion(
                    in_channels=input_channel,
                    out_channels=output_channel,
                    temb_channels=time_embed_dim,
                    dropout=0,
                    num_layers=layers_per_block,
                    resnet_eps=norm_eps,
                    resnet_time_scale_shift=resnet_time_scale_shift,
                    resnet_act_fn=act_fn,
                    resnet_groups=norm_num_groups,
                    resnet_pre_norm=True,
                    add_downsample=not is_final_block,
                    temporal_num_attention_heads=motion_num_attention_heads[i],
                    temporal_max_seq_length=motion_max_seq_length,
                    temporal_transformer_layers_per_block=temporal_transformer_layers_per_block[i],
                    temporal_double_self_attention=False,
                )
            else:
                raise ValueError(
                    "Invalid `block_type` encountered. Must be one of `CrossAttnDownBlockMotion` or `DownBlockMotion`"
                )

            self.down_blocks.append(down_block)

            for _ in range(layers_per_block):
                controlnet_block = nn.Conv2d(output_channel, output_channel, kernel_size=1, has_bias=True)
                self.controlnet_down_blocks.append(controlnet_block)

            if not is_final_block:
                controlnet_block = nn.Conv2d(output_channel, output_channel, kernel_size=1, has_bias=True)
                self.controlnet_down_blocks.append(controlnet_block)

        self.down_blocks = nn.CellList(self.down_blocks)
        self.controlnet_down_blocks = nn.CellList(self.controlnet_down_blocks)

        # mid
        mid_block_channels = block_out_channels[-1]

        controlnet_block = nn.Conv2d(mid_block_channels, mid_block_channels, kernel_size=1, has_bias=True)
        self.controlnet_mid_block = controlnet_block

        if transformer_layers_per_mid_block is None:
            transformer_layers_per_mid_block = (
                transformer_layers_per_block[-1] if isinstance(transformer_layers_per_block[-1], int) else 1
            )

        self.mid_block = UNetMidBlock2DCrossAttn(
            in_channels=mid_block_channels,
            temb_channels=time_embed_dim,
            dropout=0,
            num_layers=1,
            transformer_layers_per_block=transformer_layers_per_mid_block,
            resnet_eps=norm_eps,
            resnet_time_scale_shift=resnet_time_scale_shift,
            resnet_act_fn=act_fn,
            resnet_groups=norm_num_groups,
            resnet_pre_norm=True,
            num_attention_heads=num_attention_heads[-1],
            output_scale_factor=mid_block_scale_factor,
            cross_attention_dim=cross_attention_dim[-1],
            dual_cross_attention=False,
            use_linear_projection=use_linear_projection,
            upcast_attention=upcast_attention,
            attention_type="default",
        )

    @classmethod
    def from_unet(
        cls,
        unet: UNet2DConditionModel,
        controlnet_conditioning_channel_order: str = "rgb",
        conditioning_embedding_out_channels: Optional[Tuple[int, ...]] = (16, 32, 96, 256),
        load_weights_from_unet: bool = True,
        conditioning_channels: int = 3,
    ) -> "SparseControlNetModel":
        r"""
        Instantiate a [`SparseControlNetModel`] from [`UNet2DConditionModel`].

        Parameters:
            unet (`UNet2DConditionModel`):
                The UNet model weights to copy to the [`SparseControlNetModel`]. All configuration options are also
                copied where applicable.
        """
        transformer_layers_per_block = (
            unet.config.transformer_layers_per_block if "transformer_layers_per_block" in unet.config else 1
        )
        down_block_types = unet.config.down_block_types

        for i in range(len(down_block_types)):
            if "CrossAttn" in down_block_types[i]:
                down_block_types[i] = "CrossAttnDownBlockMotion"
            elif "Down" in down_block_types[i]:
                down_block_types[i] = "DownBlockMotion"
            else:
                raise ValueError("Invalid `block_type` encountered. Must be a cross-attention or down block")

        controlnet = cls(
            in_channels=unet.config.in_channels,
            conditioning_channels=conditioning_channels,
            flip_sin_to_cos=unet.config.flip_sin_to_cos,
            freq_shift=unet.config.freq_shift,
            down_block_types=unet.config.down_block_types,
            only_cross_attention=unet.config.only_cross_attention,
            block_out_channels=unet.config.block_out_channels,
            layers_per_block=unet.config.layers_per_block,
            downsample_padding=unet.config.downsample_padding,
            mid_block_scale_factor=unet.config.mid_block_scale_factor,
            act_fn=unet.config.act_fn,
            norm_num_groups=unet.config.norm_num_groups,
            norm_eps=unet.config.norm_eps,
            cross_attention_dim=unet.config.cross_attention_dim,
            transformer_layers_per_block=transformer_layers_per_block,
            attention_head_dim=unet.config.attention_head_dim,
            num_attention_heads=unet.config.num_attention_heads,
            use_linear_projection=unet.config.use_linear_projection,
            upcast_attention=unet.config.upcast_attention,
            resnet_time_scale_shift=unet.config.resnet_time_scale_shift,
            conditioning_embedding_out_channels=conditioning_embedding_out_channels,
            controlnet_conditioning_channel_order=controlnet_conditioning_channel_order,
        )

        if load_weights_from_unet:
            ms.load_param_into_net(controlnet.conv_in, unet.conv_in.parameters_dict())
            ms.load_param_into_net(controlnet.time_proj, unet.time_proj.parameters_dict())
            ms.load_param_into_net(controlnet.time_embedding, unet.time_embedding.parameters_dict())
            ms.load_param_into_net(controlnet.down_blocks, unet.down_blocks.parameters_dict())
            ms.load_param_into_net(controlnet.mid_block, unet.mid_block.parameters_dict())

        return controlnet

    @property
    # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.attn_processors
    def attn_processors(self) -> Dict[str, AttentionProcessor]:
        r"""
        Returns:
            `dict` of attention processors: A dictionary containing all attention processors used in the model with
            indexed by its weight name.
        """
        # set recursively
        processors = {}

        def fn_recursive_add_processors(name: str, module: nn.Cell, processors: Dict[str, AttentionProcessor]):
            if hasattr(module, "get_processor"):
                processors[f"{name}.processor"] = module.get_processor()

            for sub_name, child in module.name_cells().items():
                fn_recursive_add_processors(f"{name}.{sub_name}", child, processors)

            return processors

        for name, module in self.name_cells().items():
            fn_recursive_add_processors(name, module, processors)

        return processors

    # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_attn_processor
    def set_attn_processor(self, processor: Union[AttentionProcessor, Dict[str, AttentionProcessor]]):
        r"""
        Sets the attention processor to use to compute attention.

        Parameters:
            processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`):
                The instantiated processor class or a dictionary of processor classes that will be set as the processor
                for **all** `Attention` layers.

                If `processor` is a dict, the key needs to define the path to the corresponding cross attention
                processor. This is strongly recommended when setting trainable attention processors.

        """
        count = len(self.attn_processors.keys())

        if isinstance(processor, dict) and len(processor) != count:
            raise ValueError(
                f"A dict of processors was passed, but the number of processors {len(processor)} does not match the"
                f" number of attention layers: {count}. Please make sure to pass {count} processor classes."
            )

        def fn_recursive_attn_processor(name: str, module: nn.Cell, processor):
            if hasattr(module, "set_processor"):
                if not isinstance(processor, dict):
                    module.set_processor(processor)
                else:
                    module.set_processor(processor.pop(f"{name}.processor"))

            for sub_name, child in module.name_cells().items():
                fn_recursive_attn_processor(f"{name}.{sub_name}", child, processor)

        for name, module in self.name_cells().items():
            fn_recursive_attn_processor(name, module, processor)

    # Copied from diffusers.models.unets.unet_2d_condition.UNet2DConditionModel.set_default_attn_processor
    def set_default_attn_processor(self):
        """
        Disables custom attention processors and sets the default attention implementation.
        """
        if all(proc.__class__ in ADDED_KV_ATTENTION_PROCESSORS for proc in self.attn_processors.values()):
            processor = AttnAddedKVProcessor()
        elif all(proc.__class__ in CROSS_ATTENTION_PROCESSORS for proc in self.attn_processors.values()):
            processor = AttnProcessor()
        else:
            raise ValueError(
                f"Cannot call `set_default_attn_processor` when attention processors are of type {next(iter(self.attn_processors.values()))}"
            )

        self.set_attn_processor(processor)

    def _set_gradient_checkpointing(self, module, value: bool = False) -> None:
        if isinstance(module, (CrossAttnDownBlockMotion, DownBlockMotion, UNetMidBlock2DCrossAttn)):
            module.gradient_checkpointing = value

    def construct(
        self,
        sample: ms.Tensor,
        timestep: Union[ms.Tensor, float, int],
        encoder_hidden_states: ms.Tensor,
        controlnet_cond: ms.Tensor,
        conditioning_scale: float = 1.0,
        timestep_cond: Optional[ms.Tensor] = None,
        attention_mask: Optional[ms.Tensor] = None,
        cross_attention_kwargs: Optional[Dict[str, Any]] = None,
        conditioning_mask: Optional[ms.Tensor] = None,
        guess_mode: bool = False,
        return_dict: bool = False,
    ) -> Union[SparseControlNetOutput, Tuple[Tuple[ms.Tensor, ...], ms.Tensor]]:
        """
        The [`SparseControlNetModel`] forward method.

        Args:
            sample (`torch.Tensor`):
                The noisy input tensor.
            timestep (`Union[torch.Tensor, float, int]`):
                The number of timesteps to denoise an input.
            encoder_hidden_states (`torch.Tensor`):
                The encoder hidden states.
            controlnet_cond (`torch.Tensor`):
                The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`.
            conditioning_scale (`float`, defaults to `1.0`):
                The scale factor for ControlNet outputs.
            class_labels (`torch.Tensor`, *optional*, defaults to `None`):
                Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings.
            timestep_cond (`torch.Tensor`, *optional*, defaults to `None`):
                Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the
                timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep
                embeddings.
            attention_mask (`torch.Tensor`, *optional*, defaults to `None`):
                An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask
                is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large
                negative values to the attention scores corresponding to "discard" tokens.
            added_cond_kwargs (`dict`):
                Additional conditions for the Stable Diffusion XL UNet.
            cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`):
                A kwargs dictionary that if specified is passed along to the `AttnProcessor`.
            guess_mode (`bool`, defaults to `False`):
                In this mode, the ControlNet encoder tries its best to recognize the input content of the input even if
                you remove all prompts. A `guidance_scale` between 3.0 and 5.0 is recommended.
            return_dict (`bool`, defaults to `True`):
                Whether or not to return a [`~models.controlnet.ControlNetOutput`] instead of a plain tuple.
        Returns:
            [`~models.controlnet.ControlNetOutput`] **or** `tuple`:
                If `return_dict` is `True`, a [`~models.controlnet.ControlNetOutput`] is returned, otherwise a tuple is
                returned where the first element is the sample tensor.
        """
        sample_batch_size, sample_channels, sample_num_frames, sample_height, sample_width = sample.shape
        sample = ops.zeros_like(sample)

        # check channel order
        channel_order = self.config.controlnet_conditioning_channel_order

        if channel_order == "rgb":
            # in rgb order by default
            ...
        elif channel_order == "bgr":
            controlnet_cond = ops.flip(controlnet_cond, dims=[1])
        else:
            raise ValueError(f"unknown `controlnet_conditioning_channel_order`: {channel_order}")

        # prepare attention_mask
        if attention_mask is not None:
            attention_mask = (1 - attention_mask.to(sample.dtype)) * -10000.0
            attention_mask = attention_mask.unsqueeze(1)

        # 1. time
        timesteps = timestep
        if not ops.is_tensor(timesteps):
            if isinstance(timestep, float):
                dtype = ms.float64
            else:
                dtype = ms.int64
            timesteps = ms.Tensor([timesteps], dtype=dtype)
        elif len(timesteps.shape) == 0:
            timesteps = timesteps[None]

        # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
        timesteps = timesteps.broadcast_to((sample.shape[0],))

        t_emb = self.time_proj(timesteps)

        # timesteps does not contain any weights and will always return f32 tensors
        # but time_embedding might actually be running in fp16. so we need to cast here.
        # there might be better ways to encapsulate this.
        t_emb = t_emb.to(dtype=sample.dtype)

        emb = self.time_embedding(t_emb, timestep_cond)
        emb = emb.repeat_interleave(sample_num_frames, dim=0)
        encoder_hidden_states = encoder_hidden_states.repeat_interleave(sample_num_frames, dim=0)

        # 2. pre-process
        batch_size, channels, num_frames, height, width = sample.shape

        sample = sample.permute(0, 2, 1, 3, 4).reshape(batch_size * num_frames, channels, height, width)
        sample = self.conv_in(sample)

        batch_frames, channels, height, width = sample.shape
        sample = sample[:, None].reshape(sample_batch_size, sample_num_frames, channels, height, width)

        if self.concat_conditioning_mask:
            controlnet_cond = ops.cat([controlnet_cond, conditioning_mask], axis=1)

        batch_size, channels, num_frames, height, width = controlnet_cond.shape
        controlnet_cond = controlnet_cond.permute(0, 2, 1, 3, 4).reshape(
            batch_size * num_frames, channels, height, width
        )
        controlnet_cond = self.controlnet_cond_embedding(controlnet_cond)
        batch_frames, channels, height, width = controlnet_cond.shape
        controlnet_cond = controlnet_cond[:, None].reshape(batch_size, num_frames, channels, height, width)

        sample = sample + controlnet_cond

        batch_size, num_frames, channels, height, width = sample.shape
        sample = sample.reshape(sample_batch_size * sample_num_frames, channels, height, width)

        # 3. down
        down_block_res_samples = (sample,)
        for downsample_block in self.down_blocks:
            if hasattr(downsample_block, "has_cross_attention") and downsample_block.has_cross_attention:
                sample, res_samples = downsample_block(
                    hidden_states=sample,
                    temb=emb,
                    encoder_hidden_states=encoder_hidden_states,
                    attention_mask=attention_mask,
                    num_frames=num_frames,
                    cross_attention_kwargs=cross_attention_kwargs,
                )
            else:
                sample, res_samples = downsample_block(hidden_states=sample, temb=emb, num_frames=num_frames)

            down_block_res_samples += res_samples

        # 4. mid
        if self.mid_block is not None:
            if hasattr(self.mid_block, "has_cross_attention") and self.mid_block.has_cross_attention:
                sample = self.mid_block(
                    sample,
                    emb,
                    encoder_hidden_states=encoder_hidden_states,
                    attention_mask=attention_mask,
                    cross_attention_kwargs=cross_attention_kwargs,
                )
            else:
                sample = self.mid_block(sample, emb)

        # 5. Control net blocks
        controlnet_down_block_res_samples = ()

        for down_block_res_sample, controlnet_block in zip(down_block_res_samples, self.controlnet_down_blocks):
            down_block_res_sample = controlnet_block(down_block_res_sample)
            controlnet_down_block_res_samples = controlnet_down_block_res_samples + (down_block_res_sample,)

        down_block_res_samples = controlnet_down_block_res_samples
        mid_block_res_sample = self.controlnet_mid_block(sample)

        # 6. scaling
        if guess_mode and not self.config.global_pool_conditions:
            scales = ops.logspace(-1, 0, len(down_block_res_samples) + 1)  # 0.1 to 1.0
            scales = scales * conditioning_scale
            down_block_res_samples = [sample * scale for sample, scale in zip(down_block_res_samples, scales)]
            mid_block_res_sample = mid_block_res_sample * scales[-1]  # last one
        else:
            down_block_res_samples = [sample * conditioning_scale for sample in down_block_res_samples]
            mid_block_res_sample = mid_block_res_sample * conditioning_scale

        if self.config.global_pool_conditions:
            down_block_res_samples = [
                ops.mean(sample, axis=(2, 3), keep_dims=True) for sample in down_block_res_samples
            ]
            mid_block_res_sample = ops.mean(mid_block_res_sample, axis=(2, 3), keep_dims=True)

        if not return_dict:
            return (down_block_res_samples, mid_block_res_sample)

        return SparseControlNetOutput(
            down_block_res_samples=down_block_res_samples, mid_block_res_sample=mid_block_res_sample
        )

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetModel.attn_processors: Dict[str, AttentionProcessor] property

RETURNS	DESCRIPTION
Dict[str, AttentionProcessor]	dict of attention processors: A dictionary containing all attention processors used in the model with
Dict[str, AttentionProcessor]	indexed by its weight name.

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetModel.construct(sample, timestep, encoder_hidden_states, controlnet_cond, conditioning_scale=1.0, timestep_cond=None, attention_mask=None, cross_attention_kwargs=None, conditioning_mask=None, guess_mode=False, return_dict=False)

The [SparseControlNetModel] forward method.

PARAMETER	DESCRIPTION
sample	The noisy input tensor. TYPE: `torch.Tensor`
timestep	The number of timesteps to denoise an input. TYPE: `Union[torch.Tensor, float, int]`
encoder_hidden_states	The encoder hidden states. TYPE: `torch.Tensor`
controlnet_cond	The conditional input tensor of shape (batch_size, sequence_length, hidden_size). TYPE: `torch.Tensor`
conditioning_scale	The scale factor for ControlNet outputs. TYPE: `float`, defaults to `1.0` DEFAULT: 1.0
class_labels	Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. TYPE: `torch.Tensor`, *optional*, defaults to `None`
timestep_cond	Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the self.time_embedding layer to obtain the final timestep embeddings. TYPE: `torch.Tensor`, *optional*, defaults to `None` DEFAULT: None
attention_mask	An attention mask of shape (batch, key_tokens) is applied to encoder_hidden_states. If 1 the mask is kept, otherwise if 0 it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. TYPE: `torch.Tensor`, *optional*, defaults to `None` DEFAULT: None
added_cond_kwargs	Additional conditions for the Stable Diffusion XL UNet. TYPE: `dict`
cross_attention_kwargs	A kwargs dictionary that if specified is passed along to the AttnProcessor. TYPE: `dict[str]`, *optional*, defaults to `None` DEFAULT: None
guess_mode	In this mode, the ControlNet encoder tries its best to recognize the input content of the input even if you remove all prompts. A guidance_scale between 3.0 and 5.0 is recommended. TYPE: `bool`, defaults to `False` DEFAULT: False
return_dict	Whether or not to return a [~models.controlnet.ControlNetOutput] instead of a plain tuple. TYPE: `bool`, defaults to `True` DEFAULT: False

Source code in mindone/diffusers/models/controlnet_sparsectrl.py

def construct(
    self,
    sample: ms.Tensor,
    timestep: Union[ms.Tensor, float, int],
    encoder_hidden_states: ms.Tensor,
    controlnet_cond: ms.Tensor,
    conditioning_scale: float = 1.0,
    timestep_cond: Optional[ms.Tensor] = None,
    attention_mask: Optional[ms.Tensor] = None,
    cross_attention_kwargs: Optional[Dict[str, Any]] = None,
    conditioning_mask: Optional[ms.Tensor] = None,
    guess_mode: bool = False,
    return_dict: bool = False,
) -> Union[SparseControlNetOutput, Tuple[Tuple[ms.Tensor, ...], ms.Tensor]]:
    """
    The [`SparseControlNetModel`] forward method.

    Args:
        sample (`torch.Tensor`):
            The noisy input tensor.
        timestep (`Union[torch.Tensor, float, int]`):
            The number of timesteps to denoise an input.
        encoder_hidden_states (`torch.Tensor`):
            The encoder hidden states.
        controlnet_cond (`torch.Tensor`):
            The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`.
        conditioning_scale (`float`, defaults to `1.0`):
            The scale factor for ControlNet outputs.
        class_labels (`torch.Tensor`, *optional*, defaults to `None`):
            Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings.
        timestep_cond (`torch.Tensor`, *optional*, defaults to `None`):
            Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the
            timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep
            embeddings.
        attention_mask (`torch.Tensor`, *optional*, defaults to `None`):
            An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask
            is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large
            negative values to the attention scores corresponding to "discard" tokens.
        added_cond_kwargs (`dict`):
            Additional conditions for the Stable Diffusion XL UNet.
        cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`):
            A kwargs dictionary that if specified is passed along to the `AttnProcessor`.
        guess_mode (`bool`, defaults to `False`):
            In this mode, the ControlNet encoder tries its best to recognize the input content of the input even if
            you remove all prompts. A `guidance_scale` between 3.0 and 5.0 is recommended.
        return_dict (`bool`, defaults to `True`):
            Whether or not to return a [`~models.controlnet.ControlNetOutput`] instead of a plain tuple.
    Returns:
        [`~models.controlnet.ControlNetOutput`] **or** `tuple`:
            If `return_dict` is `True`, a [`~models.controlnet.ControlNetOutput`] is returned, otherwise a tuple is
            returned where the first element is the sample tensor.
    """
    sample_batch_size, sample_channels, sample_num_frames, sample_height, sample_width = sample.shape
    sample = ops.zeros_like(sample)

    # check channel order
    channel_order = self.config.controlnet_conditioning_channel_order

    if channel_order == "rgb":
        # in rgb order by default
        ...
    elif channel_order == "bgr":
        controlnet_cond = ops.flip(controlnet_cond, dims=[1])
    else:
        raise ValueError(f"unknown `controlnet_conditioning_channel_order`: {channel_order}")

    # prepare attention_mask
    if attention_mask is not None:
        attention_mask = (1 - attention_mask.to(sample.dtype)) * -10000.0
        attention_mask = attention_mask.unsqueeze(1)

    # 1. time
    timesteps = timestep
    if not ops.is_tensor(timesteps):
        if isinstance(timestep, float):
            dtype = ms.float64
        else:
            dtype = ms.int64
        timesteps = ms.Tensor([timesteps], dtype=dtype)
    elif len(timesteps.shape) == 0:
        timesteps = timesteps[None]

    # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
    timesteps = timesteps.broadcast_to((sample.shape[0],))

    t_emb = self.time_proj(timesteps)

    # timesteps does not contain any weights and will always return f32 tensors
    # but time_embedding might actually be running in fp16. so we need to cast here.
    # there might be better ways to encapsulate this.
    t_emb = t_emb.to(dtype=sample.dtype)

    emb = self.time_embedding(t_emb, timestep_cond)
    emb = emb.repeat_interleave(sample_num_frames, dim=0)
    encoder_hidden_states = encoder_hidden_states.repeat_interleave(sample_num_frames, dim=0)

    # 2. pre-process
    batch_size, channels, num_frames, height, width = sample.shape

    sample = sample.permute(0, 2, 1, 3, 4).reshape(batch_size * num_frames, channels, height, width)
    sample = self.conv_in(sample)

    batch_frames, channels, height, width = sample.shape
    sample = sample[:, None].reshape(sample_batch_size, sample_num_frames, channels, height, width)

    if self.concat_conditioning_mask:
        controlnet_cond = ops.cat([controlnet_cond, conditioning_mask], axis=1)

    batch_size, channels, num_frames, height, width = controlnet_cond.shape
    controlnet_cond = controlnet_cond.permute(0, 2, 1, 3, 4).reshape(
        batch_size * num_frames, channels, height, width
    )
    controlnet_cond = self.controlnet_cond_embedding(controlnet_cond)
    batch_frames, channels, height, width = controlnet_cond.shape
    controlnet_cond = controlnet_cond[:, None].reshape(batch_size, num_frames, channels, height, width)

    sample = sample + controlnet_cond

    batch_size, num_frames, channels, height, width = sample.shape
    sample = sample.reshape(sample_batch_size * sample_num_frames, channels, height, width)

    # 3. down
    down_block_res_samples = (sample,)
    for downsample_block in self.down_blocks:
        if hasattr(downsample_block, "has_cross_attention") and downsample_block.has_cross_attention:
            sample, res_samples = downsample_block(
                hidden_states=sample,
                temb=emb,
                encoder_hidden_states=encoder_hidden_states,
                attention_mask=attention_mask,
                num_frames=num_frames,
                cross_attention_kwargs=cross_attention_kwargs,
            )
        else:
            sample, res_samples = downsample_block(hidden_states=sample, temb=emb, num_frames=num_frames)

        down_block_res_samples += res_samples

    # 4. mid
    if self.mid_block is not None:
        if hasattr(self.mid_block, "has_cross_attention") and self.mid_block.has_cross_attention:
            sample = self.mid_block(
                sample,
                emb,
                encoder_hidden_states=encoder_hidden_states,
                attention_mask=attention_mask,
                cross_attention_kwargs=cross_attention_kwargs,
            )
        else:
            sample = self.mid_block(sample, emb)

    # 5. Control net blocks
    controlnet_down_block_res_samples = ()

    for down_block_res_sample, controlnet_block in zip(down_block_res_samples, self.controlnet_down_blocks):
        down_block_res_sample = controlnet_block(down_block_res_sample)
        controlnet_down_block_res_samples = controlnet_down_block_res_samples + (down_block_res_sample,)

    down_block_res_samples = controlnet_down_block_res_samples
    mid_block_res_sample = self.controlnet_mid_block(sample)

    # 6. scaling
    if guess_mode and not self.config.global_pool_conditions:
        scales = ops.logspace(-1, 0, len(down_block_res_samples) + 1)  # 0.1 to 1.0
        scales = scales * conditioning_scale
        down_block_res_samples = [sample * scale for sample, scale in zip(down_block_res_samples, scales)]
        mid_block_res_sample = mid_block_res_sample * scales[-1]  # last one
    else:
        down_block_res_samples = [sample * conditioning_scale for sample in down_block_res_samples]
        mid_block_res_sample = mid_block_res_sample * conditioning_scale

    if self.config.global_pool_conditions:
        down_block_res_samples = [
            ops.mean(sample, axis=(2, 3), keep_dims=True) for sample in down_block_res_samples
        ]
        mid_block_res_sample = ops.mean(mid_block_res_sample, axis=(2, 3), keep_dims=True)

    if not return_dict:
        return (down_block_res_samples, mid_block_res_sample)

    return SparseControlNetOutput(
        down_block_res_samples=down_block_res_samples, mid_block_res_sample=mid_block_res_sample
    )

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetModel.from_unet(unet, controlnet_conditioning_channel_order='rgb', conditioning_embedding_out_channels=(16, 32, 96, 256), load_weights_from_unet=True, conditioning_channels=3) classmethod

Instantiate a [SparseControlNetModel] from [UNet2DConditionModel].

PARAMETER	DESCRIPTION
unet	The UNet model weights to copy to the [SparseControlNetModel]. All configuration options are also copied where applicable. TYPE: `UNet2DConditionModel`

Source code in mindone/diffusers/models/controlnet_sparsectrl.py

@classmethod
def from_unet(
    cls,
    unet: UNet2DConditionModel,
    controlnet_conditioning_channel_order: str = "rgb",
    conditioning_embedding_out_channels: Optional[Tuple[int, ...]] = (16, 32, 96, 256),
    load_weights_from_unet: bool = True,
    conditioning_channels: int = 3,
) -> "SparseControlNetModel":
    r"""
    Instantiate a [`SparseControlNetModel`] from [`UNet2DConditionModel`].

    Parameters:
        unet (`UNet2DConditionModel`):
            The UNet model weights to copy to the [`SparseControlNetModel`]. All configuration options are also
            copied where applicable.
    """
    transformer_layers_per_block = (
        unet.config.transformer_layers_per_block if "transformer_layers_per_block" in unet.config else 1
    )
    down_block_types = unet.config.down_block_types

    for i in range(len(down_block_types)):
        if "CrossAttn" in down_block_types[i]:
            down_block_types[i] = "CrossAttnDownBlockMotion"
        elif "Down" in down_block_types[i]:
            down_block_types[i] = "DownBlockMotion"
        else:
            raise ValueError("Invalid `block_type` encountered. Must be a cross-attention or down block")

    controlnet = cls(
        in_channels=unet.config.in_channels,
        conditioning_channels=conditioning_channels,
        flip_sin_to_cos=unet.config.flip_sin_to_cos,
        freq_shift=unet.config.freq_shift,
        down_block_types=unet.config.down_block_types,
        only_cross_attention=unet.config.only_cross_attention,
        block_out_channels=unet.config.block_out_channels,
        layers_per_block=unet.config.layers_per_block,
        downsample_padding=unet.config.downsample_padding,
        mid_block_scale_factor=unet.config.mid_block_scale_factor,
        act_fn=unet.config.act_fn,
        norm_num_groups=unet.config.norm_num_groups,
        norm_eps=unet.config.norm_eps,
        cross_attention_dim=unet.config.cross_attention_dim,
        transformer_layers_per_block=transformer_layers_per_block,
        attention_head_dim=unet.config.attention_head_dim,
        num_attention_heads=unet.config.num_attention_heads,
        use_linear_projection=unet.config.use_linear_projection,
        upcast_attention=unet.config.upcast_attention,
        resnet_time_scale_shift=unet.config.resnet_time_scale_shift,
        conditioning_embedding_out_channels=conditioning_embedding_out_channels,
        controlnet_conditioning_channel_order=controlnet_conditioning_channel_order,
    )

    if load_weights_from_unet:
        ms.load_param_into_net(controlnet.conv_in, unet.conv_in.parameters_dict())
        ms.load_param_into_net(controlnet.time_proj, unet.time_proj.parameters_dict())
        ms.load_param_into_net(controlnet.time_embedding, unet.time_embedding.parameters_dict())
        ms.load_param_into_net(controlnet.down_blocks, unet.down_blocks.parameters_dict())
        ms.load_param_into_net(controlnet.mid_block, unet.mid_block.parameters_dict())

    return controlnet

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetModel.set_attn_processor(processor)

Sets the attention processor to use to compute attention.

PARAMETER	DESCRIPTION
processor	The instantiated processor class or a dictionary of processor classes that will be set as the processor for all Attention layers. If processor is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. TYPE: `dict` of `AttentionProcessor` or only `AttentionProcessor`

Source code in mindone/diffusers/models/controlnet_sparsectrl.py

def set_attn_processor(self, processor: Union[AttentionProcessor, Dict[str, AttentionProcessor]]):
    r"""
    Sets the attention processor to use to compute attention.

    Parameters:
        processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`):
            The instantiated processor class or a dictionary of processor classes that will be set as the processor
            for **all** `Attention` layers.

            If `processor` is a dict, the key needs to define the path to the corresponding cross attention
            processor. This is strongly recommended when setting trainable attention processors.

    """
    count = len(self.attn_processors.keys())

    if isinstance(processor, dict) and len(processor) != count:
        raise ValueError(
            f"A dict of processors was passed, but the number of processors {len(processor)} does not match the"
            f" number of attention layers: {count}. Please make sure to pass {count} processor classes."
        )

    def fn_recursive_attn_processor(name: str, module: nn.Cell, processor):
        if hasattr(module, "set_processor"):
            if not isinstance(processor, dict):
                module.set_processor(processor)
            else:
                module.set_processor(processor.pop(f"{name}.processor"))

        for sub_name, child in module.name_cells().items():
            fn_recursive_attn_processor(f"{name}.{sub_name}", child, processor)

    for name, module in self.name_cells().items():
        fn_recursive_attn_processor(name, module, processor)

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetModel.set_default_attn_processor()

Disables custom attention processors and sets the default attention implementation.

Source code in mindone/diffusers/models/controlnet_sparsectrl.py

def set_default_attn_processor(self):
    """
    Disables custom attention processors and sets the default attention implementation.
    """
    if all(proc.__class__ in ADDED_KV_ATTENTION_PROCESSORS for proc in self.attn_processors.values()):
        processor = AttnAddedKVProcessor()
    elif all(proc.__class__ in CROSS_ATTENTION_PROCESSORS for proc in self.attn_processors.values()):
        processor = AttnProcessor()
    else:
        raise ValueError(
            f"Cannot call `set_default_attn_processor` when attention processors are of type {next(iter(self.attn_processors.values()))}"
        )

    self.set_attn_processor(processor)

mindone.diffusers.models.controlnet_sparsectrl.SparseControlNetOutput dataclass

Bases: BaseOutput

The output of [SparseControlNetModel].

PARAMETER	DESCRIPTION
down_block_res_samples	A tuple of downsample activations at different resolutions for each downsampling block. Each tensor should be of shape (batch_size, channel * resolution, height //resolution, width // resolution). Output can be used to condition the original UNet's downsampling activations. TYPE: `tuple[torch.Tensor]`
mid_down_block_re_sample	The activation of the middle block (the lowest sample resolution). Each tensor should be of shape (batch_size, channel * lowest_resolution, height // lowest_resolution, width // lowest_resolution). Output can be used to condition the original UNet's middle block activation. TYPE: `torch.Tensor`

Source code in mindone/diffusers/models/controlnet_sparsectrl.py

@dataclass
class SparseControlNetOutput(BaseOutput):
    """
    The output of [`SparseControlNetModel`].

    Args:
        down_block_res_samples (`tuple[torch.Tensor]`):
            A tuple of downsample activations at different resolutions for each downsampling block. Each tensor should
            be of shape `(batch_size, channel * resolution, height //resolution, width // resolution)`. Output can be
            used to condition the original UNet's downsampling activations.
        mid_down_block_re_sample (`torch.Tensor`):
            The activation of the middle block (the lowest sample resolution). Each tensor should be of shape
            `(batch_size, channel * lowest_resolution, height // lowest_resolution, width // lowest_resolution)`.
            Output can be used to condition the original UNet's middle block activation.
    """

    down_block_res_samples: Tuple[ms.Tensor]
    mid_block_res_sample: ms.Tensor