Concept Gradient: Concept-based Interpretation Without Linear Assumption

We assume $x$ is the input instance, $y=f\left(x\right)$ is the $k$-dimensional model output and $c=g\left(x\right)$ is the $m$-dimensional concepts. For standard feature-based explanation, which aims to measure the important of each input dimension of $x$, gradient-based methods have been widely used. The gradient map $\nabla f\left(x\right)$ measures how small changes on each input dimension affects the output and serves as the foundation of many existing explanation methods (Ancona et al., 2017; Shrikumar et al., 2017; Sundararajan et al., 2017). This paper is the first formally extending the notion of gradient to concept-based explanation, and by doing so we enable explanation of any general nonlinear concept. We define the Concept Gradient (CG) to measure how small perturbations on each concept affects the label prediction:

In practice, CG can be easily computed by fine-tuning the original target model on the concept labels to create a non-linear concept model (see more in Section 4.2). Note that when computing the gradient of a multivariate function such as $f$, we follow the convention that $\nabla f\left(x\right)\in R^{d\times k}$ where the $\left(\nabla f\right(x){)}_{ij}=\frac{\partial f_j\left(x\right)}{\partial x_i}$. And $\nabla g(x{)}^{†}\in R^{m\times d}$ is the pseudo-inverse of $\nabla g\left(x\right)$. $\text{CG}\left(x\right)$ will thus be an $m\times k$ matrix and its $(i,j)$ element measures the contribution of concept $i$ to label $j$. Pseudo-inverse is a generalized version of matrix inversion—when the inverse does not exist, it forms the (unique) inverse mapping from the column space to the row space of the original matrix, while leaving all the other spaces untouched. Assume the Singular Value Decomposition (SVD) of $\nabla g\left(x\right)$ is $U_g{\Sigma }_g{V}_g^T$, then $\nabla g(x{)}^{†}$ is defined as $V_g{\Sigma }_g^{†}{U}_g$ where ${\Sigma }_g^{†}$ is formed from ${\Sigma }_g$ by taking the inverse of all the non-zero elements and leaving all the zeros alone.

Similar to CAV, CG can be computed at any layer of a neural network by setting $x$ as the neurons in a particular layer. We will discuss a guideline for layer selection in Section 3.5.

Before formally deriving CG, we use a few special cases to illustrate the method. First, for the case when $x,y$ and $c$ are all scalars ($k=m=d=1$), there exists a function mapping $c$ to $y$ locally around a particular $\hat{c}=g\left(\hat{x}\right)$ as long as $g^{\prime }\left(\hat{x}\right)\ne 0$. Basic on calculus, under this condition in 1D case we have $\frac{dc}{dx}=1/\left(\frac{dx}{dc}\right)$ at $\hat{x}$, so concept gradient exactly recovers $\frac{dy}{dc}$:

This can be easily extended to the case when $m=d$ and $\nabla g\left(\hat{x}\right)$ is invertible. In this case, there exists an unique function $g^{-1}\left(c\right)$ mapping $c$ to $x$ locally around $\hat{c}$, and by the chain rule we show $\text{CG}\left(\hat{x}\right)$ is equivalent to the derivative of $y$ with respect to $c$:

The pseudo-inverse in (2) extends this derivation to the general case when $m,d$ are in arbitrary dimensions, and we will formally derive it in Section 3.4.

In previous concept attribution methods such as CAV, a canonical setting is to measure the importance of a single concept. Here we try to simplify CG in this setting when concepts are scalars and discuss the relationship with CAV.

When $m=1$, $\nabla g\left(x\right)$ is a $d\times 1$ vector and by the definition of pseudo-inverse, $\nabla g(x{)}^{†}=\frac{\nabla g(x{)}^T}{\parallel \nabla g\left(x\right){\parallel }^2}$, which leads to the following simplified concept gradient when explaining a single concept:

In this case, CG is simply a normalized version of the inner product between the gradient of concept ($\nabla g\left(x\right)$) and the gradient of prediction model ($\nabla f\left(x\right)$). This is the main formulation we are going to use in practice, since we found applying CG to each concept individually will match better with human perception, as will be discussed in Section 3.3.

In the special case when $g\left(x\right)=v_C$ is a linear function (the assumption of CAV), we have $\text{CG}\left(x\right)=v_C^T\nabla f\left(x\right)/\parallel v_C{\parallel }^2$, which is almost identical to conceptual sensitivity score in Eq 1 except a slightly different normalization term where CAV normalizes the inner product by $1/\parallel v_C\parallel$. Furthermore, the sign of CG and CAV will be identical which explains why CAV is capable of retrieving important concepts under the linearly separable case.

Here we use a simple example to demonstrate that the normalization term could be important in some special cases. Consider $f$ as the following network with two-dimensional input $[x_0,x_1]$:

and $c_0=x_0$, $c_1=x_1$. Then we know since $y=0.1z_0+z_1=0.1x_0+x_1$, the contribution of $c_1$ should be 10 times larger than $c_0$. In fact, $\frac{dy}{d{c}_0}=0.1,\frac{dy}{d{c}_1}=1$ and it’s easy to verify that CG will correctly obtain the gradient no matter which layer is chosen for computing (4). However, the results will be wrong when a different normalization term is used when computing concept explanation on the hidden layer $h$. Since $c_0=100h_0,c_1=h_1,y=10h_0+h_1$, we have

Therefore, the normalization term used in CAV (in red color) will lead to a conclusion that $c_0>c_1$, while CG (in blue color) will correctly get the actual gradient and conclude $c_1>c_0$. This is mainly because CG is formally derived from the actual gradient, as we will discuss below. In contrast, CAV is based on the intuition of “gradient sensitivity” and not the exact chain-rule gradient, so is subject to per-dimension scaling.

Although the normalization term can be important in some special cases, in practice we do not find the attribution results to be much different with different normalization terms, probably because such extreme per-dimensional scaling rarely happens in well-trained neural networks. We compared different methods of calculating CG (including different normalization schemes) empirically in Section 4.2. Thus, in practice if the concept can be accurately modeled by a linear function, CAV is capable of retrieving the concept gradient. However, in general the linear separability assumption does not hold and CG can handle any nonlinear concept.

When attributing the prediction to multiple concepts, our flexible framework enables two options: 1) treating each concept independently using single-concept attribution formulation (4) 2) Combining all the concepts together into $c\in R^m$ and run CG. Intuitively, option 2 takes the correlations between concepts into account while option 1 does not. When both concept A and B are important for prediction but concept A is slightly more important than concept B, option 1 will identify both of them to be important while option 2 may attribute the prediction to concept B only. For example, when $y=x_0+x_1,c_0=x_0,c_1=x_0+0.1x_1$, option 1 will identify both concepts to be important, while option 2 will produce negative score for $c_0$. Details can be found in the appendix. We further compared these two options on real datasets. The results are presented in Section 4.2. We verified that empirically applying pseudo-inverse individually for each concept is better aligned with the attributes labeled by human.

Here we derive CG in the general case when $\nabla g\left(x\right)$ is not invertible. For simplicity, we assume $\nabla g\left(x\right)$ has full column rank, which implies $g$ is surjective (this is always true when $m=1$). Otherwise we can directly constraint $c$ within the row space of $\nabla g\left(x\right)$ and the arguments below will still go through.

Our goal is to construct a mapping from $c$ to $x$ and analyze the gradient. However, there are infinite many functions from $c$ to $x$ that can locally inverse $g\left(x\right)$ since the dimension of concept ($m$) could be much smaller than input dimension $d$. Despite infinite number of choices, we show the gradient of such function always follows a particular form:

The proof of the theorem is deferred to the appendix. Intuitively, this implies the gradient of $h$ will take a particular form in the space of $\nabla g(x{)}^T$ while being arbitrary in its null space since any change in the null space cannot locally affect $c$. We can verify that $g\left(x\right)$ is locally unchanged in the null space of $\nabla g(x{)}^T$, as

This is saying there are multiple choices in $\Delta x$ to achieve the same affect on $c$, since any additional perturbation in $\text{null}\left(\nabla g\right(x{)}^T)$ won’t make any change to $c$ locally. For example, when we want to change the concept of “color” in an image by $\Delta x$, we can have $\Delta x$ only including the minimal change (e.g., only changing the color), or have $\Delta x$ including change of color and any arbitrary change to another orthogonal factor (e.g., shape). For concept-based attribution, it is natural to consider the minimal space of $x$ that can cover $c$, which corresponds to setting $G_{\perp }$ as $0$ in (6).

If we pick such $h$ then $\nabla h\left(\hat{c}\right)=\nabla g(\hat{x}{)}^{†}$, so we can represent $y$ as a function of $c$ locally by $y=p\left(c\right):=f\left(h\right(c\left)\right)$ near $\hat{c}$. By chain rule we then have

In summary, although there are infinitely many functions from $c$ to $y$ since the input space has larger dimension than the concept space, if we consider a subspace in $x$ such that change of $x$ will affect $c$ (locally) and ignore the orthogonal space that is irrelevant to $c$, then any function mapping from $c$ to $y$ through this space will have gradient equal to Concept Gradient defined in (2).

The representation of input $x$ is relevant to CG, as the information contained differs between representations. Similar to CAV, CG faces the challenge of properly selecting a layer to perform calculation. For a feed-forward neural network model, information irrelevant to the target task is removed when propagating through the layers. Let us denote the representation of $x$ in the $l^{\text{th}}$ layer of the target model $f$ as $x_{f_l}$. We hypothesized that the optimal layer $l^{\ast }$ for performing CG is where the representation $x_{f_{l^{\ast }}}$ contains minimally necessary and sufficient information to predict concept labels. Let $H$ denote information entropy and $I$ denote mutual information. Here we overload $X$ and $C$ to denote input and concept random variables.

Since $H\left(X_{f_l}|C\right)=H\left(X_{f_l}\right)-I(X_{f_l};C)$ and $H\left(X_{f_l}\right)\ge H\left(X_{f_{l+1}}\right)$, $l^{\ast }$ corresponds to the latest layer in $f$ where $X_{f_l}$ can predict $C$. Intuitively, the representation of $x_{f_l}$ needs to contain sufficient information to correctly predict concepts to ensure the concept gradients $\nabla g\left(x\right)$ are accurate. On the other hand, if there is redundant information in $x_{f_l}$ that can be utilized to predict the target $y$ and concept $c$, then $g$ may not rely on the same information as $f$, which causes misalignment in gradients $\nabla g\left(x\right)$ and $\nabla f\left(x\right)$ leading to underestimation of concept importance.

The algorithm for selecting the optimal layer to perform CG is simple. The model $g$ is initialized with weights from $f$ and all the weights are initially frozen. Starting from the last layer, we unfreeze the layer weights and finetune $g$ to predict concepts. We train until the model converges and evaluate the concept prediction accuracy on a holdout validation set. The next step is to unfreeze the previous layer and repeat the whole process until the concept prediction accuracy saturates and no longer improves as more layers are unfrozen. We have then found the optimal layer for CG as well as the concept model $g$.

The purpose of this example is to test whether TCAV and CG can recover the actual gradient $\frac{\partial y}{\partial c}$ in a synthetic scenario where the derivative of target $y$ with respect to concept $c$ uniquely exist and can be expressed in closed-form. Theoretically, we have shown CG is capable of exactly recovering the derivative via chain rule in Eq 3. On the other hand, CAV can only retrieve the gradient attribution if the concepts are linearly separable in some input latent representation. We find that the linear separability assumption doesn’t hold even in such simple scenarios.

Dataset We define the inputs $x_0,x_1$, concepts $c_0,c_1$, and target $y$ as follows

The coefficients are randomly generated $k_0=0.5388,k_1=0.9198,{\alpha }_0=0.3633,{\alpha }_1=0.2271$. Clearly, $\frac{\partial y}{\partial c_0}={\alpha }_0$ and $\frac{\partial y}{\partial c_1}={\alpha }_1$. The derivative of concepts with respect to inputs can also be represented in closed-form expression

Training First, we trained a 4-hidden-layer, fully-connected neural network model $f$ that maps $(x_0,x_1)$ to $y$. This serves as the target model to be interpreted. For CAV, we calculate the CAVs corresponding to the two concepts $(c_0,c_1)$ in the first 3 hidden layers. Note that the last hidden layer cannot be used to calculate CAV otherwise the concept saliency score $S$ would degenerate to a constant for all input $(x_0,x_1)$. For CG, we trained two 4-hidden-layer, fully-connected neural network models $g_0,g_1$ that maps $(x_0,x_1)$ to $c_0$ and $c_1$, respectively.

Evaluation Fig 1 visualizes the concept prediction results with CAV and CG. Here we fixed one of the input variables ($x_0$ or $x_1$) to a constant value and show the relation between the remaining input variable and the predicted concepts. CG captures the concept relation significantly better than the linear functions of CAV (for all layers). CG concept predictions are almost identical to the ground truth. Next we compare the concept importance attribution between CAV and CG. For CAV, we calculate the concept saliency $S$ as concept importance. For CG, we calculate the concept gradient via chain rule as shown in Eq 3. Recall the ground truth importance attribution for $y$ is ${\alpha }_0$, ${\alpha }_1$ for $c_0$, $c_1$, respectively, constant for every input sample. The mean square error for the predicted concept importance is $5.64\times {10}^{-2}$, $3.47\times {10}^{-2}$, $6.63\times {10}^{-2}$, and $3.6\times {10}^{-3}$ for CAV (net.0), CAV (net.2), CAV (net.4), and CG, respectively. The error of CG is an order less than even the best of the CAVs. Thus, we have shown that CG is capable of capturing the concept relation better, which leads to more accurate gradient estimation and outperforming CAV in concept importance attribution.

In this experiment, our goal is to quantitatively benchmark how well CG is capable of correctly retrieving relevant concepts in a setting where the ground truth concept importance is available.

Dataset We conducted the experiment on the CUB-200-2011 (Wah et al., 2011) dataset, a dataset for fine-grained bird image classification. It consists of 11k bird images, 200 bird classes, and 312 binary bird attributes. These attributes are descriptors of bird parts (e.g. bill shape, breast pattern, eye color) that can be used for classification. We followed experimental setting and preprocessing in Koh et al. (2020) where class-wise attributed labels are derived from instance-wise attribute labels via majority vote for denoising and sparse attributes are filtered out, which leaves 112 attributes as concepts for interpretation.

Training We first trained models to predict the class labels as the target model $f$. We then finetuned $f$ to predict concepts labels, freezing some model weights, to serve as the concept model $g$. The study is conducted on three CNN architectures: Inception v3, Resnet50, and VGG16. We performed extensive study on how finetuning different portions of the model as well as which layer is used for evaluating concept gradient affect CG’s importance attribution. We also performed trained CAVs on different model layers as baselines for comparison. The layer for evaluating CG or CAV defaults to previous layer of finetuning, unless specified otherwise. More training details are in the Appendix.

Evaluation We evaluate the performance of concept importance attribution by measuring the concept recall. Specifically, we treat concept prediction as a multilabel classification problem. For an input instances, there are multiple concepts with positive labels. A good importance attribution method should assign highest concept importance to concepts associated with positive labels. We rank the concepts according to their attributed importance and take the top $k$ to calculate recall@$k$. Higher recall implies better alignment between predicted and ground truth concept importance. Table 1 compares the best result of TCAV and CG on all 3 model architectures. CG outperforms TCAV significantly on every model we have experimented with.

For CG, we experimented with finetuning with different portions of the model weights frozen. We plotted the CG concept recalls in Fig 2. The plus sign in x-axis implies all layers after the specified layer are also finetuned. For reference, we also plotted the concept prediction accuracy. The first thing we notice is that as the number of finetuned layers increases, the concept validation accuracy increases until some layer, then plateaus. This is consistent with our analysis in Section 3.5. The mutual information between the representation of $X_l$ and the concept $C$ gradually reduces in deeper layers. Therefore as more layers are unfrozen and finetuned, more information can be used to predict concepts. The optimal recall occurs when the concept accuracy plateaus, as predicted in our analysis. The same phenomenon is observed in all 3 architectures: Mixed_7b for Inception v3, layer4.1 for Resnet50, and features.34 for VGG16.

For CAV, we experimented with different layers of the model. We plotted the CAV concept recalls in Fig 3. Similar to the CG results, the concept accuracy are provided for reference to show how well the CAVs capture the concepts. We observe that the CAVs in later layers perform better in both recall and concept accuracy. The trend is generally monotonic. The accuracy never saturates since linear functions are insufficient to predict the concepts, even for the final layer where the concept is closest to being linearly separable in the representation of $x$. CAV is equivalent to CG in the final layer since the final layer is a linear layer. Interestingly, the result is the best for CAV, but the worst for CG in the final layer. This is also verified in Fig 2 and Fig 3, where the worst result of CG matches the best result of CAV at the final layer. Therefore, the performance of CG dominates that of CAV’s.

Finally, we compare different methods of calculating the concept gradient, including pseudo-inverse jointly with all concepts versus each concept separately as well as various gradient normalization schemes. We ran all experiments on Inception v3 with finetuned layers Mixed_7b+. The results are presented in Table 2. Joint represents the pseudo-inverse being performed on all the concepts jointly while individual represents the pseudo-inverse being performed on each individual concept separately. Un-normalized represents pure inner product between $\nabla f\left(x\right)$ and $\nabla g\left(x\right)$ without normalization, normed $\nabla g\left(x\right)$ with normalized $\nabla g\left(x\right)$, and cosine with both normalized $\nabla f\left(x\right)$ and $\nabla g\left(x\right)$. Recall that individual pseudo-inverse is exactly inner product normalized with squared concept norm. We observe that the concept attribution with joint pseudo-inverse is not aligned with human perception of concept importance while all other methods performs equally well. This supports the argument that the gradient norms in trained neural network are well-behaved and it is unlikely to encounter cases where normalization influences the attribution results significantly.

The purpose of this experiment is to provide intuition and serve as a sanity check by visualizing instances and how CG works.

Dataset We conducted the experiment on the Animals with Attributes 2 (AwA2) dataset (Xian et al., 2018), an image classification dataset with 37k animal images, 50 animal classes, and 85 binary attributes for each class. These concepts cover a wide range of semantics, from low level colors and textures, to high level abstract descriptions (e.g. “smart”, “domestic”). We further filtered out 60 concepts that is visible in the input images to perform interpretation.

Evaluation The evaluation is performed on the validation set. Fig 4 visualizes the instances with the highest CG importance attribution for 6 selected concepts, filtering out samples from the same class (top 1 instance in the top 3 classes). The concepts are selected to represent different levels of semantics. The top row contains colors (low-level), the middle row contains textures (medium-level), and the bottom row contains body components (high-level). Observe that CG is capable of handling different levels of semantics simultaneously well, owing to the expressiveness of non-linear concept model $g$. Additionally, we presented random sampled instances from the validation set and listed top-10 most important concepts as attributed by CG (see the appendix). We intentionally avoided curating the visualization samples to demonstrate true importance attribution performance of CG. The most important concepts for each instance passed the sanity check. There are no contradictory concept-class pairings and importance is attributed to concepts existent in the images.