SeSQL: Yet Another Large-scale Session-level Chinese Text-to-SQL Dataset

In the early stage of this work, we observed that one annotator tended to have a very limited number of ways for advancing a session, probably due to thinking habits and background knowledge. In other words, annotators usually followed a few fixed patterns to ask new questions in order to improve annotation speed. Therefore, if we let one annotator to complete a whole session, the constructed data would probably contain strong annotator-related biases Mor et al. (2019) and be less diverse.

To deal with this issue, we adopted an iterative annotation workflow, as illustrated in Figure 2. The basic idea is that one annotator only completes one NL question and one SQL query, and previous submissions are intensively reviewed by subsequent annotators. There are six possible subtasks for an annotator to complete at a time.

Subtask 1: knowing the context. The annotator first reads all previous NL questions in order to 1) know what the session is about, and 2) avoid asking identical or similar questions.

Subtask 2: checking the previous submission. The annotator must carefully check and correct the submission of the previous annotator, which usually consists of two parts, i.e., a current-round SQL query, and a previous-round NL question (if not first-round). We find that this step is very important for avoiding error accumulation.

Subtask 3: writing an NL question. The annotator writes a qualified NL question for the current-round SQL query. On the one hand, the question should correctly and exactly express the meaning of the SQL query. On the other hand, the question should be expressed in a flexible and natural manner, imitating human conversation in real life.

Subtask 4: composing an SQL query. The annotator composes a new next-round SQL query, which is detailed in Section 3.4.

Subtask 5: verifying corrections via interaction. If the annotator (A) finds and corrects mistakes in the previous submission of another annotator (B). Our annotation tool will deliver the original submission along with the corrections to annotator B for his confirmation. If annotator B agrees with A, then the issue is settled; otherwise, a senior annotator is called to make a final decision.

Subtask 6: making a request for ending a session. An annotator may make a request for ending a session after completing subtask 2, when he fails to think of anything more to ask. Then a senior annotator handles the request.

Following Yu et al. (2019b), we require the number of question/SQL pairs in a session should range between 3 and 10. A session is automatically terminated if the number reaches 10.

Discussion. The one-annotator-one-session workflow, adopted by CHASE, means that a session is completed by a single annotator.

As we discussed in Section 1, it may introduce strong annotator-related bias, since annotator usually have a limited number of ways to advance a session. We observe that our iterative workflow can effectively alleviate this issue. Another advantage of our iterative workflow is that a previous submission is reviewed timely, which can avoid error accumulation and improve data quality. In contrast, data review can only be performed only after a whole session is completed in CHASE. Nevertheless, it would be very expensive to compare the two workflows via strict quantitative experiments, which is beyond the scope of this work.

Collecting DBs is a non-trivial work. For simplicity, we reuse all 201 DBs with 813 tables of the DuSQL dataset¹¹1The license and data is at https://www.luge.ai/#/luge/dataDetail?id=13. Wang et al. (2020b).

After looking into the data, we find that there are a lot of noises in the original DBs of DuSQL, which is also pointed out by Guo et al. (2021). Most noises fall into four categories: 1) primary or foreign keys are not given; 2) the value type of a cell does not match its column type; 3) some cells do not have values; 4) a duplicate value occurs in the primary key column.

In order to improve the quality of DBs and make sure that all legal SQL queries can be successfully executed, our six senior annotators have manually checked and corrected all DBs²²2Please note that we ask annotators not to introduce identification information and ask them to anonymize the existing identification information. before real annotation. Each annotator handles about 35 DBs.

Creating suitable initial queries are crucial for session-level text-to-SQL data creation, since they directly influence subsequent annotations. The suitability of initial queries depends on two aspects, i.e., simplicity and diversity. Regarding to the first aspect, we find that queries at easy and medium difficulty levels are the most appropriate as initial queries. We follow definition of difficulty levels in Yu et al. (2018). The second aspect indicates that initial queries should cover as many SQL keywords as possible. In order to satisfy both aspects, we induced 60 SQL query templates from single-round Spider and DuSQL, and each template contains some slots corresponding to masked table/column names and cell values. Given a DB, we require the initial query matches one of the templates (simplicity), and create at most one initial query for one template (diversity).

We create 5,028 valid initial queries in total, among which 1,761 are from DuSQL, and 3,267 are written from scratch by our senior annotators.

As discussed in Section 3.1, subsequent questions and queries are created by multiple randomly selected annotators, each contributing one current-round NL question and one next-round SQL query. This subsection focuses on how to create a next-round SQL query given existing context.

Similar to context-dependent QA Bertomeu et al. (2006), it is crucial to make as realistic as possible the thematic transition and context dependency between adjacent utterances, where theme refers to users’ information need, and context dependency is concerned with manners in reusing previous content. In this step (i.e., S3), we mainly consider the thematic transition, since reusing previous content is usually a natural choice for annotators. As for the context dependency information, we follow CHASE and create explicit annotation (see Section 3.6).

Seven categories of thematic transition. To capture theme change and encourage diversity, we design seven transition categories to represent the relationship between the current-round and next-round queries, i.e., $y_j$ and $y_{j+1}$, as illustrated in Table 1. Please note that we also allow annotators to compose a thematically “unrelated” query, which sometimes happens in real-world scenario.

Figure 2 illustrates concrete operations in the bottom left corner. Given $y_j$, the annotator first selects a transition category; then our annotation tool suggests several potential SQL templates according to $y_j$ and the selected category; finally the annotator selects an SQL template and fills it with DB elements to complete an SQL query.

This step is performed by our senior annotators. If an ordinary annotator makes a request to terminate a session, the annotation tool will transmit the request to a senior annotator. If the senior annotator agrees, then he must carefully review all previous questions and SQL queries, and correct all found mistakes. After that, he writes an NL question for the final-round SQL query.

In order to capture context dependency and make our dataset more widely applicable, we perform this step in a separate manner after all sessions are completed via the above four steps (S1-S4).

Each session is then assigned to one senior annotator. The annotator first goes through all NL questions, and decides whether each question is context-dependent. Then, the context-dependent question is rewritten into a corresponding context-independent one. There are in total 17,704 context-dependent questions, accounting for 65.5% of all questions (see Table 3). As a result, SeSQL can serve as a single-round Chinese text-to-SQL dataset as well, like DuSQL Wang et al. (2020b). Moreover, it can also support reserach on question completion techniques.

Context dependency types. Inspired by CHASE Guo et al. (2021) and context-dependent QA Bertomeu et al. (2006), we ask annotators to explicitly annotate the way that a context-dependent question depends on its previous questions. There are five types, i.e., independent, co-reference, ellipsis, hybrid of co-reference and ellipsis, and others. Such annotation can help us to better understand results of text-to-SQL parsers.

Annotators and Training. We recruit 28 undergraduate students as our part-time annotators, and 6 master students as senior annotators, including three co-authors of this paper. All of them come from the computer science department of our university and are familiar with the SQL language.

Before real annotation, we train all annotators for several times so that they understand the text-to-SQL parsing task, the annotation workflow, and the annotation tool, etc. During real annotation, we have also held several meetings to discuss common mistakes and settle disputes. Our annotation project lasts for about half a year.

Annotation tool. We build an online browser-based annotation tool to facilitate this work. Figure 4 in Appendix B shows the annotation interface.

Payment. All annotators were paid for their work based on the quality and quantity of their annotations. According to the annotation time recorded by our annotation tool, the average salary per hour is 25 RMB for ordinary annotators, and 35 RMB for senior annotators.³³3The average salary is about 20 RMB for a part-time KFC employee in our city. A total of 106K RMB is paid to annotators.

Overall performances. Table 5 shows the overall performances of five baseline models, where the first row shows performances of three session-level models (i.e., EditSQL, IGSQL and EX-RATSQL) on the SeSQL’s session-level data, and the second row shows performances of RATSQL and LGESQL on the single-round data of SeSQL. IGSQL and LGESQL have achieved the best performances on the session-level and single-round data, respectively. But the results on the session-level data are far from satisfactory, reflected in two aspects. First, the best performances on IM, the primary metric in the session-level setting, only achieves 29.0% on the test set. Second, the best QM accuracy achieved by IGSQL is 59.5%, where the best QM accuracy on the single-round data is 71.0%. That is, there is a large room for both QM and IM improvements on SeSQL. We believe SeSQL can facilitate the research on session-level text-to-SQL parsing.

Comparison between SeSQL and CHASE. We use different combination of SeSQL and CHASE as training data, and use three separate dev sets, in order to understand data similarity and discrepancy. To avoid DB overlap, which would corrupt the cross-DB text-to-SQL parsing task, we remove all DBs that also appears in any of the three dev sets from each training data, along with corresponding question/SQL pairs. Table 6 shows the results.

First of all, it is clear that CHASE-C and CHASE-T are highly discrepant and incompatible. Using whole CHASE as training data leads to performances drop on CHASE-C dev set, compared with using only CHASE-C. In other words, the extra CHASE-T only introduces more noisy information than helpful information. However, using whole CHASE increases performances on CHASE-T dev set, compared with using only CHASE-T.

Second, using only SeSQL as training data achieves acceptable cross-dataset performances on both CHASE-C dev set, which is much higher than using CHASE-T as training data. The same trend goes to CHASE-T dev set. This indicates that SeSQL possesses a higher level of generalization ability.

Third, using both SeSQL and CHASE-C as training data leads to higher performances on CHASE-C dev set than using only CHASE-C. Similarly, using both SeSQL and CHASE-T as training data leads to higher performances on CHASE-T dev set than using only CHASE-T as well. Such consistent improvement indicates that SeSQL is of higher quality and compatible with both CHASE-C and CHASE-T.

Finally, using either CHASE-C or CHASE-T as extra training data increases QM and IM on SeSQL dev set slightly, compared with using only SeSQL. We suspect this may be due to the increased data volume added by both datasets.

Despite SeSQL improves cross-dataset generalization, the model generalization ability across different datasets is still weak, even if these datasets are built on the same DBs (e.g., SeSQL and CHASE-C). We believe SeSQL can facilitate the research of text-to-SQL parsing, especially on the cross-dataset generalization of text-to-SQL models.

According to the fine-grained annotation information, we report QM results on SeSQL’s test set in Table 7. There are three main findings that are applicable to all baseline models. First, among all thematic transitions, all models do not perform well on transitions of Combining queries (Com.) and hybrid of other transitions. As described in Section 4, these transitions usually result in complex query generation. Second, as shown in the column of “Context Dependency”, QM performances on context-independent (Indep.) pairs is higher than that on context-dependent pairs, i.e., the other four dependency types. Furthermore, all base models do not perform well on questions that omit important historical information, i.e., labeled as “ellipse”. This proves that how to effectively use historical information is challenging. Finally, due to the difficulty increase in SQL generation, QM performances decreases as the round increases, which is consistent with the conclusions in other session-level datasets, e.g., SParC and CHASE.

Then we analyze significance of fine-grained annotations by comparing different models. From Table 7, there are three interesting findings to verify the importance of fine-grained annotations in revealing the effectiveness of model components. First, both IGSQL and EditSQL perform better than EX-RATSQL on non-first round questions, as they refer to the previous-round SQL query during the generation of the current-round SQL query. As all we known, in the context-dependent setting, the historical questions and generated SQL queries are very important to the current SQL generation. Second, IGSQL outperforms EditSQL on all transition types and dependency types, where IGSQL incorporates a graph encoder into EditSQL to model DB schema items together with items mentioned in historical questions. The performances on these fine-grained annotations verify that this graph encoder effectively captures historical information of questions and DB schema items. Third, EX-RATSQL, which only uses a relation-aware transformer to model the historical questions and DB schema items, performs best on the transition type of Tab., in which there is weak correlation between the historical rounds and the current round. Based on the above conclusions, we believe these annotations can help to reveal the advantages and limitations of the model, so as to help to improve models.