MySQL一个innodb_thread_concurrency设定不当引发的故障

一、问题来源
欢迎关注我的《深入理解MySQL主从原理 32讲 》，如下：
这是一个朋友问我的典型案例。整个故障现象表现为，MySQL数据库频繁的出现大量的请求不能响应。下面是一些他提供的证据：
1、show processlist
从状态信息来看出现如下情况：
insert操作：状态为update
update/delete操作：状态为updating
select操作：状态为sending data
因此可以推断应该是语句执行期间出现了问题，由于篇幅原因只给出一部分，并且我将语句部分也做了相应截断：
 
show processlist----------------------------
......
11827639    root    dbmis    Execute    9    updating    UPDATE
17224594    root    dbmis    Execute    8    Sending data    SELECT sum(exchange_coin) as exchange_coin FROM
17224595    root    dbmis    Execute    8    update    INSERT INTO
17224596    root    dg    Execute    8    update    INSERT INTO
17224597    root    dbmis    Execute    8    update    INSERT INTO
17224598    root    dbmis    Execute    7    update    INSERT INTO
17224599    root    dbmis    Execute    7    Sending data    SELECT COUNT(*) AS tp_count FROM
17224600    root    dg    Execute    7    update    INSERT INTO
17224601    root    dbmis    Execute    6    update    INSERT INTO
17224602    root    dbmis    Execute    6    Sending data    SELECT sum(exchange_coin) as exchange_coin FROM
17224606    root    dbmis    Execute    5    update    INSERT INTO
17224619    root    dbmis    Execute    2    update    INSERT INTO
17224620    root    dbmis    Execute    2    update    INSERT INTO
17224621    root    dbmis    Execute    2    Sending data    SELECT sum(exchange_coin) as exchange_coin
17224622    root    dg    Execute    2    update    INSERT INTO
17224623    root    dbmis    Execute    1    update    INSERT INTO
17224624    root    dbmis    Execute    1    update    INSERT INTO
17224625    root    dg    Execute    1    update    INSERT INTO
17224626    root    dbmis    Execute    0    update    INSERT INTO
2、系统IO/CPU
从vmstat来看，CPU使用不大，而IO也在可以接受的范围内（vmstat wa%不高且b列为0）如下：
 
vmstat--------------------------------------
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0 927300 3057100      0 53487316    0    0     5   192    0    0  3  1 96  0  0
iostat--------------------------------------
Linux 3.10.0-693.el7.x86_64 (fang-data1)     09/23/2019     _x86_64_    (32 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.72    0.00    0.52    0.45    0.00   96.31
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               9.73    11.28    3.93  264.54   415.23  2624.20    22.64     0.25    0.93    3.25    0.90   0.80  21.61
sda              10.13    11.59    6.34  264.22   450.68  2624.20    22.73     0.01    0.05    2.55    1.00   0.93  25.19
sdc              11.60    11.36    5.03  263.12   453.02  2592.44    22.71     0.17    0.62    5.08    0.53   0.81  21.60
sde               0.01     0.10    0.11  160.45     6.69   920.23    11.55     0.16    1.01    1.80    1.01   0.83  13.32
sdd              11.26    11.30    2.23  263.18   412.90  2592.44    22.65     0.17    0.65   10.37    0.56   0.82  21.78
md126             0.00     0.00   11.30  468.80   164.79  5216.64    22.42     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.11   58.80     6.69   920.23    31.47     0.15    2.56    1.96    2.56   2.16  12.74
dm-1              0.00     0.00    0.06    0.08     0.24     0.31     8.00     0.01   41.80    1.20   72.78   0.83   0.01
dm-2              0.00     0.00   11.24  408.66   164.55  5216.33    25.63     0.14    0.32    1.02    0.30   0.46  19.29
这就比较奇怪了，一般来说数据库不能及时响应请求很大可能是由于系统负载过高。如果说DML还可能是Innodb锁造成的堵塞，但是大量sending data状态下的select操作一般可能都和系统负载过高有联系，但是这里系统负载还在可以接受的范围内。
 
二、pstack分析
借助pstack查看线程的栈帧，查看pstack发现如下（由于篇幅限制只给出部分说明问题的部分）：
 
1、insert 线程：
 
Thread 85 (Thread 0x7fbb0d42b700 (LWP 20174)):
#0  0x00007fbfae164c73 in select () from /lib64/libc.so.6
#1  0x0000000000987c0f in os_thread_sleep (tm=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/os/os0thread.cc:287
#2  0x00000000009e4dea in srv_conc_enter_innodb_with_atomics (trx=trx@entry=0x7fba4802f9c8) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/srv/srv0conc.cc:276
#3  srv_conc_enter_innodb (trx=trx@entry=0x7fba4802f9c8) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/srv/srv0conc.cc:511
#4  0x000000000093b948 in innobase_srv_conc_enter_innodb (trx=0x7fba4802f9c8) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/handler/ha_innodb.cc:1280
#5  ha_innobase::write_row (this=0x7fb8440ab260, record=0x7fb8440ab650 "") at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/handler/ha_innodb.cc:6793
#6  0x00000000005b440f in handler::ha_write_row (this=0x7fb8440ab260, buf=0x7fb8440ab650 "") at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:7351
#7  0x00000000006dd3a8 in write_record (thd=thd@entry=0x1d396c90, table=table@entry=0x7fb8440aa970, info=info@entry=0x7fbb0d429400, update=update@entry=0x7fbb0d429480) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_insert.cc:1667
#8  0x00000000006e2541 in mysql_insert (thd=thd@entry=0x1d396c90, table_list=<optimized out>, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_REPLACE, ignore=false) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_insert.cc:1072
#9  0x00000000006fa90a in mysql_execute_command (thd=thd@entry=0x1d396c90) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_parse.cc:3500
2、update线程
 
Thread 81 (Thread 0x7fbb24b67700 (LWP 27490)):
#0  0x00007fbfae164c73 in select () from /lib64/libc.so.6
#1  0x0000000000987c0f in os_thread_sleep (tm=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/os/os0thread.cc:287
#2  0x00000000009e4dea in srv_conc_enter_innodb_with_atomics (trx=trx@entry=0x7fb94003c608) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/srv/srv0conc.cc:276
#3  srv_conc_enter_innodb (trx=trx@entry=0x7fb94003c608) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/srv/srv0conc.cc:511
#4  0x000000000093ae4e in innobase_srv_conc_enter_innodb (trx=0x7fb94003c608) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/handler/ha_innodb.cc:1280
#5  ha_innobase::index_read (this=0x7fb95c05b540, buf=0x7fb95c2ae4f0 "/377/377/377", key_ptr=<optimized out>, key_len=<optimized out>, find_flag=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/handler/ha_innodb.cc:7675
#6  0x00000000005ab6e0 in ha_index_read_map (find_flag=HA_READ_KEY_EXACT, keypart_map=3, key=0x7fb940017048 "7/307/017e/257h", buf=<optimized out>, this=0x7fb95c05b540) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:2753
#7  handler::read_range_first (this=0x7fb95c05b540, start_key=<optimized out>, end_key=<optimized out>, eq_range_arg=<optimized out>, sorted=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:6717
#8  0x00000000005aa206 in handler::multi_range_read_next (this=0x7fb95c05b540, range_info=0x7fbb24b65240) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:5871
#9  0x0000000000804acb in QUICK_RANGE_SELECT::get_next (this=0x7fb94000f720) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/opt_range.cc:10644
#10 0x000000000082ae2d in rr_quick (info=0x7fbb24b65410) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/records.cc:369
#11 0x0000000000766e1b in mysql_update (thd=thd@entry=0x1d1f2250, table_list=<optimized out>, fields=..., values=..., conds=0x7fb9400009c8, order_num=<optimized out>, order=<optimized out>, limit=18446744073709551615, handle_duplicates=DUP_ERROR, ignore=false, found_return=found_return@entry=0x7fbb24b65800, updated_return=updated_return@entry=0x7fbb24b65d60) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_update.cc:744
3、select线程
 
Thread 66 (Thread 0x7fbb3c355700 (LWP 16028)):
#0  0x00007fbfae164c73 in select () from /lib64/libc.so.6
#1  0x0000000000987c0f in os_thread_sleep (tm=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/os/os0thread.cc:287
#2  0x00000000009e4dea in srv_conc_enter_innodb_with_atomics (trx=trx@entry=0x7fb988354858) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/srv/srv0conc.cc:276
#3  srv_conc_enter_innodb (trx=trx@entry=0x7fb988354858) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/srv/srv0conc.cc:511
#4  0x000000000093ae4e in innobase_srv_conc_enter_innodb (trx=0x7fb988354858) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/handler/ha_innodb.cc:1280
#5  ha_innobase::index_read (this=0x7fb9880e33a0, buf=0x7fb988351b50 "/377/377/377/377", key_ptr=<optimized out>, key_len=<optimized out>, find_flag=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/storage/innobase/handler/ha_innodb.cc:7675
#6  0x00000000005ab6e0 in ha_index_read_map (find_flag=HA_READ_AFTER_KEY, keypart_map=7, key=0x7fb988134a48 "", buf=<optimized out>, this=0x7fb9880e33a0) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:2753
#7  handler::read_range_first (this=0x7fb9880e33a0, start_key=<optimized out>, end_key=<optimized out>, eq_range_arg=<optimized out>, sorted=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:6717
#8  0x00000000005aa206 in handler::multi_range_read_next (this=0x7fb9880e33a0, range_info=0x7fbb3c353400) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/handler.cc:5871
#9  0x0000000000804acb in QUICK_RANGE_SELECT::get_next (this=0x7fb988002050) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/opt_range.cc:10644
#10 0x000000000082ae2d in rr_quick (info=0x7fb98809c210) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/records.cc:369
#11 0x00000000006d44fd in sub_select (join=0x7fb98809a728, join_tab=0x7fb98809c180, end_of_records=<optimized out>) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_executor.cc:1259
#12 0x00000000006d2823 in do_select (join=0x7fb98809a728) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_executor.cc:936
#13 JOIN::exec (this=0x7fb98809a728) at /home/install/lnmp1.5/src/mysql-5.6.40/sql/sql_executor.cc:194
好了有了这些栈帧视乎发现一些共同点他们都处于innobase_srv_conc_enter_innodb函数下，本函数正是下面参数实现的方式：
 
innodb_thread_concurrency
innodb_concurrency_tickets
所以我随即告诉他检查这两个参数，如果设置了可以尝试取消。过后数据库故障得到解决。
 
三、参数和相关说明
实际上涉及到的参数主要是innodb_thread_concurrency和innodb_concurrency_tickets。将高压力下线程之间抢占CPU而造成线程上下文切换的情况尽量阻塞在Innodb层之外，这就需要innodb_thread_concurrency参数了。同时又要保证对于那些（长时间处理线程）不会长时间的堵塞（短时间处理线程），比如某些select操作需要查询很久，而某些select操作查询量很小，如果等待（长时间的select操作）结束后（短时间select操作）才执行，那么显然会出现（短时间select操作）饥饿问题，换句话说对（短时间select操作）是不公平的， 因此就引入了innodb_concurrency_tickets参数。
 
1、innodb_thread_concurrency
同一时刻能够进入Innodb层的会话（线程）数。如果在Innodb层干活的会话（线程）数量超过这个参数的设置，新会话（线程）将不能从MySQL层进入到Innodb层，它们将进入一个短暂的睡眠状态。休眠多久则通过参数innodb_thread_sleep_delay参数指定，如果还设置了参数innodb_adaptive_max_sleep_delay那么Innodb将会自动调整休眠时间，具体的算法实际上就在srv_conc_enter_innodb_with_atomics函数中，感兴趣的可以执行查看。
其次这种休眠实际上是一个定时醒来的时钟，通过::nanosleep或者select（多路IO转接函数）进行实现，定时唤醒后会话（线程）重新判断是否可以进入Innodb层。函数os_thread_sleep部分如下：
 
#elif defined(HAVE_NANOSLEEP)
    struct timespec    t;
    t.tv_sec = tm / 1000000;
    t.tv_nsec = (tm % 1000000) * 1000;
    ::nanosleep(&t, NULL);
#else
    struct timeval  t;
    t.tv_sec = tm / 1000000;
    t.tv_usec = tm % 1000000;
    select(0, NULL, NULL, NULL, &t);
关于到底如何设置这个值，官方文档有如下建议：
 
Use the following guidelines to help find and maintain an appropriate setting:
- If the number of concurrent user threads for a workload is less than 64, set
innodb_thread_concurrency=0.
- If your workload is consistently heavy or occasionally spikes, start by setting
innodb_thread_concurrency=128 and then lowering the value to 96, 80, 64, and so on, until
you find the number of threads that provides the best performance. For example, suppose your
system typically has 40 to 50 users, but periodically the number increases to 60, 70, or even 200.
You find that performance is stable at 80 concurrent users but starts to show a regression above
this number. In this case, you would set innodb_thread_concurrency=80 to avoid impacting
performance.
- If you do not want InnoDB to use more than a certain number of virtual CPUs for user threads
(20 virtual CPUs, for example), set innodb_thread_concurrency to this number (or possibly
lower, depending on performance results). If your goal is to isolate MySQL from other applications,
you may consider binding the mysqld process exclusively to the virtual CPUs. Be aware,
however, that exclusive binding could result in non-optimal hardware usage if the mysqld process
is not consistently busy. In this case, you might bind the mysqld process to the virtual CPUs but
also allow other applications to use some or all of the virtual CPUs.
- innodb_thread_concurrency values that are too high can cause performance regression due
to increased contention on system internals and resources.
- In some cases, the optimal innodb_thread_concurrency setting can be smaller than the
number of virtual CPUs.
- Monitor and analyze your system regularly. Changes to workload, number of users, or computing
environment may require that you adjust the innodb_thread_concurrency setting
可以发现要合理的设置这个值并不那么容易并且要求较高。
 
2、innodb_concurrency_tickets
实际上这里的tickets可以理解为MySQL层和Innodb层交互的次数，比如一个select一条数据就是需要Innodb层返回一条数据然后MySQL层进行where条件的过滤然后返回给客户端，抛开where条件过滤的情况，如果我们一条语句需要查询100条数据，那么实际上需要进入Innodb层100次，那么实际上消耗的tickets就是100。当然对于insert select这种操作，需要的tickets是普通select的两倍，因为查询需要进入Innodb层一次，insert需要再次进入Innodb层一次，后面我们就使用insert select的方式来模拟堵塞的情况，最后还会给出说明。
 
这样我们也就理解为什么innodb_concurrency_tickets可以避免（长时间处理线程）长时间堵塞（短时间处理线程）的原因了。假设innodb_concurrency_tickets为5000（默认值），有一个需要查询100W行数据的大select操作和一个需要查询100行数据的小select操作，大select操作先进行，但是当查询了5000行数据后将丢失CPU使用权，小select操作将会进行并且一次性完成。
 
最后关于这里涉及的参数可以继续参考官方文档中的说明，我们线上并没有设置这些参数，因为感觉很难设置合适，如果设置不当反而会遇到问题，就如本案例一样。
 
3、事务操作状态
实际上如果是处于这种堵塞情况，我们完全可以在information_schema.innodb_trx和show engine innodb status中看到如下：
 
---TRANSACTION 162307, ACTIVE 133 sec sleeping before entering InnoDB （这里）         
mysql tables in use 2, locked 2
767 lock struct(s), heap size 106968, 212591 row lock(s), undo log entries 15451
MySQL thread id 14, OS thread handle 140736751912704, query id 1077 localhost root Sending data
insert into testui select * from testui
---TRANSACTION 162302, ACTIVE 320 sec, thread declared inside InnoDB 1
mysql tables in use 2, locked 2
2477 lock struct(s), heap size 336344, 609049 row lock(s), undo log entries 83582
MySQL thread id 13, OS thread handle 140737153779456, query id 1050 localhost root Sending data
insert into testti3 select * from testti3
mysql> select trx_id,trx_state,trx_query,trx_operation_state,trx_concurrency_tickets from information_schema.innodb_trx /G
*************************** 1. row ***************************
                 trx_id: 84325
              trx_state: RUNNING
              trx_query: insert into  baguait4 select * from testgp
    trx_operation_state: sleeping before entering InnoDB（这里）
trx_concurrency_tickets: 0
*************************** 2. row ***************************
                 trx_id: 84319
              trx_state: RUNNING
              trx_query: insert into  baguait3 select * from testgp
    trx_operation_state: sleeping before entering InnoDB
trx_concurrency_tickets: 0
我们可以看到事务操作状态被标记为‘sleeping before entering InnoDB’。但是需要注意一点的是对于只读事务比如select操作而言，show engine innodb status可能看不到。但是遗憾的是案例中朋友并没有采集trx_operation_state的值。
 
四、模拟测试
这里我们简单模拟，我们一共启用3个事务，其中两个insert select操作，一个单纯的select操作，当然这里的都是耗时操作，涉及的表每个表都有大概100W的数据。
 
同时为了方便观察我们需要设置参数：
 
innodb_thread_concurrency=1
innodb_concurrency_tickets=10
操作步骤如下：
 
S1 S2 S3
insert into baguait4 select * from testgp
insert into baguait3 select * from testgp
select * from baguait1
如果多观察几次你可以看到如下的现象：
 
mysql> select trx_id,trx_state,trx_query,trx_operation_state,trx_concurrency_tickets from information_schema.innodb_trx /G show processlist;
*************************** 1. row ***************************
                 trx_id: 84529
              trx_state: RUNNING
              trx_query: insert into  baguait4 select * from testgp
    trx_operation_state: sleeping before entering InnoDB
trx_concurrency_tickets: 0
*************************** 2. row ***************************
                 trx_id: 84524
              trx_state: RUNNING
              trx_query: insert into  baguait3 select * from testgp
    trx_operation_state: inserting
trx_concurrency_tickets: 1
*************************** 3. row ***************************
                 trx_id: 422211785606640
              trx_state: RUNNING
              trx_query: select * from baguait1
    trx_operation_state: sleeping before entering InnoDB
trx_concurrency_tickets: 0
3 rows in set (0.00 sec)
+----+-----------------+-----------+---------+---------+------+------------------------+--------------------------------------------+-----------+---------------+
| Id | User            | Host      | db      | Command | Time | State                  | Info                                       | Rows_sent | Rows_examined |
+----+-----------------+-----------+---------+---------+------+------------------------+--------------------------------------------+-----------+---------------+
|  1 | event_scheduler | localhost | NULL    | Daemon  | 3173 | Waiting on empty queue | NULL                                       |         0 |             0 |
|  6 | root            | localhost | testmts | Query   |   70 | Sending data           | insert into  baguait3 select * from testgp |         0 |             0 |
|  7 | root            | localhost | testmts | Query   |   68 | Sending data           | insert into  baguait4 select * from testgp |         0 |             0 |
|  8 | root            | localhost | testmts | Query   |   66 | Sending data           | select * from baguait1                     |    120835 |             0 |
|  9 | root            | localhost | NULL    | Query   |    0 | starting               | show processlist                           |         0 |             0 |
+----+-----------------+-----------+---------+---------+------+------------------------+--------------------------------------------+-----------+---------------+
5 rows in set (0.00 sec)
mysql>
mysql>
mysql>
mysql>
mysql> select trx_id,trx_state,trx_query,trx_operation_state,trx_concurrency_tickets from information_schema.innodb_trx /G show processlist;
*************************** 1. row ***************************
                 trx_id: 84529
              trx_state: RUNNING
              trx_query: insert into  baguait4 select * from testgp
    trx_operation_state: sleeping before entering InnoDB
trx_concurrency_tickets: 0
*************************** 2. row ***************************
                 trx_id: 84524
              trx_state: RUNNING
              trx_query: insert into  baguait3 select * from testgp
    trx_operation_state: sleeping before entering InnoDB
trx_concurrency_tickets: 0
*************************** 3. row ***************************
                 trx_id: 422211785606640
              trx_state: RUNNING
              trx_query: select * from baguait1
    trx_operation_state: fetching rows
trx_concurrency_tickets: 3
3 rows in set (0.00 sec)
+----+-----------------+-----------+---------+---------+------+------------------------+--------------------------------------------+-----------+---------------+
| Id | User            | Host      | db      | Command | Time | State                  | Info                                       | Rows_sent | Rows_examined |
+----+-----------------+-----------+---------+---------+------+------------------------+--------------------------------------------+-----------+---------------+
|  1 | event_scheduler | localhost | NULL    | Daemon  | 3177 | Waiting on empty queue | NULL                                       |         0 |             0 |
|  6 | root            | localhost | testmts | Query   |   74 | Sending data           | insert into  baguait3 select * from testgp |         0 |             0 |
|  7 | root            | localhost | testmts | Query   |   72 | Sending data           | insert into  baguait4 select * from testgp |         0 |             0 |
|  8 | root            | localhost | testmts | Query   |   70 | Sending data           | select * from baguait1                     |    128718 |             0 |
|  9 | root            | localhost | NULL    | Query   |    0 | starting               | show processlist                           |         0 |             0 |
+----+-----------------+-----------+---------+---------+------+------------------------+--------------------------------------------+-----------+---------------+
5 rows in set (0.00 sec)
我们可以观察到trx_operation_state的状态3个操作都在交替的变化，但是总有2个处于‘sleeping before entering InnoDB’状态。并且我们可以观察到trx_concurrency_tickets总是不会大于10的。因此我们有理由相信在同一时刻只有一个操作进入了Innodb层。但是需要注意的是在show engine innodb status中观察不到select的操作如下：
 
------------
TRANSACTIONS
------------
Trx id counter 84538
Purge done for trx's n:o < 84526 undo n:o < 0 state: running but idle
History list length 356
Total number of lock structs in row lock hash table 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 422211785609424, not started
0 lock struct(s), heap size 1160, 0 row lock(s)
本文最后更新于 2023-3-16 14:15，某些文章具有时效性，若有错误或已失效，请在网站留言或联系站长：17tui@17tui.com

相关分类

Mysql Access MsSql oracle SQLite教程 MongoDB Redis 其它数据库

热门排行

最近更新