dynmap mariadb corruptions
despair86 opened this issue ยท 3 comments
[22:22:08 ERROR]: [dynmap] Tile write error - The table 'Tiles' is full
- checked mysql db, db is only 5.2GB (and with large-file support compiled in properly, it allows 32-bit db instances to run just fine)
- mysql reserves ~4MB of innodb table pages for expansion
- exhausting the reserved space triggers mysql to reserve more pages up to that limit, but temporarily returns an error for the time being.
if the first attempt fails, move the tile update to the back of the queue and try again later. current behaviour simply drops the tiles on the floor until a manual render is triggered (which may also fail for the same reason)
Otherwise, I haven't found a way to change the reserve size in mysql itself.
Data_free shows the 4MB of slack reserved for further updates
mysql> show table status from mc_dynmap;
+-----------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Faces | InnoDB | 10 | Dynamic | 28 | 585 | 16384 | 0 | 0 | 0 | NULL | 2019-11-25 16:36:19 | 2019-11-27 04:32:32 | NULL | utf8_general_ci | NULL | | |
| Maps | InnoDB | 10 | Dynamic | 6 | 2730 | 16384 | 0 | 0 | 0 | 7 | 2019-11-25 16:36:19 | 2019-11-26 03:24:29 | NULL | utf8_general_ci | NULL | | |
| MarkerFiles | InnoDB | 10 | Dynamic | 3 | 5461 | 16384 | 0 | 0 | 0 | NULL | 2019-11-25 16:36:19 | 2019-11-27 04:39:48 | NULL | utf8_general_ci | NULL | | |
| MarkerIcons | InnoDB | 10 | Dynamic | 85 | 1349 | 114688 | 0 | 0 | 0 | NULL | 2019-11-25 16:36:19 | 2019-11-26 02:50:15 | NULL | utf8_general_ci | NULL | | |
| SchemaVersion | InnoDB | 10 | Dynamic | 0 | 0 | 16384 | 0 | 0 | 0 | NULL | 2019-11-25 16:36:19 | NULL | NULL | utf8_general_ci | NULL | | |
| StandaloneFiles | InnoDB | 10 | Dynamic | 0 | 0 | 16384 | 0 | 0 | 0 | NULL | 2019-11-25 16:36:19 | NULL | NULL | utf8_general_ci | NULL | | |
| Tiles | InnoDB | 10 | Dynamic | 185199 | 28594 | 5295652864 | 0 | 0 | 4194304 | NULL | 2019-11-25 21:19:18 | 2019-11-27 04:39:51 | NULL | utf8_general_ci | NULL | | |
+-----------------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
7 rows in set (0.00 sec)
update: it is now corrupting its own database
2019-12-01 00:08:13 0x21 InnoDB: Assertion failure in thread 33 in file os0file.cc line 1699
InnoDB: Failing assertion: offset > 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
00:08:13 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=6
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 67513 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x16cf95a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
/opt/local/sbin/mysqld'my_print_stacktrace+0x29 [0x8c838d9]
/opt/local/sbin/mysqld'handle_fatal_signal+0x43a [0x844f58a]
/lib/libc.so.1'__sighndlr+0x15 [0xfe746645]
/lib/libc.so.1'call_user_handler+0x1e9 [0xfe73a1b1]
/lib/libc.so.1'_lwp_kill+0x7 [0xfe74ac97] [Signal 6 (ABRT)]
/lib/libc.so.1'raise+0x2b [0xfe6d9ca2]
/lib/libc.so.1'abort+0xfb [0xfe6b23cb]
/opt/local/sbin/mysqld'_Z18ut_print_timestampP6__FILE+0x0 [0x90ebf99]
/opt/local/sbin/mysqld'_ZL19os_file_io_completeRK9IORequestiPhS2_mmm.constprop.160+0x12b [0x8d5945b]
/opt/local/sbin/mysqld'_ZL10os_file_ioRK9IORequestiPvmyP7dberr_t+0x6e0 [0x8d5a720]
/opt/local/sbin/mysqld'_ZL17os_file_read_pageR9IORequestiPvymPmb+0x107 [0x8d5a907]
/opt/local/sbin/mysqld'_Z17os_file_read_funcR9IORequestiPvym+0x21 [0x8d5afc1]
/opt/local/sbin/mysqld'_Z11os_aio_funcR9IORequestmPKc13pfs_os_file_tPvymbP10fil_node_tS4_+0x7c [0x8d5c91c]
/opt/local/sbin/mysqld'_Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8_+0x732 [0x8f38772]
/opt/local/sbin/mysqld'_ZL17buf_read_page_lowP7dberr_tbmmRK9page_id_tRK11page_size_tb+0x11e [0x8edd63e]
/opt/local/sbin/mysqld'_Z13buf_read_pageRK9page_id_tRK11page_size_t+0x48 [0x8ede648]
/opt/local/sbin/mysqld'_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x5b3 [0x8eb72f3]
/opt/local/sbin/mysqld'_Z19xdes_get_descriptormmRK11page_size_tP5mtr_t+0x1dd [0x8f3ac4d]
/opt/local/sbin/mysqld'_ZL18fseg_free_page_lowPhRK9page_id_tRK11page_size_tbP5mtr_t.constprop.58+0x76 [0x8f3d826]
/opt/local/sbin/mysqld'_Z14fseg_free_pagePhmmbP5mtr_t+0x10d [0x8f3ed8d]
/opt/local/sbin/mysqld'_Z17btr_page_free_lowP12dict_index_tP11buf_block_tmP5mtr_t+0x65 [0x8e7d955]
/opt/local/sbin/mysqld'_Z13btr_page_freeP12dict_index_tP11buf_block_tP5mtr_t+0x30 [0x8e7d9e0]
/opt/local/sbin/mysqld'_Z12btr_compressP9btr_cur_tmP5mtr_t+0x10f9 [0x8e89429]
/opt/local/sbin/mysqld'_Z26btr_cur_compress_if_usefulP9btr_cur_tmP5mtr_t+0xbd [0x8e9220d]
/opt/local/sbin/mysqld'_Z26btr_cur_pessimistic_updatemP9btr_cur_tPPmPP16mem_block_info_tS4_PP9big_rec_tP5upd_tmP9que_thr_tyP5mtr_t+0x5f9 [0x8e9a1d9]
/opt/local/sbin/mysqld'_ZL17row_upd_clust_recmP10upd_node_tP12dict_index_tPmPP16mem_block_info_tP9que_thr_tP5mtr_t+0x2ba [0x8dfa39a]
/opt/local/sbin/mysqld'_ZL18row_upd_clust_stepP10upd_node_tP9que_thr_t+0x8dd [0x8dff0dd]
/opt/local/sbin/mysqld'_Z7row_updP10upd_node_tP9que_thr_t+0x66 [0x8e00d36]
/opt/local/sbin/mysqld'_Z12row_upd_stepP9que_thr_t+0x90 [0x8e01010]
/opt/local/sbin/mysqld'_ZL36row_update_for_mysql_using_upd_graphPKhP14row_prebuilt_t.isra.116+0x23c [0x8dbe2dc]
/opt/local/sbin/mysqld'_Z20row_update_for_mysqlPKhP14row_prebuilt_t+0x45 [0x8dc2905]
/opt/local/sbin/mysqld'_ZN11ha_innobase10update_rowEPKhPh+0xf4 [0x8ccad74]
/opt/local/sbin/mysqld'_ZN7handler13ha_update_rowEPKhPh+0x204 [0x84b1db4]
/opt/local/sbin/mysqld'_Z12mysql_updateP3THDR4ListI4ItemES4_y15enum_duplicatesPyS6_+0x1eec [0x8a9431c]
/opt/local/sbin/mysqld'_ZN14Sql_cmd_update23try_single_table_updateEP3THDPb+0x21a [0x8a96aea]
/opt/local/sbin/mysqld'_ZN14Sql_cmd_update7executeEP3THD+0x44 [0x8a96eb4]
/opt/local/sbin/mysqld'_Z21mysql_execute_commandP3THDb+0x436d [0x89ffb7d]
/opt/local/sbin/mysqld'_Z11mysql_parseP3THDP12Parser_state+0x40b [0x8a0423b]
/opt/local/sbin/mysqld'_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1cdd [0x8a0602d]
/opt/local/sbin/mysqld'_Z10do_commandP3THD+0x29a [0x8a0724a]
/opt/local/sbin/mysqld'handle_connection+0x329 [0x8aded59]
/opt/local/sbin/mysqld'pfs_spawn_thread+0x1b0 [0x9011fc0]
/lib/libc.so.1'_thrp_setup+0x81 [0xfe7462f1]
/lib/libc.so.1'_lwp_start+0x0 [0xfe7465a0]
Please read http://dev.mysql.com/doc/refman/5.7/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the
problem, so please do resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (16c215d0): UPDATE dynmap_Tiles SET HashCode=364739319, LastUpdate=1575158893347, Format=0, Image=_binary'<89>PNG
^Z
\0\0\0^MIHDR\0\0\0<80>\0\0\0<80>^H^F\0\0\0<C3>>a<CB>\0\01<88>IDATx^<ED><9D><89>sdW<95><E6><F9>^S<86><80><98><9E>6<D8><C6><B5><97>TU<DA><F7>%<A5>LI<99>R*<B5><EF>*<A9><B4><94>j<DF><CB><B5>y<93>m<BC>b6<B7><C1><98><D5><CB>\0n:p<D3><D0><AC>M<83>Y^F<DA>^N<C0>40^Xz`<8C><89>a<8D><89><89> <9A><98><98><98><98><A0><CF><DC><DF>y<F9>e=\'<B2>MO<83><B1>D<DE><88>ESC/<F3><BD><97>o<B9><E7>;<DF>Y<EE><92><AF>xE<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><94>J<A9><BC><9C>Ks^?<D3><E3><C5><FB>J<E5>O<A8>t<CF>vY<F1><BE>R<F9>^S(e<8D>e<AB><FD>+<FD><B6>/<B5><BB>^D<80>?<A5><B2><B3>a<E7><EA><D0><91>A<EB>[<EA><B5><C1><C3><83>v SU^B<C0><9F>B<D9>Z<B5>uu<EA><DC><A4><A1><F5>#<C7>G\\<F8><B9><F0><F9><F4>PC \0ESC<B9> <F8><DE><C5>^ESC8<98>s
<AD>^_;9j<B9>^C9<CB>.<F7><F9>v<A6><A3><BC>^D<80><8D>X6Wl^<9D><BD>8c^S<A7><C7><83><F0>^Gl<F8><E8><B0>k~<EF>B<C6><D8><DF><B3><B7><DB><F7>g<F6>eJ\0<D8>HeG<ED><F6><D5>\\<D0><F6><FE><FD>Y<C3><D6>G<9A><DE>o<C3><C7><86>/^?^O<D4><EF><F5>\05W^B<C0>F([<82><C6>^O^_^]2<E8>^^!c<EB><B1><F3><83><87><D0><FE>!<D7>~<F6><C3>\^_s<80>Y(<85><81><EB><BC>T\'<AB>V<B1><EB>^H{<E0>@<A4><F9>|w<A1>^_ESC<B1><F4>|<8F>e<82><D0>G^B^C<A0><ED>h}W^Pzz><ED>`^Y9>\\^B<C0>z,e!<9C>ESC^O<F6>^]ESC^N<BD>#`<84><DE>
<BF><92><B5><BE> <D8><CC><BE>t<A0><FC>!?<DE><BB>^X<CE> <9F>S<D3>)KN%^K<C7>{^Wz<AD>g<AE><A7>^D<80><F5>T<88><E3><D1><F6><FE><FD><FD>N<E1>h<<9A>?~j<CC>^E<CE><B1><D1>@<F7><A3>\'Fl<E8><F0><90>^_<C7>\'<80>^Ed\"0^O<80>^A<E1><B3><AF><F8>^^<A5><F2>2,d<EE>^P.^=T>xx<C0>^E<89>M<C7><B9><9B>83<E1>v<BE>;x<F7>^H<B5>o<A9><CF><B5>^\<90><B0>^]>6d<E9><B0>^E^LD\0<CE>^^<E1><B7>^C<C1>G(<BE>W<A9><BC><8C><CA><D6><EA><AD><AB>^H^Q[<8D>V#t<BE>#t<B7><F5><81><FA><D9><87><D0><B1><F7>h56<DE><C1>^Q<B4>^]g/ESCX<A0>?<D8>~^D^N``^E>c.<B2><FB><FB>J\0x9<96><DD>-<BB>V<A1>v<B4>z<EC><E4><98>k=<B6>{<E2><CC><B8>S>l<80>^P^Q:<9F>^A\0<C0>p<9A>_<88>^D^_<B1>D<D6>?^C
<CC>^D[<80>At<C0><EF><89>^H<8A><EF>]*^?<84><D2><DA><DA><9A><F8><F8>\'?<F9><F8><A5>K<97>V <D7>f<CE>O<DB><D8><89>(k<87>#<87><B6><A3><BD><D8>w<84><CC>9<84>p<D0>:l<E0>4^_<F6>s>f<80><EF>^H<9B><DF><FA><F7><F0><DB><C8>^T^L;\0<F8>^L@8V<FC>,<A5><F2>^R^V <FE>^?<FE><F3>?<DB><CF>^?<F5>+<FB>e<BE>><F0><97><EF>q<C1><A2><A1><D3><E7><A6><82>C^Wi9<DB><C9><B3><93>^A\0!<DC>
Connection ID (thread ID): 5847
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2019-12-01T00:08:13.809721Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-12-01T00:08:13.810027Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2019-12-01T00:08:13.810112Z 0 [Note] /opt/local/sbin/mysqld (mysqld 5.7.24) starting as process 72950 ...
2019-12-01T00:08:13.824076Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-12-01T00:08:13.824190Z 0 [Note] InnoDB: Uses event mutexes
2019-12-01T00:08:13.824270Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-12-01T00:08:13.824349Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-12-01T00:08:13.831173Z 0 [Note] InnoDB: Number of pools: 1
2019-12-01T00:08:13.831486Z 0 [Note] InnoDB: Not using CPU crc32 instructions
2019-12-01T00:08:13.834427Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-12-01T00:08:13.865424Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-12-01T00:08:13.933098Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2019-12-01T00:08:13.933795Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3744579676
2019-12-01T00:08:13.964820Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3749822464
2019-12-01T00:08:13.984166Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3753207313
2019-12-01T00:08:13.984481Z 0 [Note] InnoDB: Database was not shutdown normally!
