Add basic framework for a performance testing suite
rdw-software opened this issue ยท 0 comments
Goals:
- Can be run via CLI (no dependency on the WOW API)
- Can define maximum threshold after which the test is considered failed (optional)
- Reports the results with millisecond precision
- Should be run automatically on push/PR via a separate GitHub action
- Should use only standard Lua so that we can run it ingame, too (optionally)
- The Performance Options that are added as part of #398 should include a button to run the full suite ingame
- Optional: Can compare performance for Lua 5.1 and LuaJIT
The reason we want this is to benchmark sorting offline, and ensure it stays under a set threshold (that we consider acceptable in terms of User Experience) even when more items are added in future patches.
Having an offline suite to run on different systems should also give us some data points for seeing how older hardware might be affected. Additionally, we might want to see whether there's a a significant difference in performance inside the game vs. outside, so being able to run it ingame is another requirement (no external DLLs or Lua libraries that rely on functionality that isn't also available ingame).
Come to think of it, we likely need to support two timing methods, GetTimePreciseSec()
(ingame) and os.time()
(or a higher precision clock?) offline. Not sure about this yet, so this part remains TBD.