Zume

Zume

Mods
4M Downloads

Add Automated Smoke Testing

Nolij opened this issue ยท 4 comments

commented

Tested platforms should include:

  • Fabric
    • Latest Snapshot
    • Latest Release
    • 21.1
    • 20.1
    • 18.2
    • 16.5
    • 14.4
  • Legacy Fabric
    • 12.2
    • 8.9
    • 7.10
  • Babric b7.3
  • NeoForge
    • Latest Release
    • Latest Snapshot (if snapshot builds are usable when this is implemented)
    • 21.1
    • 20.4
    • 20.1
  • LexForge
    • 20.5
    • 20.4
    • 20.1
    • 19.2
    • 18.2
    • 16.5
    • 14.4
    • 12.2
    • 8.9
    • 7.10

Testing should verify that each listed version & platform launches without crashing.

Additionally, the following are nice to haves that may only be applied to the most-used versions:

  • Screenshot testing
  • Config reload testing

Ideally this should be able to be performed locally as I imagine launch testing alone would take an extremely long time in CI on GitHub-provided runners.

commented

Would it be a good idea to run multiple instances of Minecraft concurrently to improve testing time?

commented

Would it be a good idea to run multiple instances of Minecraft concurrently to improve testing time?

This would probably be infeasible on GitHub runners, but on a sufficiently powerful PC it should be very doable, hence my desire for it to be able to be run locally. Ideally I'll run all the tests at the same time on my PC.

commented

This would probably be infeasible on GitHub runners

You can use a matrix action to run multiple runners at once; up to 20 concurrently running jobs are allowed per project if I recall correctly.

commented

We can use https://github.com/coactions/setup-xvfb to run the game in a GitHub action without any special configuration of OpenGL, but we'll need a headless launcher so that we can test production builds. Testing deobf builds run through Gradle is probably unwise for Zume due to the obfuscation/jar compression that often breaks things.