I may be misinterpreting the point of custom layouts.
At the moment in mythic raiding my layout is set to "By Role". This ends up being 2 tanks followed by 3 healers in the first row. Then one healer, followed by dps in the second row.
I've been trying to set up a custom layout to have one tank in the first row, followed by 4 healers, then the other tank in the second row followed by dps. Is this something I could set up? I've been unsuccessful so far

1. Goto the layout editor, create a custom layout.
2. Add 2 extra headers of type player (click in the + Tab to add new headers)
3. First Header: roles section: check Tank, uncheck other roles
4. Second header, roles section: check Healer, uncheck other roles
5. Third header: roles section, check dps&none roles, uncheck tank&healer roles.
6. You must increase the units per column for the third headers (the dps header), to be able to display more than 5 players, because by default only one column and five rows are displayed).
7. Dont forget to enable the new created custom layout in "Layouts> By Group" tab.

Thanks for the reply!
I tried that now and If I do that it puts two tanks in the first row ("group 1"), no one else. Then it puts the healers in second row ("group 2").
I also tried making the second header also only show tanks, but it just becomes a mirror of the first header with same players in both.

My goal would be something like this:

Following your instructions I got something like this:

Any ideas on what I might be doing wrong? I can screenshot the layout if it would help.

Sorry this kind of layout is not possible, Its not possible to put tanks in a column, healers in a row and fill all the holes with dps.
The more similar layout would be something like:

t1 h1 h2 h3 h4 d1
t2 d2 d3 d4 d5 d6 
xx d7 d9 d9 da db
xx dc dd de df dg 

tx = tanks
hx = healers
dx = dps/any
xx = holes (nothing displayed)
And to define this layout the healers&dps header must be detached from the tanks header (Using the detach header option in the editor). When headers are detached they can be moved individually.

Ah okay thank you 👍 I'll stick with the normal "By Role" layout