Out of the box, we're supplied with relevant questions and programming solutions we can customize a test with. It didn't take long at all for us to get a list of questions the team liked together in a test, and start testing candidates in our funnel.
There's a large pool of format of questions- we don't need to format the coding questions into weird multiple choice questions where you select the correct code as other platforms have you do. Along the same vein, we can have the candidate complete coding projects that are similar to work our developers do each day: find and fix the existing bug(s), implement a new feature, or perform a code review (similar to Bitbucket or Github pull request code reviews).
We ended up writing a few of our own questions, which wasn't difficult, that allowed us to test candidates how they'd perform against some of our own code. The code reviews especially have been great at determining the candidate's skill level based on the feedback they give.
Being able to have the candidate download/upload code from Devskiller so they can modify it in their own IDE, or allowing them to grab the code from a temporary Github repository is a big plus.
Some bugs had come up, such as not being able to turn on/off an option in a test question. These were resolved quickly in a few hours after sending an email to the support email.
The test isn't fully automated for some types of questions (as you would expect for code reviews), but there's also not a way to give automatic partial credit on multiple-choice questions. This requires us to go through the test questions individually to give partial credit (4 / 5 correct doesn't seem like a 0% to us), but the time isn't significant.
Brackets seem to break the SQL questions, which is odd and hangs some candidates up.
We've cut down the amount of in-person interviews we need to hold by 65+% by being able to realize what the level of experience the candidate demonstrates within code very similar to our own. When we do bring in a candidate, we have a good baseline idea of where their skillset it at, and can tailor the in-person interview to allow us to determine the candidate's rate-of-growth and dive deeper into what they know.