When I'm going through updates to give karma, I'm usually using fedora-easy-karma. While it's a great tool, the information that it provides is a but raw and doesn't provide much in the way of information if you're not already familiar with the package under test. When I'm testing something that I'm not familiar with, I usually end up switching back and forth between fedora-easy-karma, other console windows to run 'yum info' for package information and repoquery for dependency info in addition to multiple browser windows for bugs and test cases.
I'd love to have a tool that shows all of that information in one place so that the karma-giving process is a little easier. It would be similar to fedora-easy-karma but taking the concept one step farther - creating a GUI tool that pulls together more information about the package to test. This information can include (but is not limited to):
Installed/Available packages for testing
Dependency tree
Existing test cases
Bodhi Comments
Package Description
Using my most un-awesome Inkscape skills, I put together a very rough mockup of what the interface could look like (source svg file):
This has been proposed at least once but we've ended up spending resources elsewhere (testing Fedora and AutoQA). While update karma isn't always the most popular topic, a well written GUI front end would make the process of testing updates less painful.
Therefore, I'm proposing a f-e-k GUI as an idea for Fedora as part of Google Summer of Code. If you're interested in the idea (either as a potential student or interested party), let me know!
I'm a big fan of using virtualenv to keep development environments separated and want to use IPython notebooks with those virtualenvs. One might think that you could just install ipython and go but it turns out that there are a couple other steps needed to make this work.
Before working with the virtualenv, we need zeromq-devel before we can build and install pyzmq in the virtualenv
yum install zeromq-devel
To create a new virtualenv that's ready for IPython notebook:
While I could sit here and write pages on what I learned at PyCon, I
think that it would be a bit more productive and readable to list a few
of the highlights. Going into PyCon, my major interests were testing
technique (for AutoQA), data analysis and visualization (mostly for
school right now but I'm hoping to make that useful for Fedora). Most of
the data analysis stuff I learned can't be summarized well as bullet points but it should show up as I make progress on my school projects
and start making those results more public.
One of the biggest non-python things I learned while at PyCon was about the need to coordinate with others before an event like PyCon. Outside of FUDCon, PyCon 2012 was my first conference and I wasn't quite sure what to expect. While talking with other Fedora contributors (lmacken, threebean, dmalcom), I really wish that I had stayed for the sprints following PyCon. In particular, it would have been nice to work with Ralph and Luke on moksha and learn a bit more about what they're planning for Fedora Messaging and Bodhi 2.0 but it isn't the end of the world.
Pandas is really powerful and relatively easy to use
People are working on better ways to record presentations and facilitate remote presentations
I will be talking about this more in other posts
I tried working with dmalcom to start creating an AutoQA test for the python extension analysis code he's been working on and discovered that I don't know nearly enough about creating new tests for AutoQA. We found some oddities in there that I don't quite understand but I'm looking into whether that is an issue with the code or just a lack of experience on my part
Testing Technique
If you need more human time to analyse and process results than you need to actually run your tests, you have failed
I think that this is applicable to a point - sometimes it doesn't make sense to automate but at the same time,
Fine grained unit tests around bad code will tend to "bake in" the crappy API that is already there. Writing tests around bad code is great as a safety net for refactoring but don't take it too far
Overly complicated unit test setup is a test smell - either you're doing something wrong in the tests or the code under test needs to be refactored
Unfortunately, this tends to be what some of my tests end up looking like. I'm not sure why this never registered as a code smell but it will now
You should care for your unit tests and mocks like you would care for your code base
If the tests are difficult to run or take too long, they will be ignored. If they aren't testing what you think they're testing, they're useless.
What am I going to do with all of this?
Learn how to write new AutoQA tests and make sure that our documentation is good
Even though we can't support external tests well right now, I still want to understand what it would take to write and submit a new test in order to get a better idea of where the pain points are and what should change in AutoQA so that we can start accepting external tests.
Start looking at how we can break up AutoQA to be more modular
Some of this has already been brought up on autoqa-devel and we're planning to start working on it after 0.8 is released.
Decoupling some parts should lend in better self-test coverage
Refactor the test runner for AutoQA's self tests
It isn't structured well and I've learned a lot since I wrote it the first time.
The tests need to be easier to run and better set up so that we can start writing more of them
Granted, all of this is contingent on how much time I have leftover after testing Fedora 17. The beta release is coming up and there is no shortage of testing to do.
One thing that I think we've all done in the past is write comparisons of different tools and frameworks. I've certainly done several in the past and they were read one or twice (by nobody other than my team mates) and are forgotten unless the topic comes up again.
I wrote up a comparison of Nose and py.test about a year ago in the context of finding a framework for testing AutoQA. I was expecting it to be much of the same where only people interested in AutoQA would even read the thing.
Much to my surprise, I just saw the conclusion from that analysis in a talk at PyCon 2012. Don't get me wrong, I'm not complaining. I think it's awesome that someone else found it useful enough to reference - that's the point with licensing stuff under a CC license. I think of it as an incentive to make sure that stuff like that is reasonably well written.
One of the things that I've been thinking about lately is how to do more testing of builds in updates-testing without needing to rely on pre-written scripted test cases. There's nothing wrong with scripted testcases but they're a bit painful to write and they aren't always the best way to test a package.
Session-Based Test Management (SBTM) is a technique for testing that doesn't rely on pre-scripted test cases and produces more useful output than "I did some testing on foo". Ideally, this output can be used to both obtain some record of how foo was tested in addition to providing some loose documentation for others on how foo could be tested in the future.
I figured that I would do a simplistic testing session to demonstrate how to use a testing session to decide what kind of karma to give to critpath update in updates-testing. I chose to do parted-2.3-11.fc15 because
I'm already familiar with at least part of what parted is supposed to do
It's interface is relatively simple
parted is critpath and the newest F15 build is still in updates testing (at the time I'm writing this).
Yes, this is a somewhat manufactured test and doing a testing session for updates-testing validation is probably overkill. However, I want to start small and simple until I can better figure out how I want to format testing logs and the tools best used for a testing session.
Now that my session is done, I want to go through the raw log and do the following
Add the actual commands that I used to my log
I'm leaving out the command output because it can be rather verbose and I've included the screen log if there is something interesting in there.
Change the order of the log to reflect what I actually did
Research anything that I was confused by
Determine whether there are any bugs to file
Come to a conclusion on what kind of karma to file.
After editing, my final testing log looks like this. I think that the notes should be self explanatory on how I actually put them together but I still have some questions to answer:
What is a better way to record things like '@action' and '@unclear'? I don't like the use of @blah for this, it doesn't feel natural but I'm not really thinking of anything better at the moment Do I limit reporting my results to giving karma? In this case, there were no bugs to file and I didn't attempt to verify any fixes but I can see how referencing a testing log in a bug report or fix verification report would be useful.
Are testing logs like this useful to anyone? I probably need to do something more complicated before attempting to answer this. If logs like this areuseful, how do I share my testing logs so that others can find them and use them? fedorapeople.org is great for storing files but there are limits to how tied together everything can be if other people started doing this, too.
I'm going to keep trying to figure out if this style of testing is beneficial for fedora but would love to hear thoughts on the topic (good idea, bad idea, suggestions for improvement etc.). Otherwise, I'm planning to do another, more complicated session soon in an attempt to figure some of this out.
