In theory you should always write tests while you are developing your code. In the software industry this is called test driven development . This practice will usually generate some good thoughts about how you can make the code more robust and more reusable. It’s very difficult to change architecture after you have coded yourself into a hole. It’s a whole lot easier to do it right the first time than going back a second time and refactoring what you did the first time. When you have to go back and write your tests after you wrote the code it will tend to cause more rework versus writing the tests before you actual develop the code. Sometimes it’s hard to write tests as you go as requirements are changing all the time. This can cause frustration because you feel your tests are wasted time. A common excuse I hear is requirements are always changing so I have to go back there anyways. But if your developing the code the requirements are usually pretty set at that point. At least the overall idea. Maybe the guts of the code will change but the public interface shall remain the same (at least close to the same). I had an experience on a project that I am working on that made this point pretty clear. My task within the project was the write some tests around existing code. Basically adding code coverage to an existing model. Seems simply enough until I started looking into the model and discovered a lot of very tightly coupled code. A lot of tightly coupled public interface code.
Starting this process I was able to test a lot of the public interface. The parts that were not tightly coupled. Maybe about half of the public methods were in a testable state. A unit testable state to be exact. Then I ran into a major road block. I ran into some public interface methods that required a chain of methods to be called first in order to be able to test them. If TDD was used during development process this would have been avoided. The developer would have realized that the code was to coupled to be tested. So instead of saving that time I spent even more time refactoring and trying to decouple the methods. The end solution ended up a lot better but it required a lot more effort than just writing the test before the code or at least right after you have written your code.
It’s a good practice to get into even though at first it may be difficult to get into the habit. It’s even harder to do it on a regular basis when things are moving so fast. I believe that once you get good at it though it become’s natural and really part of your process. Also the more familiar you are with architecture in general and architecture within the project the easier it will become. You can see your vision ahead of time and predict what the code should look like. To sum it up always write your tests as you go. If you need to write them after your code that’s better than leaving them and letting them get behind.
I use Jenkins to run my Continuous Integration server. I am the sole developer on the project but I still find it useful to run the tests on a different machine. There are a couple of benefits to having a Jenkins server running. First of all it lets my local machine be available at all times. Meaning I am never taking up resources to run my entire test suite. Also I have a pretty large test suite so it’s nice not to have to wait for it to finish. Also the more selenium tests you have running the longer the process will take. Also when you run selenium tests locally it pops open the browser each time so you have to keep minimizing it. Moving on to the point of this post.
Usually when I have an issue with my Jenkins server I assume something happened during the test run. Jenkins must have not ran correctly no way can it be my code. Well I got burned by this. I let my Jenkins server runs fail for about a week before I really took the time to investigate the actual issue. I was assuming something happened specially with Jenkins. Maybe an update I did on Firefox or Jenkins. Well it turned out that it wasn’t Jenkins fault but instead it was mine. I had updated my code base to use the latest version of Yii but forgot to update the Jenkins server to that version. I did this a while ago so that is why I wasn’t thinking of it right away. I was not using the features of the latest version until my recent code push. To make a long story short always trust Jenkins and investigate issues right away. Most of the time it will be a problem with your code not how Jenkins is handling it. At the end I have a passing Continuous Integration server but the process took a lot longer than it should have.
I was recently working on a Gem I created called Toolshed. The purpose of this tool is to turn everyday tasks into automated ones. At least to a certain degree. This will allow you more time to do the heavy lifting tasks. For example some of the functionality that this tool provides is creating a Github pull request, updating Pivotal ticket status as well as many Git more tasks. All of this would normally require you to go up into the sites interface click around and enter the data. As we all know that takes time and time is valuable. So down to the part that I had trouble with when I was creating tests for this tool.
I first started to create all of the tests locally and verify that they worked there. I had no trouble with this and everything appeared to be working. I was not getting any test failures. After creating a couple of them I wanted to see if I could get it working up on Travis CI. Travis works well with a ruby project plus since the Gem is open source it’s free. So I setup Travis and everything seemed to be working right. The first couple of runs seemed to pass without issue. Over the next few days I noticed that Travis was failing randomly. I would push up a small change and it would fail. I would run it again and it would pass using the same code base. It seemed random and a CI that has random failures is useless. The theory behind that is if errors happening randomly there they will also happen within your application. So I needed to investigate this further. Since I was unable to recreate the issues on my local machine I decided to try it out on a different machine. I created an ubuntu environment and started fresh. Once I got to that environment I was able to reproduce the problems I was having on Travis pretty easily. I was able to debug real time which helps instead of pushing new commits up to Travis.
What I noticed was that my Git commands were not finishing before it started the next one. So it would be doing a git remote update and then the next command would run before that finished. This would cause it to fail because it depend on remote update finishing first. So I had to create a solution for this and here is that solution.
until (system("git remote update"))
This will sleep and wait until it hears a response. This will now wait for each command to run in it’s specific order. I have now tested it out several times on Travis and am no longer having this issue.
I have been working with Android for years now. When I first started to work with it I never really thought about adding tests. As of late my application has gotten big enough to where tests are important. After doing some research I found that I was able to do both unit tests and functional tests. Unit tests are difficult in Android though because most of your code uses the Android helpers and you need to create mock objects. I decided instead of writing unit tests I would write some simple functional tests for the time being. To do that I used a tool called Robotium.
This tools goes along side of the Android functional test framework and makes things very simple. For example you can do this using Robotium.
From my experience with it it’s been a great wrapper. It has saved me hours of writing my own wrappers for my tests. Also the time it took to create these functional tests did not take long either. I encourage anybody who has an Android application to use this wrapper to get their functional tests up and running.
I have used both Jenkins and Travis CI professionally. Both tools will do the job but there are some distinct differences between them. I believe that you can get the job done with either one of them and the purpose of this post isn’t to pick one or the other. I think they bot viable tools and depending on the situation one may be better than the other.
Travis is free for open source projects. However it will cost you if your project is private. The cost starts at $129 per month and goes up from there. The cost isn’t that bad if you have many projects that you want to host and don’t have your own server to host them. They charge you a service fee not a per project fee. The price for Jenkins is free if you set it up on your own server. You can use it both for open source and private projects. There is also Jenkins hosting available if you are unable to configure the server yourself. That would come at a cost though.
Travis is maintenance free because it’s being hosted for you. They handle all of the server updates. All you have to do is maintain a configuration file. The configuration file is basically the toolset you are using. For example the Ruby version or PHP version you are using. Jenkins requires little maintenance and has a built in GUI tool. The purpose of the tool is to allow for easy updates. If you are hosting on your own server you will have to maintain that server to. Maintaining a server is a little work but not a ton of effort required for that either. Basically just updates from the operating system.
Travis has a very fast setup time. All you need to do is create a configuration file. This takes little effort to get going. Jenkins has a little bit of setup time ranging from a couple of hours to a couple of days. This depends on the types of projects you are setting up. There is a lot of documentation out there to setup a Jenkins server to. This is available in languages such as PHP, Ruby on Rails and Android. From my personal experience I have found that setting it up is pretty straight forward. Also once it’s complete it’s very simple to maintain.
My opinion is that either of these continue integration servers will do the job. It really depends on if you would rather spend the time upfront setting it up or just pay a monthly fee. If your project is open source Travis would be the best option in my opinion because there is no setup time and no fee. If your project is private Jenkins would be the best option in my opinion because the setup time is minimal and maintaining the server is pretty simple. I think either one you use would work. It just depends on personal preference and the situation your project is in. I use both currently and love both for different reason.