I have used both fixtures and factories in writing tests. I actually use both currently I use fixtures for my personal project and factories at work. I feel that their can be benefits to both but if I had to go with one I would go with fixtures. It seems to me that you can do everything and more with fixtures. Also your test suite stays a little more organized.
The first question one may ask is what is the difference between a fixture and a factory. Well a fixture is a set of data that is static in nature. This data will get loaded for each test run. A fixture is created using a static file within your testing structure. This is different for each language you use it in. For example in Rails it will be a yaml file while in PHP it’s just an array. This could also depend on the framework your using. If you want the table to start out empty you can create an empty fixture file. Having the static file allows you to setup specific test scenarios. Then you can write your tests around those scenarios and you know the data will always be the same. So if a test shall fail you know it’s not because of a dataset change. In contrast factories create the object and the data at the same time. The database is not stocked with any preloaded data. If you need data for your specific test case you need to call the factory and have it create the data for you. Instead of creating a static file you need to create a factory file. This file will contain the name of the factory and then any custom data you need. Using a factory can have some benefits as you will get random data so your code is being tested against unknown data. Which may be reasonable as you have outside users using your system passing in random sets of data to.
So why do I feel that fixtures are better than factories if fixtures take longer to setup. I find it that factories tend to get messy as you have data objects being created all over your test suite. No factory is identical so you tend to have to work though a lot of different data scenarios. Also if you don’t setup your factory to pass in the proper data you could get false positives. For example maybe your interface validates a field to be one of three values. But you forget to setup your factory with one of those values. Now your tests are failing but the interface and application are running just fine. I suppose you could make the argument you can make the same mistake in a fixture but I feel you have to think about it more as you are filling this data in. Another reason why I don’t think factories are better is they are slower running. For example if you create your fixture files they run once per test run. But they run fast because they don’t have to use active record and create the object. Factories do create an object while creating the data. This takes extra time especially if you have setup a lot of data in your scenario. Another issue I have with using factories is you have to setup your data scenario on each test. You can keep it dry in a way by calling sub classes but again a lot of times you end up just doing factory.new because it’s a one off type of setup. So you tend to duplicate setting up factories a lot especially in a larger development team.
I feel that using either factories or fixtures will be fine. But I feel that fixtures gives you a slight edge for keeping your test suite organized. You are able to write tests against predictable scenarios. Also you are not having to repeat the same datasets over and over again. I feel that this is easier to avoid using fixtures rather than factories.
Hi Jake,
This is one of the few excellent posts comparing factories and fixtures. I feel you gave them a very balanced treatment, albeit making clear your preference for fixtures.
The 2 main criticisms that stick out in my mind are:
1) They do not go through validations in the model.
2) They are brittle and more susceptible to breaking when there are changes in the UI
I have written a guide to Rails 4.1 Testing with Test::Unit, Minitest, and Rspec which just mentions the strengths of fixtures or factories. What the guide does in depth is go over the various configurations that are used when setting up the 3 main test frameworks in different combinations of factories, fixtures, Minitest, Rspec and Capybara.
You can find the guide on my blog Ruby Reflections
Hope it may be of use to those transitioning to Rails, or specifically Rails 4.1.
Cheers!
Hi Jerry,
Thanks for your comment. I am glad that you thought my write up was balanced. I have used both pretty extensively so I believe I can see both sides. I don’t think there is an easy answer both have their benefits and down sides.
You do make a good point about fixtures not going through validations. I would have to say though that is one of fixtures greatest strengths. Fixtures don’t have to spend time creating the data real time. Instead they run before your test suite runs and are available for use throughout your test cases. So you can reference them as objects just like you would as a factory. I think as a developer you should ensure that your fixtures have proper data. If you accidentally miss making a certain data point valid your tests will most likely catch this. If it’s a serious validation flaw. If your tests don’t catch it then maybe that data is okay to be in an invalid state. Of course if your tests try to update that record you will get the validation error at that point. That will alert you that something is wrong and needs to be addressed. I think that it’s okay as long as your testing your validations through your controller tests and even integration tests. This will ensure any new data the user is entering is in a valid state. So I agree factories will catch this up front at the end of the day maybe you don’t really care about that specific record for your specific case.
I have to disagree on your second point. I don’t see them being brittle at all. I have used them pretty extensively over the years and haven’t really notice any breakage. I may have been getting lucky. Also my data model doesn’t change often so there hasn’t been a ton of updates to them. The projects that I have used them on were single developer projects so having more developers could introduce conflicts. So maybe it’s possible that it’s problematic on a larger team?
I almost think that this same argument could be made for factories also. I have been on projects where F]factories are all over the place. A schema change happens and now you have go through your entire test suite and update a bunch of factory calls. Yes, your test suite will catch them but it’s still painful to go back and update them all. Also I see it being difficult to maintain what data goes where. It just feels less organized to me.
Thanks again for your comment. In all honesty it’s all about what you prefer. I have been happy with fixtures and their performance becomes more valuable as the project goes. The more factories you add the longer your test suite takes. You can say the same for fixtures but the difference becomes larger and large as it grows.