Running the test suite against other APIs?

[julien51]

Hi,

Is it possible to run the test suite against other API and even against Google's endpoint? That would be a great acceptance/compliance test.

Please, let me know if this is doable and if so, how :)

Thanks

[devongovett]

I would imagine so, with some small changes. It mostly just makes HTTP requests and checks the responses, but there are a few places where it currently accesses the database directly so those would have to be removed...

[devongovett]

Also, I'm handling errors differently than Google does (i.e. returning Error=SomeErrorCode strings rather than whatever google returns). I'm not sure if this will cause interoperability problems, but my guess is that most clients simply look at the HTTP status code to see if things went well rather than trying to understand Google's various error pages.

[devongovett]

Started work on this. See c5d7041

[devongovett]

FYI, just added the ability to run the acceptance test suite with different APIs. Just make a JSON file with the server and auth settings (see googlereader.json as an example) and specify it on the command line when running the tests and those settings will be used instead.

[julien51]

@devongovett I finally found the time to play with it :) This is great and works pretty well, but for a few failing tests.

Using Google Reader as the backend, I have a couple failing tests:

Auth
 ✖ user-info
    Julien51 != julien.genestoux
  ✖ user-info xml
    Julien51 != julien.genestoux

Subscription
 ✖ subscription list
    [object Object] != [object Object]
 ✖ subscription OPML export
    [object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object ... Object] != [object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Tests completed in 19385 milliseconds.
934 tests of 938 passed, 4 failed.

Let me know if these are "expected" or if you need me to check into that and provide a pull request.

I'm now testing with Superfeedr's Google Reader API replacement. Will report soon.

[devongovett]

Great, thanks for running it. The issue with the user-info is one I probably should have expected. Apparently you have a different "screen name" than just the first part of your email address, so the test should probably change. As for the other tests of subscriptions, were you running the tests against a totally clean account to start with? The test suite is counting on the subscription list being empty to start with so the subscriptions are consistent. I just created a second Google account to test with so I wouldn't mess up my Google Reader anyway... Let me know if that helps.

[julien51]

@devongovett Ok, I just submitted a second pull request which tests for inclusion rather than full equality so that we can get them working for even if you don't have an empty Google Reader account. Of course, there may still be some cases where that would fail (say you have these feeds with specific titles in your subscription list) but that should be good.

I'll start fixing our API now so that these acceptance tests pass there too now!

[devongovett]

Yeah, and really the test suite should unsubscribe from all feeds it subscribed to, and remove all the tags that it made before exiting in order to reset the account. This won't work if you are in a non-empty account and happen to actually want to be subscribed to one of those feeds though. So use this with a non-test account at your own risk. :)