Test automation with Robot Framework: Features and Limits

I have been in testing for more than 5 years. Of these, I have been in automation for about 2.5 years. Starting from automation in Python, I came across testing automation in Robot Framework at some point on a certain project, and after a deep study, I became a fan of it. In this article, I will try to tell you about its limitations and, of course, describe all its advantages.

Few words and pictures to get you acquainted with the Robot Framework

Before we discuss the pros and cons, let's talk briefly about the Robot Framework. In matter, if it may be the first time someone has seen this name.

Robot Framework is a keyword-driven framework explicitly designed for test automation. It's written in Python, but to write tests, you usually only need to use the keywords included in the framework without the need to program in Python. You can just load the necessary libraries, such as SeleniumLibrary, and write the test. In this article, I will give a general introduction to the Robot Framework, but if, after reading this, you want to go deeper into the topic, I suggest looking at the official documentation. 

Well, let's move on to the 'pictures'. This is what a simple project in IDE can look like (the example is from Wikipedia):

Blue and green - folders with files for page description and tests, respectively. In this way, you can implement the page object pattern.

Brown - drivers for different browsers.

Red - test body.

Yellow - the console from which you can run tests and see the console messages (full logs are not here, but about that later).

As you can see, the test is full of BDD-style 'wrappers' (you may not use this syntax, but it can be convenient here). The implementation is in page objects, for example:

In the standard Settings section, we see the Selenium library loading, and in the other standard Keywords section, we see the implementation of our self-written keywords.

I think this is enough to get a general idea. A detailed description of working with Robot Framework is a little beyond the scope of my article.

Pros and cons

In this part, I will talk about the pros and cons of the Robot Framework, which can be encountered. Given the specifics of the tasks on the project, you will probably have your own pitfalls as well. I will list, moving, in my opinion, from the more significant advantage/disadvantage to the less significant one.

PROS

Low threshold for entry

As I wrote above, the Robot Framework is a keyword-driven framework, not a programming language. Although the syntax is similar to Python,  knowledge of programming is less required or, let's say, its use is not necessary where the complexity of the task itself allows it. However, variables, loops, functions that return values, etc., can be used as needed. The closest alternatives may seem to be Pytest and Selenide, but they require more user training than the Robot Framework. For example, one of the built-in standard libraries is BuiltIn. There you can find keywords like Sleep, Log, Run Keyword If, Should Be Equal As Strings, etc., and easily write something like:

Run Keyword If '${status}' == 'PASS' SomeAction

Web and Mobile Support

Robot Framework works well with Mobile+Web (end-to-end etc.)

Web tests can work with Chrome, FF, and IE. The mobile part works with real local devices on Android and iOS, as well as with devices from the SauceLabs farm. The limitation is that a real local iOS device can only be tested with a Mac. And in general, iOS requires much more attention because the same web driver for it has to be rebuilt by yourself.

Tags

It is possible to set tags for tests. Tags can consist of any information we need to identify the test: test ID, list of components the test belongs to, etc. This way, we provide a link between tests and requirements (traceability) and set the necessary information to configure how to run tests. Having specified one tag in the launcher, we can launch all tests that refer to a specific component, or we can explicitly list the test cases that should be launched (useful for regression testing). 

Good reports out of the box

You don't have to invent anything to provide standard reporting. Reports are created automatically without a single additional command. It is possible to combine the results of different test runs. Three files are created as a result of the default test run:

Output.xml - test results in XML format. Useful for merging results with the rebot command. Example:

Log.html - detailed results in HTML format. More useful for test developers. Example:

Report.html - high-level results without detail. Useful to show to 'outside' people and management. Example:

BDD out of the box

The Gherkin language syntax with its Given, When, Then, and And notations is enabled by default, and any step can be written in this notation or without it. Whether you can use the notations or not, tests simply ignore them. For example, these two keywords are identical from the framework's point of view:

Welcome page should be open

And welcome page should be open

Parallel run

The standard preinstalled way is the robot command. Parallel execution is made possible by an alternative test launcher called Pabot. Of course, the tests must be designed for this and must refrain from interfering with each other.

CONS

Lack of debugging capability with built-in tools

Basically classic breakpoints. You either have to output something additional to the log or put temporary slips and thus bypass this problem. There are some ways described on the web to add debugging, but for the level of Robot Framework's target audience, it can be a bit complicated.

No AWS support

AWS (Amazon Web Services, a commercial public cloud, mobile device farm) does not support tests on the Robot Framework. AWS works in such a way that code is executed on the Amazon side, and the Robot Framework tests are not allowed. But for example another farm, SauceLabs, is built differently and works perfectly with Robot Framework.

Poor support for third-party libraries

Off-the-shelf, existing libraries on the web are sometimes not properly supported. Users are scarce, and they pass into the category of dead. For example, working with emails, comparing screenshots, etc. But you can, of course, write your own libraries in pure Python, though (and Robot Framework allows this)

For sure it is worth mentioning that for most purposes you don't have to write anything in Python and existing libraries are good enough.

Conclusions

Choosing the Robot Framework tool for your project can be right and can allow you to fulfill your obligations on time and with proper quality. However, we must understand that this is certainly not the 'Holy Grail', there are many 'buts' to keep in mind.

A tool is just a means to an end, and you don't always have to use a microscope to hit nails, even if it looks spectacular. I'm not going to over-criticize or over-praise the Robot Framework. I can only say that it is definitely a good tool for its purpose