When the reflow CLI is executed, it will compare its version against the API server running. It will warn upon any minor version increases, and fail on any major version increases.
This tool generally follows semantic versioning. Given a version number [MAJOR.MINOR.PATCH]:
- A MAJOR version increase is made when the API has changed in such a way that the CLI is now incompatible
- A MINOR version increase is made when a backwards compatible functionality is added
- A PATCH version increase is made when a backwards compatible fix/chore is implemented.
For stability, it is always recommended to always run the latest version of the reflow CLI.
This file is also available at https://reflow.io/changelog
June 07, 2022
Add missing dependency (esbuild) into npm package
June 07, 2022
Unlimited size snippets. Snippets now bundle dependencies. Snippets can bundle dependencies as text via "!inline!" as a prefix to the import path statement.
June 06, 2022
Fix element psuedo-selectors from being un-interactable: propagate interactions to parent elements.
May 17, 2022
Introduce low-interval graphql polling on the client to handle appsync subscription failures
May 16, 2022
Fix a recording edge case where the selected element wouldn't be correctly picked when there were multiple aria-label elements with the same value
Documentation when opening the test recorder in the cloud to recommend its run locally
May 13, 2022
--stable option in CLI to optionally enable CLI to do a stable run.
Speed up CLI by reducing unnecessary data load when making test runs
May 11, 2022
Allow extraction of more element attributes, including input values.
May 3, 2022
Track application releases via sourcemap content hashing
Capability to download sourcemap content.
Capability to associate multiple browser sequences with an application
Capability to disable screenshot stabilisation for faster action execution.
Optimized replay pathways: reducing all wait times on a per-action basis. Approximately 70% speedup.
Move screenshot comparison into a worker thread
Move screenshot comparison outside hot pathway: do not wait for comparison before progression
Do not load chatwoot unless button is pressed
Install browsers on reflow install
Remove duplicate page load on loading up a recorder instance.
April 19, 2022
Embed support widget (chatwoot)
April 15, 2022
Automatically install browsers after installing reflow locally
April 9, 2022
Improve mobile view of recording pane: make panel persistent, fix padding
April 8 2022
Expire images in frontend S3 cache after 15 minutes : this should prevent stale images from being shown on client, but still allow browser to cache the presigned image.
April 5 2022
Save test actions in parallel.
Sync theme between devtools pane and reflow main app
Auto reload chrome when DNS propagation appears to be an issue whilst waiting for recorder
Fix image comparison assertions inside deep iframe
Adjust backup to screencast frames to run more often
Increase expiry timeout on presigned images in UI to avoid them expiring after browser cache expires but before refresh.
Mar 25 2022
Mar 23 2022
CLI command to run a single image comparison. Local usage only.
CLI commands without API keys cache credentials on local machine for 15 minutes -- eliminating the need for re-login with sequential commands
New Action: "Execute Snippet" and corresponding "Push Snippets" "Pull Snippets" CLI commands to syncronize them between the server and a local folder.
Run Metric Graphs showing all run results and duration at the top of the run history.
Mar 17 2022
Handle additional image cropping edge cases where targeted element is hidden with width outside of the image width
Mar 17 2022
Increase Apollo Timeout in API Key CLI.
Mar 17 2022
Increase Apollo Timeout in JWT CLI to account for lower internet connection speeds.
Mar 17 2022
Fix edge case in image comparison when cropping an image to a size bigger than the destination; could happen upon significant visual changes.
Mar 16 2022
Ensure CLI-passed variables are also saved in test metadata for CLI filtering.
Mar 16 2022
Capability to pass device-emulation-profile name to CLI
Mar 16 2022
More resilient authentication via force refreshing session every 20 minutes.
Mar 16 2022
No longer require API key for CLI
pull command -- default to login if not available
Mar 14 2022
CLI command to download all images for a given test run. Requires API key.
Mar 13 2022
Capability to add an API key that allows cli/api access without login. Available to AWS Marketplace users only
Mar 08 2022
Capability to add a description to a Test Action
Mar 07 2022
Implement test name/description/device changes via UI
Handle ambiguity better when performing the "best selector" search.
Feb 28 2022
Documentation when opening the test dashboard with no pre-existing tests
Feb 28 2022
Auto healing button now also prompts to auto heal visual regressions
Feb 23 2022
Capability to send a kill command to a cloud runner to stop an in-progress run
Feb 22 2022
Preload the recorder whilst waiting for VM to start
Feb 21 2022
Always show picture of "best effort" element selector upon a failed-to-find element interaction
Add Auto heal button to rewrite the test action with that "best effort" selector.
More stability improvements in element match
Frontend optimization around realtime subscription updates
Feb 17 2022
Sort customer usage records
Enable sorting of more test/run grid columns
Feb 16 2022
Expose customer usage records for AWS Marketplace customers in the billing page.
Remove overlay of element dimensions on captured screenshots
Better handling of record page interactions where device scale does not match viewport scale.
Feb 14 2022
Serverless recorders in preparation for multi-region / custom runners push
Isolate runs to not run on shared VMs; each team now on an isolated virtual cluster. Breaking change due to API protocol change.
Expose more runner information in a Runners UI.
Expose more details about the Run's runner in API response
Feb 03 2022
Track speed of DOM changes for action element relative to the prior action finish time and use that to power element waiting loops. Speeds up replayed element interactions without sacrificing reliability.
Backend upgraded (now uses appsync pipeline resolvers), with some non-breaking graphql api changes introduced
Increased reliability around page frame changes; more retry loops whilst waiting for page frame to load.
January 16 2022
Software Licenses / AWS Marketplace integration
Integration with new backend infrastructure stack; adjusted API URL
When an initial URL is invalid, the recorder still opens and provides an appropriate error
November 23 2021
Dark mode now also applies to authenticator component
November 05 2021
Capability to use an emulated mobile device to record / run tests.
Capability to execute a test against a different device to the one configured in the test.
Capability to compose pipeline actions with different devices to the one configured in the test.
Visual Diff (Percentage) of the associated action page screenshot against its baseline is rendered on the test run action.
Capability to explicitly use a given baseline in the execute run UI.
When running a run without pre-defining a baseline, the most recent successful run for the same device/test version is used.
If it is the first run of that device, the most recent successful run of that test version is used. If it is the first
run of that test version, then it is not compared with any other run.
November 01 2021
Tab width of watch recording tabs works better on mobile
Lowered datastore overhead
October 29 2021
Nodes now register/heartbeat against a manager process that points them towards things to execute.
Avoids run contention and enables higher parallelism, usage based pricing. Breaking change, as polling for
runs now unsupported
Screenshot stability now powered by SSIM. Custom Mix of SSIM and pixeldiff used for visualistation of diff.
Playwright version now pinned to avoid playwright screenshot bug
First / last actions now show post/prior screenshot respectively (previously were just a blank screenshot)
Retry loops added to page stability functions
Frame injection ensured before executing findElement calls; fixing failure caused by frame too slow.
Fix capturing click/drag events when viewport is scrolled
Screenshots of selected nodes are only done when the custom action form is open.
October 20 2021
Assertions can now be raised as either warnings or errors (previously, only errors).
Assertions configured as warnings do not block the flow, but are highlighted when viewing results.
October 16 2021
October 15 2021
Highlight the elements that reflow interacts with in UI
Highlight mouse coordinates on selector / page images for mouse click or drag events
Completed elements now link appropriately to old test actions
Exponential-Backoff retry loops around element node screenshotting to improve reliability
<component>=<text> selector scores.
October 12 2021
Better support for refreshing devtools by creating a cdp session per connected one, instead of sharing a global cdp session
Better support for iframe actions where frame url doesn't exactly match
Better support for click/drag by tracking mouse location statefully.
CDP event for mouse release could, under some conditions, have incorrect X/Y params so instead "most recent"
mouse move event is now used
Add a backup method to CDP screencast frame/frame acks. CDP only has frame/frame-ack as a best-effort API. This
adds an interval check so that if we know about DOM updates but haven't had a new frame since before DOM updates
and a timeout (2.5s) the CDP screencast session is refreshed via stop/start APIs.
Show visual comparisons of elements after assert step.
Never show again option for action delete confirmations.
Enable chromium sandbox when run on local machines to reduce untrusted test execution risk.
October 4 2021
Only use S3 to store non-draft images
Visual screenshot assert: capability to asserts a given element matches the exact screenshot of an element when the test was first recorded
In custom action selector form UI, the selected element will now show a screenshot of the captured element just below the preferred selector.
Find typescript more reliably when invoked in a non-npm directory.
September 31 2021
Remove browser event memoization layer; recalculate selectors just before supplementing action details.
Ensure reflow only gets evaluated once per frame via stripping the magic symbol and replacing it with a hardcoded text constant on document root.
Increase performance inside browser by memoizing events and only recalculating selectors once per event.
Increase performance by only sending mouse move events when mouse is down.
September 30 2021
Fill actions with a selector containing
input[type="password"] get their text title replaced with '*'s.
Show number of found elements and associated score for each selector in selector layout UI.
Support for clicking / dragging elements.